jlevy/repren

Rename anything

repren's Issues

"--at-once" option

Hi,

I can't seem to get the "--at-once" option working. My OS is "Linux Mint 17 Qiana".
I'm not really sure what version of Python is being used. "python2.7 -V" gives "Python 2.7.16", but When I do "python -V" gives "Python 2.7.6". I tested on a file containing "hello\nthere". It seems to find the match, but not do any changes:

repren -p toss_repren --at-once -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt
Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
Changed 0 files (0 rewritten and 0 renamed)

When I drop the "--at-once" option, it works:

repren -p toss_repren -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt

• modify: ./toss.txt: 1 matches
  Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
  Changed 1 files (1 rewritten and 0 renamed)

I also did "bash run.sh" to run the tests. Most seemed to work but at the end I got "ls_portable test2
tail: cannot open ‘+2’ for reading: No such file or directory". Thanks in advance for your help.

More extensive large-scale test

This is a big omission. Currently testing informally.

Need some end-to-end automated test to cover lots of corner cases by doing some complex renames on a complex tree of files and validating the matches. Can do a sequence of operations including rewrites, renames, and swaps, record the tallies, put them all back, and validate the final result is identical using diff -r.

Perhaps simplest to automate with a silly script harness a bit like https://github.com/jlevy/procdog/blob/master/tests/run.sh

Nondestructive mode can create empty directories

When using dry-run mode (-n), files won't be modified.

However, if we are renaming entire paths, the implementation of transform_file() temporarily creates the parent paths of the new, rewritten path as part of the dry run (to better reproduce real behavior, and because it counts matches).

This means it can leave behind new, empty directories.

This isn't a really big deal, but it's arguably a bug.

TypeError: can't use a string pattern on a bytes-like object

Trying to replace a string in files inside a directory I get the following error:
Dry run: No files will be changed Using 1 patterns: 'xxx' -> 'yyy' Found 6617 files in: .eclipse/ Traceback (most recent call last): File "/usr/bin/repren", line 534, in <module> dry_run=options.dry_run) File "/usr/bin/repren", line 409, in rewrite_files rewrite_file(path, patterns, do_renames=do_renames, do_contents=do_contents, by_line=by_line, dry_run=dry_run) File "/usr/bin/repren", line 375, in rewrite_file counts = transform_file(transform, path, dest_path, by_line=by_line, dry_run=dry_run) File "/usr/bin/repren", line 339, in transform_file counts = transform_stream(transform, stream_in, stream_out, by_line=by_line) File "/usr/bin/repren", line 307, in transform_stream (new_line, new_counts) = transform(line) File "/usr/bin/repren", line 374, in <lambda> transform = lambda contents: multi_replace(contents, patterns, source_name=path) File "/usr/bin/repren", line 209, in multi_replace for match in regex.finditer(input_str): TypeError: can't use a string pattern on a bytes-like object

The exact command given is: repren --from xxx --to yyy --dry-run .eclipse/

My guess is that this happens because there are binary files in the directory.

Could there be an option to skip binary files?

How to rename directories?

This is an example:

$ mkdir 1
$ cd 1
$ touch {1,2,3,4,5}.txt
$ mkdir {6,7,8,9}
$ cp *txt 6
$ repren -n --renames --from "^" --to "aaa_" [6789]
Dry run: No files will be changed
Using 1 patterns:
  '^' -> 'aaa_'
Found 5 files in: 6, 7, 8, 9
- rename: 6/4.txt -> aaa_6/4.txt
- rename: 6/1.txt -> aaa_6/1.txt
- rename: 6/3.txt -> aaa_6/3.txt
- rename: 6/5.txt -> aaa_6/5.txt
- rename: 6/2.txt -> aaa_6/2.txt
Read 5 files (0 chars), found 0 matches (0 skipped due to overlaps)
Dry run: Would have changed 5 files (0 rewritten and 5 renamed)

$ repren --renames --from "^" --to "aaa_" [6789]
Using 1 patterns:
  '^' -> 'aaa_'
Found 5 files in: 6, 7, 8, 9
Traceback (most recent call last):
  File "/home/alexey/bin/repren", line 531, in <module>
    dry_run=options.dry_run)
  File "/home/alexey/bin/repren", line 406, in rewrite_files
    rewrite_file(path, patterns, do_renames=do_renames, do_contents=do_contents, by_line=by_line, dry_run=dry_run)
  File "/home/alexey/bin/repren", line 372, in rewrite_file
    counts = transform_file(transform, path, dest_path, by_line=by_line, dry_run=dry_run)
  File "/home/alexey/bin/repren", line 357, in transform_file
    move_file(source_path, dest_path, clobber=False)
  File "/home/alexey/bin/repren", line 299, in move_file
    shutil.move(source_path, dest_path)
  File "/usr/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/usr/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: 'aaa_6/4.txt'

So it is a bug plus feature request (rename folders only, or at least rename them like files)

Automatic testing

Since repren has a test script, we should run it automatically with Travis CI.

I'm going to take a look at this now.

is it possible randomly rename all variables in python file

great code thanks
is it possible randomly rename all variables in python file

so have file from github and want to make it not recognizable for others?

Use unicode for internal processing?

Firstly, many thanks for putting this up - I think it's brilliant. In addition to code refactoring this is tremendously useful for people editing and revising books (with LaTeX, markdown and/or reStructured Text sources). For this purpose it would be very helpful to have repren deal also with non-ascii characters (e.g. accented characters in foreign words).

Would you consider adding an --encoding option to enable users to specify file encoding? repren could then decode the inputs read from pattern and input files to unicode, do all the internal processing in unicode and then encode again when writing output.

--dry-run --renames broken

When doing renames with no replacements, --dry-run is broken in 0.3.6.

--from does not accept pattern starting with #

$ repren --from '#include <a.h>' --to '#include <b.h>' .
error: found no parse patterns

Same with --literal:

$ repren --literal --from '#include <a.h>' --to '#include <b.h>' .
error: found no parse patterns

This is because it interprets the # as a comment character (in function parse_patterns()).

It would be nice if this comment logic did not apply when patterns are supplied directly via --from --to, and/or when --literal is given.

Backup destination configuration

First off, thanks for the great tool.

I like the fact it auto-creates backups. The only thing it seems to be missing is the ability to configure where backups are stored. Is this currently possible?

Windows support

Repren works on Mac and Linux. Currently no testing has been done on Windows. Would be great to have some help with this.

clearer help about default mode

It is not clear from the documentation or from the --help page which mode is the default one.

I'd suggest either adding this into --help page:

- repren: Multi-pattern string replacement and file renaming
+ repren: Multi-pattern string replacement (default) and file renaming tool

or adding a command-line option:

  -p PAT_FILE, --patterns=PAT_FILE
                        file with multiple replacement patterns (see below)
+  --replace-only         do search/replace on file contents (default)
  --full                do file renames and search/replace on file contents
  --renames             do file renames only; do not modify file contents

or adding a simple example to the README that does nothing but a simple search-and-replace:

repren --from center --to centre .

This is the mode that I use most, but it was tricky to find out the exact syntax from the docs.

Add option to skip binary files

./bin/k2pdfopt_linux is a binary file which should not be touched:

repren --dry-run --literal --from agC --to ag_c .
Dry run: No files will be changed
Using 1 patterns:
  'agC' -> 'ag_c'
Found 200 files in: .
- modify: ./bin/k2pdfopt_linux: 1 matches
- modify: ./zsh/auto-load/others/ag.zsh: 3 matches
Read 200 files (14410423 chars), found 4 matches (0 skipped due to overlaps)
Dry run: Would have changed 2 files (2 rewritten and 0 renamed)

`--at-once option

nt5a190812 [zhi:github_issue_aut_zhi_s]:*--at-once option^ (cntn Hi,

I can't seem to get the "--at-once" option working. My OS is "Linux Mint 17 Qiana".
I'm not really sure what version of Python is being used. "python2.7 -V" gives "Python 2.7.16", but When I do "python -V" gives "Python 2.7.6". I tested on a file containing "hello\there". It seems to find the match, but not do any changes:

repren -p toss_repren --at-once -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt
Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
Changed 0 files (0 rewritten and 0 renamed)

When I drop the "--at-once" option, it works:

repren -p toss_repren -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt

• modify: ./toss.txt: 1 matches
  Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
  Changed 1 files (1 rewritten and 0 renamed)

I also did "bash run.sh" to run the tests. Most seemed to work but at the end I got "ls_portable test2
tail: cannot open ‘+2’ for reading: No such file or directory". Thanks i advance for your help.

Regular expressions dialect

Hi there, great tool, really useful.

From the documentation, it is implicitly deduced that the expressions may be regular expressions. It would be nice to know exactly what dialect of regular expressions is accepted and what features are available.

Thanks!

--preserve-case enhancement: also use underscore-less CamelCase translations

Currently, when --preserve-case is used to translate a CamelCase replacements to additional lowercase and uppercase replacements, it inserts underscores in between the word segments. This is cool, but on some occasions it would be useful to additionally do the replacement without the added underscores.

For example, the current behaviour is like this:

$ repren --full --from CamelCase --to DromedaryCase --preserve-case .
Using 4 patterns:
  'CAMEL_CASE' -> 'DROMEDARY_CASE'
  'CamelCase' -> 'DromedaryCase'
  'camelCase' -> 'dromedaryCase'
  'camel_case' -> 'dromedary_case'

It would be awesome if it could be extended to do this:

$ repren --full --from CamelCase --to DromedaryCase --preserve-case .
Using 6 patterns:
  'CAMEL_CASE' -> 'DROMEDARY_CASE'
  'CAMELCASE' -> 'DROMEDARYCASE'
  'CamelCase' -> 'DromedaryCase'
  'camelCase' -> 'dromedaryCase'
  'camel_case' -> 'dromedary_case'
  'camelcase' -> 'dromedarycase'

Perhaps this could be opt-in.

Support non-recursive renames on directories?

repren recurses into directories if they are supplied as arguments. This always makes sense for search/replace.

One feature I've heard: If you're doing renames and not search/replace, you might want to rename directories without recursing. Not sure if this is common or the best solution. Perhaps a --no-recurse option that only applies to files/directories that are listed?

Out-of-memory errors can be destructive

...unless the user uses backups. Multi-gigabyte stuff can be tricky (lost one XXL-sized file this way, fortunately an intermediate result though), so I would add some words of caution into the help screens/README, viz. after the following passage:

"File operations are done atomically, so interruptions never leave a previously existing file truncated or partly edited."

insert this:

"Users must be careful when processing very large files with --at-once option turned on."

Support to only include certain files? Support config file?

Hi there! the tool is really useful. I think it would be great if a simple config file would be supported where the settings are specified.

also would be useful to support --include as well as --exclude. In my case for example it is easy to specify what I want repren to work on... only hpp and cpp files

Cheers!

Do not rename files, only content?

Possible to only rename the contents of files, and not the file names themselves?

I don't understand how --literal works, e.g.

python /Users/storm/bin/repren --dry-run --literal -i --from='light.mini_sov' --to='light.bedroom_mini' test
Dry run: No files will be changed
Using 1 patterns:
  'light\.mini\_sov' IGNORECASE -> 'light.bedroom_mini'

Why does the from escape the dots, while the to doesn`t?
I've tried including -i and --literal twice, but to no avail :)

Python 3 support

Support both or simply migrate to Python 3.