https://github.com/mathiasrw/rexreplace
Smooth find & replace CLI as it should be: simple file selection with globs, flexible patterns with regex and dynamic replacements from js code.
https://github.com/mathiasrw/rexreplace
cli regexp replace-in-files
Last synced: about 1 year ago
JSON representation
Smooth find & replace CLI as it should be: simple file selection with globs, flexible patterns with regex and dynamic replacements from js code.
- Host: GitHub
- URL: https://github.com/mathiasrw/rexreplace
- Owner: mathiasrw
- Created: 2017-03-15T11:10:25.000Z (about 9 years ago)
- Default Branch: main
- Last Pushed: 2024-04-11T10:10:41.000Z (about 2 years ago)
- Last Synced: 2024-04-23T21:08:36.020Z (about 2 years ago)
- Topics: cli, regexp, replace-in-files
- Language: HTML
- Homepage:
- Size: 6.1 MB
- Stars: 25
- Watchers: 3
- Forks: 3
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# RexReplace
[](https://github.com/mathiasrw/rexreplace/actions)
[](https://npm-stat.com/charts.html?package=rexreplace)
[](https://www.npmjs.com/package/rexreplace)
[](https://app.fossa.io/projects/git%2Bgithub.com%2Fmathiasrw%2Frexreplace?ref=badge_shield)
[](http://open-oss.com)
> RexReplace is a versatile tool to search and replace text in files from the command line. It's inspired by how developers often need to do quick fixes or one-liners for build scripts.
**Key features**:
- Easy and intuitive notation makes you trust what you are doing
- Replacement can be javascript code - giving you Turing complete flexibility
- Pinpoint the exact files with [glob notation](https://www.tcl.tk/man/tcl8.5/tutorial/Tcl16a.html) (`docs/*.md` represents each markdown file in `docs/`)
- No more brute-forcing the right combination of `find`, `cat`, `sed`, `tr`, and `awk` to replace a text pattern in the right files
## Install
To use RexReplace from your command line
```bash
> npm install -g rexreplace
```
To use RexReplace from an npm build script:
```bash
> yarn add rexreplace --dev
# or
> npm install rexreplace --save-dev
```
## Examples
Let 'foobar' become 'xxxbar' in myfile.md
```bash
> rexreplace 'Foo' 'xxx' myfile.md
```
Hard for your fingers to write on your keyboard? We got you covered with the `rr` alias for `rexreplace`
```bash
> rr Foo xxx myfile.md
```
---
#### Catch the beginning
Let all markdown files in the `docs/` dir get headlines moved one level deeper
```bash
> rexreplace '^#' '##' docs/*.md
```
---
#### Using glob notation to pinpoint files
Fix a spell error in all javascript and typescript files in the folders `src/` and `test/` recursively.
```bash
> rexreplace 'foubar' 'foobar' '{src,test}/**/*.{js,ts}'
```
---
#### Dynamically generated content
Let the version number from package.json get into your distribution js files (use the string `VERSION` in your source files).
```bash
> rexreplace 'VERSION' 'require("./package.json").version' -j dist/*.js
```
Require have been given the alias `r` and both are expanded to understand relative paths even without `./` prepended. As the file extension is not needed eighter you will get the same result writing:
```bash
> rexreplace 'VERSION' 'r("package").version' -j dist/*.js
```
#### Reference a matching group
Let 'foobar' become 'barfoo' in myfile.md
```bash
> rexreplace '(foo)(.*)' '$2$1' myfile.md
```
RexReplace defaults to treating `€` as an alias for `$` so the following will do the same as the previous example
```bash
> rexreplace '(foo)(.*)' '€2€1' myfile.md
```
---
#### Surviving backslash-escape hell
Let `foo[bar]` become `foo.0.[bar]` in myfile.md
```bash
> rexreplace '\[' '.0.[' myfile.md
```
The `[` as a literate char must be escaped according to regex. If you run the command as a parameter (this could be from a script in package.json) you need to escape the escape:
```json
"scripts":{
"fix": "rexreplace '\\[' '.0.[' myfile.md"
}
```
RexReplace defaults to treating `§` as an alias for `\` so the following give same result:
```bash
> rexreplace '§[' '.0.[' myfile.md
```
```json
"scripts":{
"fix": "rexreplace '§[' '.0.[' myfile.md"
}
```
---
#### More relevant examples
**Per file info**
Add creation time, name of the file and human readable file size as the first line in each file in `test-run` recursively.
```bash
> rexreplace '^' 'ctime_ + name_ + size + nl' -j -M 'dist/**/*.*'
```
**Matching pairs**
Let both `"foo 'is' bar"` and `' foo "was" bar'` ' become `» foo ... bar «` independent of using `'` or `"`
```bash
> rexreplace '(['"])([^\1]+)\1' '» $2 «' myfile.md
```
---
## Usage
```bash
> rexreplace pattern replacement [fileGlob|option]+
```
| Flag | Effect |
| ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-v` | **`--version`** Print rexreplace version (can be given as only argument) [boolean] |
| `-V` | **`--verbose`** More chatty output [boolean] |
| `-L` | **`--literal`** Literal string search (no regex used when searching) [boolean] |
| `-I` | **`--void-ignore-case`** Void case insensitive search pattern. [boolean] |
| `-G` | **`--void-global`** Void global search (stop looking after the first match). [boolean] |
| `-s` | **`--dot-all`** Have `.` also match newline. [boolean] |
| `-M` | **`--void-multiline`** Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] |
| `-u` | **`--unicode`** Treat pattern as a sequence of unicode code points. [boolean] |
| `-e` | **`--encoding`** Encoding of files/piped data. [default: "utf8"] |
| `-E` | **`--engine`** What regex engine to use: [choices: "V8"] [default: "V8"] |
| `-q` | **`--quiet`** Only display errors (no other info) [boolean] |
| `-Q` | **`--quiet-total`** Never display errors or info [boolean] |
| `-H` | **`--halt`** Halt on first error [boolean] [default: false] |
| `-d` | **`--debug`** Print debug info [boolean] |
| `-€` | **`--void-euro`** Void having `€` as alias for `$` in pattern and replacement parameters [boolean] |
| `-§` | **`--void-section`** Void having `§` as alias for `\` in pattern and replacement parameters [boolean] |
| `-o` | **`--output`** Output the final result instead of saving to file. Will also output content even if no replacement has taken place. [boolean] |
| `-A` | **`--void-async`** Handle files in a synchronous flow. Good to limit memory usage when handling large files. [boolean] |
| `-B` | **`--void-backup`** Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is abrupted. [boolean] |
| `-b` | **`--keep-backup`** Keep a backup file of the original content. [boolean] |
| `-m` | **`--output-match`** Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). [boolean] |
| `-T` | **`--trim-pipe`** Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean] |
| `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] |
| `-j` | **`--replacement-js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command (null if no piped data), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] |
| `-x` | **`--exclude-re`** Exclude files with a path that matches this regular expression. Will follow same regex flags and setup as the main search. Can be used multiple times. [string] |
| `-X` | **`--exclude-glob`** Exclude files found with this glob. Can be used multiple times. [string] |
| `-h` | **`--help`** Display help. [boolean] |
## Good to know
### Features
- Patterns are described as javascript notation regex
- Pattern defaults to global multiline case-insensitive search
- Supports regex lookaheads in the pattern
- Supports backreference to matching groups in the replacement
- Data to be treated can be piped in
- See the [release note](https://github.com/mathiasrw/rexreplace/releases) for a log of changes. Descriptions are given in latest patch version.
### Quirks
- Options can only be set after the replacement parameter. "_But I like to put my options as the first thing, so I know what I am doing_" I agree, but we must sometimes sacrifice habits for consistency.
- Per default `€` is treated as an alias for `$` in the CLI input. The main reason is for you not to worry about how command line tools often have a special relationship with the `$` char. You can escape your way out of this old love story, but it often pops up in unexpected ways. Use the `-€` flag if you need to search or replace the actual euro char.
- Per default `§` is treated as an alias for `\` in the CLI input. The main reason is for you not to worry about escaping the escape of an escape. Use the `-§` flag if you need to search or replace the actual section char.
### Limitations
- RexReplace reads each file fully into memory, so working on your 8Gb log files will probably not be ideal.
- For versions of Node prior to 6, please use version 2.2.x. For versions of Node prior to 0.12, please use [the legacy version of RexReplace called rreplace](https://www.npmjs.com/package/rreplace)
### Priorities
- Flexibility regarding text pattern matching
- Easy to filter what files to be treated
- Helpful interface
- Tests (if you know how to do a test cover report on javascript code ran via the command line, please let me know)
### Not a priority
- Speed. Well... obviously, speed is important, but to what extent does a 29 millisecond command really satisfy the user compared to a 294 millisecond command? See _test->speed_ for more info.
```bash
> time cat README.md | sed 's/a/x/g' > /dev/null
cat myfile 0,00s user 0,00s system 45% cpu 0,011 total
sed 's/a/x/g' > /dev/null 0,00s user 0,00s system 43% cpu 0,029 total
> time rr a x README.md -o > /dev/null
rr x y myfile -o > /dev/null 0,21s user 0,04s system 86% cpu 0,294 total
```
## Test
### Regression
All CLI end to end tests are defined in [test/cli/run.sh](https://github.com/mathiasrw/rexreplace/blob/main/test/cli/run.sh) and all unit test are described in [`test/*.js`](https://github.com/mathiasrw/rexreplace/tree/main/test). After `git clone`'ing the repo and `npm install`'ing you can invoke them with:
```bash
> npm test
```
## Performance
_**tl;dr**:_
_Files over 5 Mb are faster with `rr` than with `sed` - but - it does not matter as any file under 25 Mb has less than 0.7 seconds in difference._
The speed test is initiated by `npm run test-speed`. The test takes files in different sizes and compares the processing time for RexReplace (`rr`) and the Unix tool `sed`. The test uses the sources of a website displaying [the book _1984_ by George Orwell](http://1984.surge.sh). The task for the tests is to remove all HTML tags by search-and-replace, so only the final text is left. The source is 888Kb, so all files up to 500Kb are generated directly from the source, while larger files are created by combining the first 500Kb several times. Each test runs 10 times to even out any temporary workload fluctuations.
Results from the latest speed test can always be seen in the [speed test log](https://github.com/mathiasrw/rexreplace/blob/main/test/speed/testlog.speed.md).
The graph visualises speed as relative to fastest overall run (`sed` on a 1kb file). This chart also has [an interactive version in log scale](https://plot.ly/~mathiasrw/1.embed), so the details in the low end can be studied better. Interestingly files of 1Kb, 5Kb takes longer for `rr` than 10Kb files.
Now, what is relevant to notice is how `sed` only takes 4.9 seconds longer for the 100Mb file - even if the difference looks drastic on the graph.
```
Speed relative to fastest tool for each file size
---------------------------------------------------
Bytes sed rr Diff (seconds)
1 1 39 0,5 <= sed is 39x faster
5 1 32 0,4
10 1 27 0,4
100 1 19 0,3
500 1 7 0,3
1000 1 4 0,3
5000 1 1 0,0 <= same speed for 5Mb file
10000 2 1 0,3
25000 2 1 1,1
50000 3 1 3,1
100000 3 1 4,9 <= rr 3.1.0 is 3x faster
```
So even though the speed evolves very differently, there is only little practical use of the focus on speed for most use cases. Replacing in 10.000 small files? Use RexReplace and get yourself a cup of coffee - or spend half an hour getting `sed` to work as you want it to and enjoy the thrilling few seconds it takes to do its magic.
Please note that speeds might look very different when files get as large as the memory available.
### Tips and Tricks for Performance
Reading many files multiple times is far more time-consuming than creating a complex regex and reading each file once.
> **Anecdote time**
>
> Imagine you need to duplicate a set of files, but within the content are references (let call them keys) that must be unique across both old and new files. We have the complete list of keys so a reasoable approach to append `_v2` to each old key in the new files by running a separate `rexreplace` command for each key.
>
> However, in a real-world scenario with 5,000 keys across 10,000 files, this approach took **57 minutes**. The bottleneck was identifying the 10,000 files 5,000 times, open + read each file 5000 times and the startup time of node x 5,000.
>
> Seeking to optimize performance we bundled all keys into a single regex search pattern like `(key1|key2|key3|...)'`. This resulted in a nasty 300KB one-line command but it reduced the runtime to **2.14 seconds**!
>
> ```sh
> # Separate commands for each key
> rexreplace key1 key1_v2 src/v2/**/*.*
> rexreplace key2 key2_v2 src/v2/**/*.*
> rexreplace key3 key3_v2 src/v2/**/*.*
> ...
> rexreplace key5000 key5000_v2 src/v2/**/*.*
> # Total time: 57 minutes
>
> # Single command for all keys
> rexreplace '(key1|key2|key3|...|key5000)' '$1_v2' src/v2/**/*.*
> # Total time: 2.14 seconds
> ```
>
> Look, not all problems can be solved like this, but the moral is stil valid: Choosing the right approach can save you a significant amount of time.
## Rumours
### Inspiration
_.oO(What should "sed" have looked like by now?)_
### Future ideas
- Default to RegExmoji lax syntax
- Support RegExmoji strict syntax
- Add support to require hjson, jsonh, yaml, ini files directly
- Test-run with info outputted about what will happen (sets -t and does not change anything)
- Let search and replace be within the names of the files (ask for overwriting. -Y = no questions)
- Let search and replace be within the path of the files (ask for overwriting. -Y = no questions)
- Let pattern and globs be piped
- Let pipe be placed into search or replace.
- Let Pattern, replacement, and globs come from a file
- Let pattern and glob be javascript code returning a string as the result
- Auto string search / replace if no regex magic is used (and verify that speed is better)
- Error != warning
- Flag for String-literal (no regex, no special chars, no escape chars) to avoid backslashes or remembering which characters needs to be escaped
- Check if https://github.com/eugeneware/replacestream is good to rely on
- Check if regex engine from spider monkey can be wrapped in something that does not need node
- Check if sd can be wrapped in a WASM module
- Set engine to RE2 https://www.npmjs.com/package/re2
- Set engine to run on streams with https://www.npmjs.com/package/replacestream
- Implement in go so that all platforms can be supported with no need for node (might be based on)
- Let https://github.com/dthree/vorpal deal with the interface? Or maybe https://www.npmjs.com/package/pretty-cli
## Related projects
Many projects are seeking to solve the same problem as RexReplace. I my oppinion they all lack the flexibility of the RexReplace CLI interface and diverse the replacement can be, but have other strong features. If our way does not suit you, we suggest you have a look at:
- [sd](https://github.com/chmln/sd) - Blazingly fast! 5.9x **faster** than RexReplace. Almost perfect. Using _"Good ol' unix philosophy to the rescue."_ for pinpointing files so not always easy. Get it via cargo for Rust.
- [replace-in-file](https://www.npmjs.com/package/replace-in-file) - same speed as RexReplace. Very good support to be used as a module for a node application. A bit cumbersome CLI notation for pinpointing files. Get via npm.
- [replace-x](https://www.npmjs.com/package/replace-x) - 3.8x slower than RexReplace. A bit cumbersome CLI notation for pinpointing files. Get it via npm.
- [replace](https://www.npmjs.com/package/replace) - 2.6x slower than RexReplace. Very chatty default output. A bit more cumbersome CLI notation for pinpointing files. Get it via npm.
---
Please note that RexReplace is an [OPEN open source software](http://open-oss.com) project.
This means that individuals making significant and valuable contributions are given commit access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
[](http://open-oss.com)
_Icon inspired by Freepik from www.flaticon.com_
## License
[](https://app.fossa.io/projects/git%2Bgithub.com%2Fmathiasrw%2Frexreplace?ref=badge_large)