Ripgrep: Add support for replace in files

I don't think I'd be willing to do this. It would change rg from a tool that will never modify your files to one that will modify your files. It is also a very complicated feature to get right to make sure you don't really mess up files on disk.

Just use sed.

Real search and replace would be an absolutely excellent feature.

But I can well believe it would be painful and complex to implement.

Perhaps there could be a reasonable middle way: if rg could output in .patch file based on the replace argument the user could then review the patch and apply it with they're happy.

Something like:

rg '(foo) (bar)' --replace '$2 $1' --diff-output > rg.diff
cat rg.diff
...
patch < rg.diff

That has the advantage that the user couldn't easily mess up a directory by happening to add an argument and the business of modifying files is left to another command.

@samuelcolvin Why would someone do that over using sed?

I guess your suggestion does satisfy one of my primary concerns, but it doesn't seem like good UX to me.

Because for me (and lots of other developers) sed is a complete anathema, it's not at all easy to get started with or reason with and it's documentation doesn't help much. I'm also not sure how well it deals with unicode.

Most people seem to open up sublime or atom or their IDE when they want to do a search and replace. For me that says a lot about the dearth of decent tools in the terminal for this.

I agree the UX isn't great but since some kind of review step is likely to be required anyway, it couldn't be that much more succinct. If you know what you want you could of course pipe the output of rg straight into patch.

(by the way, I'm a big fan of rg, sorry to sound like I'm demanding lots highly complex features. The current tool is already great.)

@samuelcolvin Thanks for explaining. I can appreciate that sed might be an opaque tool, however, its functionality is vast and I really don't want to start down the path of being a sed replacement. The patch idea is clever, but I don't think it's a particularly good fit for ripgrep.

With all that said, I am actively working towards moving more of ripgrep out into libraries (a lot of it already is), so if someone wanted to build a sed replacement down the line using the same components that make ripgrep work, I think that would actually be a feasible thing to do.

Sorry if this is a stupid question, but what's the use case for --replace if it has no way to provide actionable output?

Why is the output of ripgrep not actionable? It could be used to create a new file, for example. It could also be used to pipe input into other line-oriented command tools.

nvm, it looks good, now that i actually tried it.

sorry for the noise :)

How about an option to print out the whole file? Than I can simply pipe it over the file if I wish to.

That would make this easier: rg -C 9999999 -N '(57ed11b0add205adee02a9bd)' --replace '\'$1\'' file.js | sponge file.js

This can also work for multiple files:

for f in (rg -l '(57ed11b0add205adee02a9bd)')
  rg -C 9999999 -N '(57ed11b0add205adee02a9bd)' --replace '\'$1\'' $f | sponge $f
end

(that's fish, not bash, but differences should be subtle)

by the way this was easier to come up with than googling and reading sed tutorials, I tried...

I like @samuelcolvin suggestion on producing diffs, looks like a good compromise.

The problem using sed is that we've to search inside the file again, so rg is only useful to scan the files _quickly_. For instance, redacting a bunch of IPs from files results in a quite cumbersome expression:

rg -e "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" -c | \
  cut -d":" -f1 | \
   xargs sed -b -i "s/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/REDACTED/g"

In any case, knowing that rg has already opened the file and can do replacements with --replace I don't fully understand the negative of adding an option to make inline replaces (I might be omitting implementation reasons, probably).

Let me say this: IMO, the cumbersomeness of sed is not a reason for rg to support basic replacement. The cumbersomeness of sed is a reason to _go out and build a replacement for sed itself_. We've already covered the point that some people find sed hard to use. I understand and acknowledge that, but _I do not want to get into the business of replacing sed at the point in time_. Adding simple inline replacements is going to open the flood gates for features requests like, "sed supports doing X, please add it to ripgrep." I do not want to open those flood gates.

In any case, knowing that rg has already opened the file and can do replacements with --replace I don't fully understand the negative of adding an option to make inline replaces (I might be omitting implementation reasons, probably).

rg is a search tool that will never touch your files. Inline replacements means it becomes a search tool that will sometimes mutate your files. This is a huge change because rg now has to be very careful when it starts mutating files. It's a _completely different implementation path_ than searching and requires being absolutely sure that you don't mess up the end user's data. (What happens if the end user Ctrl-C's in the middle of writing a file?)

I, as the maintainer of this project, do not want to go down this road. I admit that the patch idea is a clever compromise, but I find the the UX to be very bad personally. It also doesn't stop the floodgates from opening to start supporting every nook and cranny of sed itself.

I suggest we drop this and wait for the components of ripgrep to get put into a library so that someone can build a better sed.

I agree with your point, making a sed replacement is a different business. However, the --replace flag suggest a different thing, the purpose is a bit confusing.

Better to wait for libripgrep, is that happening in a branch or another repo? I'm interested to contribute to it.

I second the statement that, given this, --replace is confusing.

I found this thread trying to find out how to 'accept' the changes, after realising it output what I thought was a confirmation.

I'm sure it's possible to pipe line numbers and changes to some other utility to apply the replacement, but I agree that it's non-obvious, and it would be good to be able to say in rg --help | rg -A2 replace that this can be done "by piping to command".

Better to wait for libripgrep, is that happening in a branch or another repo? I'm interested to contribute to it.

A lot of it is already done, but there is no canonical "libripgrep." I wrote up more details in #162.

I've updated the documentation of --replace.

@BurntSushi Would you be open to PRs implementing this functionality?

@ticki Not really. I don't think any of my objections to this feature were related to my personal unwillingness to implement it initially.

I am making good progress on #162. I'd rather see this functionality built out in a separate tool. My hope is that #162 makes this actually feasible to do without becoming an expert in text search.

I made a little bash script for using ripgrep as a command-line find-and-replace tool. It might be terrible. But it works great for my use cases so far. (Note: You need to replace (e with (rg on line 21.)

The idea is to first search and see what matches:

rg something

Then, press the Up arrow and edit in a replacement:

rg something --replace SOMETHING

If it looks good, press the Up arrow again and add a w at the beginning (w for write) (although my script is called pre, not wrg for historical reasons):

wrg something --replace SOMETHING

Voila! The changes should be written to disk.

(I'm not saying that ripgrep should add a feature for "writing the replacements" – I'm just sharing that a couple of lines of bash can be enough :) )

(Edit: macOS support thanks to the next comment.)

For those on MacOSX who try out lydell's bash script solution, which does not support "head -n -1" (get all except the last line); you can replace "head -n -1" with " sed '$d' " (take the input and delete the last line).

Thank you BurntSushi for ripgrep, and lydell for the simple solution.

This is more easily accomplished using sed than I think people realize. sed is quite a beast and is not very user friendly, but simple regex find and replace is easier than one may think. To perform find and replace in a directory (using ripgrep to find files with matches) you can do something akin to the following:

(GNU sed)

rg 'foo' --files-with-matches | xargs sed -i 's/foo/bar/g'

(BSD sed) <-- this includes OSx

rg 'foo' --files-with-matches | xargs sed -i '' 's/foo/bar/g'

(keep in mind here that foo can be a regex, as you would expect for ripgrep)

I'll break down what is happening here:

rg 'foo' looks for matches with foo. By using the --files-with-matches flag, we only print out the filenames for files that have matches, but none of the actual matches themselves. We pipe this output (which is a list of matching filenames) into xargs which essentially splits our list of filenames from standard in into individual command line arguments for the next program we execute. This program happens to be sed. The -i flag to sed tells sed to edit files in place. On BSD sed, we need to use -i '' to specify that we do not want file backups (the empty string indicates a lack of backup extension i.e. no backups). Lastly 's/foo/bar/g indicates we want to perform a substitution, replacing matches of the pattern foo with bar, and that we want to perform this substitution globally, i.e. for all matches in the file.

This method works very well for me, fits in the UNIX philosophy, and avoids using any sort of extra shell script. For people googling "ripgrep search and replace", hopefully this will help you use some basic sed to accomplish your goals!

@z2oh Thanks for that write up! I would be delighted if you'd be willing to contribute that to the FAQ. :-)

If you don't like sed (because of the command syntax or its regex features) you can also use perl -i, which works quite similarly but is more powerful and probably more intuitive.

sed and perl do have issues that you need to be aware of, though. Despite -i being described as an 'in-place' modification, what it actually does is delete and replace the whole file. This means that it'll have a new inode, so any open handles to the old file will be out of date, and they will both destroy symlinks, hard links, file-system attributes and flags (chattr/chflags), extended attributes (including POSIX ACLs and Spotlight meta-data), and in some cases file permissions. No idea what kind of shenanigans you might have to deal with on Windows.

I'm guessing that the difficulty of handling cases like that in a robust manner is one of the reasons doing this in rg isn't an exciting prospect, and maybe also a reason there isn't already a dedicated recursive find/replace tool that everyone uses.

sed sure is a competent tool. Just in case somebody gets confused what the differences between the sed way and my script are:

With sed you need to provide one regex to rg and one (probably the same) to sed (beware of regex differences between the two!) My script: Just one regex (passed to rg).

sed workflow:

1. rg 'foo'. Adjust the search regex until you find what you need.
2. rg 'foo' --files-with-matches | xargs sed -i 's/foo/bar/g'.
3. Hold your breath and hope you got that replacement right.

My script:

1. rg 'foo'. Adjust the search regex until you find what you need.
2. rg 'foo' -r 'bar'. See if the replacements look good.
3. wrg 'foo' -r 'bar'. Profit.

Not trying to promote my script or say that one is better than the other. But something to keep in mind :)

Edited sed workflow thanks to the tip in the next comment:

1. rg 'foo'. Adjust the search regex until you find what you need.
2. rg 'foo' --files-with-matches | xargs sed 's/foo/bar/g'. Try to spot if your replacements look good.
3. rg 'foo' --files-with-matches | xargs sed -i 's/foo/bar/g'. Profit. (Notice the added -i flag.)

The reason I say “_try_ to spot if your replacements look good” is because sed prints the _entire_ files, while rg -r only prints the relevant lines with the changes highlighted in color.

@lydell I do like your solution, but it's a little disingenuous to say:

Hold your breath and hope you got that replacement right.

the equivalent sed steps would be 2. without -i, and then 3. with it, analogously to 2. rg; 3. wrg.

Anyone found a Windows solution without installing Unix utils?

Edit: there is Goreplace by @piranha, which helped me as follows gr "night is dark" -r "day is light"

I would like a ripgrep-like sed replacement because I don't want to mentally switch between ripgrep-style and sed-style regex syntax.

I wrote a bash script that uses ripgrep to find the matching files, bash to loop over them, and ripgrep with the --passthru option to do the actual replacement it. It requires sponge from moreutils.

https://gist.github.com/frangio/462d5563d88a2982b6c23e6d2e72e93c

@frangio how about https://github.com/chmln/sd?

In case it's of use to anyone, here's a Python script I wrote that's a thin wrapper around rg --replace. You can accept or reject changes individually using the --ask flag. It also accepts all of ripgrep's flags.

https://github.com/hauntsaninja/rg-sed

I also wrote a wrapper in NodeJS that shows the ripgrep output and then asks to apply them all or separately:

https://github.com/CantGetRight82/rgw

Can we reopen this? sed workaround is cross-platform compatible due to gnu/bsd differences so it would be much better to have a solution that does not rely on external dependencies.

No. Feel free to go build one or use any of the numerous projects linked above or in the FAQ.

ned is grep with replace/sed over multiple lines... https://github.com/nevdelap/ned#replace

I started it, for my own use, before ripgrep existed, and for search ripgrep is probably much better (I don't use it because ned does what I need), but the point of ned is to make it easy to do bulk edits.

I mention at the top of the TL;DR that if you are just doing searches to use ripgrep. https://github.com/nevdelap/ned#tldr

I actually really like the patch format idea and was going to suggest it until I found this thread. I have been using something similar to the sed/perl solution above since before RipGrep was released (I used ack before I switched to RipGrep`).

I'd be happy to write a tool to convert RipGrep's output to the patch format, but I'm not sure that's possible as-is, since patch includes the original line as well as the replacement, but rg --replace doesn't print out the pre-replacement line. @BurntSushi would you be open to a PR with a flag for use in conjunction with --replace that would include the pre-replacement text as well?

The main reasons I'd prefer a patch file are because, as mentioned above, it would be nice to be able to review a large-scale change before applying it, and because patches are easily revertible, whereas reversing the search-and-replace is not (in case the replacement string already existed anywhere in the directory).

One other major benefit is that patchfiles can be used as-is with multiple other tools (such as Git). It's also possible that using multiple regex tools may eventually reveal incompatibilities between them, and that a patch-based approach would be faster than re-running a complex regex twice on every line of the files that will be modified.

@BatmanAoD I think it would be better to build a dedicated tool for such functionality. It sounds exactly like the kind of feature that will beget more features, and I'm not that interested in maintaining it.

@lydell Here is my interactive version of your script:

rgr () {
    if [ $# -lt 2 ]
    then
        echo "rg with interactive text replacement"
        echo "Usage: rgr text replacement-text"
        return
    fi
    vim --clean -c ":execute ':argdo %s%$1%$2%gc | update' | :q" -- $(rg $1 -l ${@:3})
}

You can also reuse your rg args, e.g:

$ rg foo ./my-dir/ -t py
$ rgr foo bar ./my-dir/ -t py

It looks like ruplacer has output in patch format.