sed, perl, or other CLI command to find-and-replace one string in all files?

[sneakyimp]

I'm a little embarassed to admit this, but I've never learned any *nix command which can search every file in some directory and replace some string with another string and then save the file. Can anyone recommend a command to do this that is likely to work on all linux distros? Ideally this search/replace function will support regex for search and then backreferences for replacing (i.e., you can refer to the found text with $1 or \1 or whatever).

I humbly offer this one command which I have used to replace a string in a single file. It replaces all occurrences of the string

perl -p -i.bak -e 's/searchString/replaceString/gi' file-to-search.txt

As I am unfamiliar with perl, the syntax is greek to me so I typically just go dig it up from my notes and use it almost verbatim. I'm hoping to find a solution which will search an entire directory of files, ideally letting you specify filename patterns, which is similar to the patterns we use with preg_match or with the grep command. I'd also note that I'm often frustrated with grep because its pattern matching doesn't behave exactly the same way as preg_match.

[Derokorian]

I would use two comments, find and sed.

find /folder/to/search -type f -name *.txt -exec sed -e s/searchPattern/replacePattern/ -i bak {} \;

finds all txt files in the folder (or sub folders, though find does support -mindepth and -maxdepth for more control) and then passes those files to the sed command. The file name returned by find, will get placed in the -exec arg where the {} is, and we need \; to let find know that our command for -exec is done being defined. Then there's the sed command, the main thing here is the -i which means in place with an argument of an extension to move the original file to, otherwise it replaces the occurrences in the file and then spits the result to stdout. Without more specifics of what you want to do, this is the best I can give you. Both commands are very powerful and I use them very often for a wide variety of things.

[dalecosp]

While I typically encourage the use of CLI tools, you could fairly easily script yourself a PHP solution, as long as we're not talking about hundreds of subdirectories, could you not?

[sneakyimp]

Derokorian;11050319 wrote:

find /folder/to/search -type f -name *.txt -exec sed -e s/searchPattern/replacePattern/ -i bak {} \;

finds all txt files in the folder (or sub folders, though find does support -mindepth and -maxdepth for more control) and then passes those files to the sed command. The file name returned by find, will get placed in the -exec arg where the {} is, and we need \; to let find know that our command for -exec is done being defined. Then there's the sed command, the main thing here is the -i which means in place with an argument of an extension to move the original file to, otherwise it replaces the occurrences in the file and then spits the result to stdout. Without more specifics of what you want to do, this is the best I can give you. Both commands are very powerful and I use them very often for a wide variety of things.

I'm familiar with find, not so much with sed. I find it odd that the params to sed don't have any quotes around them -- that it's all one string. Any comments on what sort of regex you can put in the searchPattern and replacePattern?

dalescop wrote:

While I typically encourage the use of CLI tools, you could fairly easily script yourself a PHP solution, as long as we're not talking about hundreds of subdirectories, could you not?

This had occurred to me, but there are a few challenges:
where do i put the php file to make it available without always typing some full path to it?
this php file would not exist when I have to make changes on some new server. is deploying it difficult?
* might be some challenges in figuring out how to get my regex into PHP unmolested from its specification on the CLI. Do quotes need escaping? Do I need quotes around my regex?

[Derokorian]

Sure you can put the argument to -e in quotes if you want. It depends on what you're trying to put in the patterns for what quotes to use and such. For example, here's 2 sed commands from my deploy script:

sed -e 's#.*LoadScript.*#<script type="text/javascript" src="{Config>GetValue(website,site_url)}scripts/main-build.js"></script>#' \
        -e '/AddScript/d' -i ${DEPLOY_DIR}app/view/main.html

sed -e 's#.*LoadStyle.*#<link rel="stylesheet" type="text/css" href="{Config>GetValue(website,site_url)}styles/main-build.css">#' \
        -e '/AddStyle/d' -i ${DEPLOY_DIR}app/view/main.html

This finds a call to LoadScript (a special thing in my framework) and replaces with a specific script tag for the built javascript. It then finds where I AddScript (more fw stuff) and deletes the entire line. The second command does the same thing for styles. Note that the pattern delimiter is somewhat loose like in php.

Here's a find with sed that finds all of my css files, removes blank lines, new lines, excessive white space and block comments, this doesn't do an in place call because I pipe all the output to a single file.

find public/styles -type f -exec sed -e ':a;N;$!ba;s/\n//g' -e 's#/\*.*\*/##g' -e 's/\s\s*/ /g' {} \; > ${DEPLOY_DIR}public/styles/main-build.css

[dalecosp]

sneakyimp;11050329 wrote:

This had occurred to me, but there are a few challenges:
where do i put the php file to make it available without always typing some full path to it?
this php file would not exist when I have to make changes on some new server. is deploying it difficult?
* might be some challenges in figuring out how to get my regex into PHP unmolested from its specification on the CLI. Do quotes need escaping? Do I need quotes around my regex?

1. in your $PATH:

   echo $PATH
   /sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/home/dalecosp/bin

   Note that your path is set in a login file; for me, using tcsh, that's ~/.cshrc, if you're a bash guy that'd be slightly dependent on how the file was to be run; see here. Also note that it would have to be set executable if you just want to type "myscript" at the CLI interface, and have the shebang set. Here's a detail of a CLI script; note the permissions include "x" for executable, and note the shebang line at file top. Calling "env php" allows the system to call php regardless of where the distro places the actual executable...

ls -l ts && cat ts
-rwxr-xr-x  1 kevin  wheel  68 Jun 12 09:21 ts*
#!/usr/bin/env php -q
<?php

$ts = $argv[1];

echo date("r",$ts);

2. In accordance with #1 above, not terribly. Make the folder "bin" in your homedir, put the script there with proper permissions, edit your .rc file for your shell profile file noted above.

3. That might, indeed, be the trickiest part and would be dependent somewhat on your shell. C shell is known to be bad about args, quotes, and special characters, and it might well be the death of this idea; you might have better luck if you are a bash user like most (and that is, indeed, one of the reasons so many prefer bash).

I suppose I didn't think about your use case too well; typically, if I do this, it's because I have several "fixes" to make and want to batch it instead of sit there and bang out the same command 42 times with different params ...

[sneakyimp]

OK I'm having a bit of trouble with the find/sed command in two ways:
1) I'm not sure how to escape slashes in my find and replace strings when I try to replace one file path with another. I've had some luck with using # instead of / as my pattern delimiter, but what happens when I need to search/replace strings with both slashes and hashes (and @ and | and whatever else) ? E.g., this barfs:

find . -type f -name "*.html" -exec sed s/old/path/to/old-file.pdf/new/path/to/new-file.pdf/g -i bak {} \;
sed: -e expression #1, char 26: unknown option to `s'
sed: -e expression #1, char 26: unknown option to `s'
sed: -e expression #1, char 26: unknown option to `s'
...

It also barfs if I precede my path slashes with a backslash

2) the -i flag doesn't work. If I fix my above command using # instead of / for the sed command, I still get the error.

sed: can't read bak: No such file or directory

The replacement command appears to work and my files get changed, but no backup files are made.

[Derokorian]

sneakyimp;11050413 wrote:

OK I'm having a bit of trouble with the find/sed command in two ways:
1) I'm not sure how to escape slashes in my find and replace strings when I try to replace one file path with another. I've had some luck with using # instead of / as my pattern delimiter, but what happens when I need to search/replace strings with both slashes and hashes (and @ and | and whatever else) ? E.g., this barfs:

find . -type f -name "*.html" -exec sed s/old/path/to/old-file.pdf/new/path/to/new-file.pdf/g -i bak {} \;
sed: -e expression #1, char 26: unknown option to `s'
sed: -e expression #1, char 26: unknown option to `s'
sed: -e expression #1, char 26: unknown option to `s'
...

It also barfs if I precede my path slashes with a backslash

There's no magic here that I know of, short of writing yourself an expression generator (or something) that will pick a valid symbol for the delimiter and go from there. But I can't imagine that being an issue unless you're generating the commands automatically... in which case the question becomes why isn't whatever is generating the commands able to generate them properly? Also, note the examples I pasted, you can pass multiple -e's to sed in a single command therefore -e 's@find/with/slashes@replace@' -e 's/find@with@ats/replace' or whatever🙂 ALSO! Its regex, so you can use wild cards and what not.

sneakyimp;11050413 wrote:

2) the -i flag doesn't work. If I fix my above command using # instead of / for the sed command, I still get the error.

sed: can't read bak: No such file or directory

The replacement command appears to work and my files get changed, but no backup files are made.

Yeah this is my bad, whatever command I was working with when I wrote that, this is how that command works. If you look at my pastes though, they are working example that I rely on pretty regularly.

I'd also note that I'm often frustrated with grep because its pattern matching doesn't behave exactly the same way as preg_match.

Just noticed this from your first post but there is an argument for grep that makes it use Perl RegEx

-P, --perl-regexp PATTERN is a Perl regular expression

[emptymind]

sneakyimp;11050317 wrote:

I'm a little embarassed to admit this, but I've never learned any *nix command which can search every file in some directory and replace some string with another string and then save the file. Can anyone recommend a command to do this that is likely to work on all linux distros? Ideally this search/replace function will support regex for search and then backreferences for replacing (i.e., you can refer to the found text with $1 or \1 or whatever).

I humbly offer this one command which I have used to replace a string in a single file. It replaces all occurrences of the string

perl -p -i.bak -e 's/searchString/replaceString/gi' file-to-search.txt

As I am unfamiliar with perl, the syntax is greek to me so I typically just go dig it up from my notes and use it almost verbatim. I'm hoping to find a solution which will search an entire directory of files, ideally letting you specify filename patterns, which is similar to the patterns we use with preg_match or with the grep command. I'd also note that I'm often frustrated with grep because its pattern matching doesn't behave exactly the same way as preg_match.

It doesn't meet all of your 'best case scenario' requirements, but for simple search and replace have you looked at the 'replace' program? It comes bundled in the mysql-server packages which may not be available 'everywhere' but are quite commonplace.

I use this quite often for simple search and replace and it supports multiple 'search and replace' items in the same run.

http://linux.die.net/man/1/replace

The biggest caveat here is that you should backup your files first (but that's mostly common sense), as it does the search and replace in the active file by default.

YMMV but thought I would share. Best of luck!

[sneakyimp]

Derokorian;11050331 wrote:

Sure you can put the argument to -e in quotes if you want. It depends on what you're trying to put in the patterns for what quotes to use and such. For example, here's 2 sed commands from my deploy script:

sed -e 's#.*LoadScript.*#<script type="text/javascript" src="{Config>GetValue(website,site_url)}scripts/main-build.js"></script>#' \
        -e '/AddScript/d' -i ${DEPLOY_DIR}app/view/main.html

sed -e 's#.*LoadStyle.*#<link rel="stylesheet" type="text/css" href="{Config>GetValue(website,site_url)}styles/main-build.css">#' \
        -e '/AddStyle/d' -i ${DEPLOY_DIR}app/view/main.html

This finds a call to LoadScript (a special thing in my framework) and replaces with a specific script tag for the built javascript. It then finds where I AddScript (more fw stuff) and deletes the entire line. The second command does the same thing for styles. Note that the pattern delimiter is somewhat loose like in php.

It's a little confusing trying to sort out what's sed syntax, what's bash syntax, and what's your framework syntax. I appreciate the explanation and examples, though.

Derokorian;11050331 wrote:

Here's a find with sed that finds all of my css files, removes blank lines, new lines, excessive white space and block comments, this doesn't do an in place call because I pipe all the output to a single file.

find public/styles -type f -exec sed -e ':a;N;$!ba;s/\n//g' -e 's#/\*.*\*/##g' -e 's/\s\s*/ /g' {} \; > ${DEPLOY_DIR}public/styles/main-build.css

That sounds like a potent command! However, the ':a;N;$!ba;s/\n//g' syntax is a total mystery to me. The other ones look pretty confusing too. It's apparent that the syntax for sed commands is quite unlike preg in php. If I want to develop any facility with this, I'll obviously need to learn this alternate syntax.

Derokorian wrote:

But I can't imagine that being an issue unless you're generating the commands automatically

It's an issue inasmuch as the pattern syntax is something new to me and my attempts to escape characters with a backslash totally didn't work. Also, even situations where one manually constructs the replacement pattern can present problems. Some questions occur to me:
what characters are permissible as expression delimiters? I've seen / and @ and # -- is there any restrictions on these? Obviously quotes are unacceptable -- or at least ill-advised
In the event that one must also use the delimiter char in the expression then how do you escape it? My attempt before in preceding slashes (my delimiter) with a backslash didn't seem to work.
* what's the best way to get the hang of sed patterns? any good tutorials or quick start guides?

[sneakyimp]

dalecosp;11050365 wrote:

1. in your $PATH:

echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/home/dalecosp/bin

Note that your path is set in a login file; for me, using tcsh, that's ~/.cshrc, if you're a bash guy that'd be slightly dependent on how the file was to be run; see here. Also note that it would have to be set executable if you just want to type "myscript" at the CLI interface, and have the shebang set. Here's a detail of a CLI script; note the permissions include "x" for executable, and note the shebang line at file top. Calling "env php" allows the system to call php regardless of where the distro places the actual executable...

ls -l ts && cat ts
-rwxr-xr-x  1 kevin  wheel  68 Jun 12 09:21 ts*
#!/usr/bin/env php -q
<?php

$ts = $argv[1];

echo date("r",$ts);

Thank you so much for this detail. I've long wondered how best to go about customizing my environment in this way and this is very helpful. On the other hand, it doesn't look like something that is very portable -- I'd have to perform this configuration and deploy my scripts on every machine if I want such commands to be available.

dalecosp;11050365 wrote:

2. In accordance with #1 above, not terribly. Make the folder "bin" in your homedir, put the script there with proper permissions, edit your .rc file for your shell profile file noted above.

One might also automate this kind of operation -- or at least distill down to a single command.

dalecosp;11050365 wrote:

3. That might, indeed, be the trickiest part and would be dependent somewhat on your shell. C shell is known to be bad about args, quotes, and special characters, and it might well be the death of this idea; you might have better luck if you are a bash user like most (and that is, indeed, one of the reasons so many prefer bash).

I am indeed a bash user by default rather than by choice. You've now made me curious about why people might choose other shells.

dalecosp;11050365 wrote:

I suppose I didn't think about your use case too well; typically, if I do this, it's because I have several "fixes" to make and want to batch it instead of sit there and bang out the same command 42 times with different params ...

If I have a lot of changes to make, I always fall back on PHP. A little glob and file_get_contents and str_replace will handle almost every find/replace problem. If things get tricky, I'm pretty well-practiced with preg_replace. I've even started to kind of enjoy regex. :p

[sneakyimp]

emptymind;11050437 wrote:

It doesn't meet all of your 'best case scenario' requirements, but for simple search and replace have you looked at the 'replace' program? It comes bundled in the mysql-server packages which may not be available 'everywhere' but are quite commonplace.

I use this quite often for simple search and replace and it supports multiple 'search and replace' items in the same run.

http://linux.die.net/man/1/replace

The biggest caveat here is that you should backup your files first (but that's mostly common sense), as it does the search and replace in the active file by default.

YMMV but thought I would share. Best of luck!

Thanks also for this tip. It appears to be installed on my workstation. Will check it out.

[Derokorian]

sneakyimp;11050545 wrote:

It's a little confusing trying to sort out what's sed syntax, what's bash syntax, and what's your framework syntax. I appreciate the explanation and examples, though.

Let me see if I can't break one set of the commands to a bit of piecemeal:

sed -e 's#.*LoadStyle.*#<link rel="stylesheet" type="text/css" href="{Config>GetValue(website,site_url)}styles/main-build.css">#' \
        -e '/AddStyle/d' -i ${DEPLOY_DIR}app/view/main.html

Can be rewritten as 2 commands, and I'll remove the bash variable for simplicity:

sed -e 's#.*LoadStyle.*#<link rel="stylesheet" type="text/css" href="styles/main-build.css">#' -i /app/view/main.html
sed -e '/AddStyle/d' -i /app/view/main.html

So now its just 2 find and replaces. The first says find a line starting with anything (.) leading up to LoadStyle and with anything (.) to the end of the line. Then it rips everything that matches, and changes it to a normal link tag to load css. The second says find any line containing AddStyle and delete the whole line (the d flag does the delete). Finally -i name_of_file which means do the replacement in place, instead of printing it to stdout.

sneakyimp;11050545 wrote:

That sounds like a potent command! However, the ':a;N;$!ba;s/\n//g' syntax is a total mystery to me. The other ones look pretty confusing too. It's apparent that the syntax for sed commands is quite unlike preg in php. If I want to develop any facility with this, I'll obviously need to learn this alternate syntax.

See http://stackoverflow.com/a/1252191 the exact overflow answer I stole it from as there is a good explanation of how it works.

sneakyimp;11050545 wrote:

* what characters are permissible as expression delimiters? I've seen / and @ and # -- is there any restrictions on these? Obviously quotes are unacceptable -- or at least ill-advised

I found this, but I don't know of any exhaustive list... looks like just about anything though.

sneakyimp;11050545 wrote:

In the event that one must also use the delimiter char in the expression* then how do you escape it? My attempt before in preceding slashes (my delimiter) with a backslash didn't seem to work.

Are you using single quotes to surround your pattern? if not you need to escape the escape, similar to escaping in double quotes in php. {char} becomes an escape sequence in double quotes or without quotes in bash, so you would need to double them up (iirc). Or you can use single quotes, which means pass this literal string without any bash interpretation as an argument to the command.

sneakyimp;11050545 wrote:

* what's the best way to get the hang of sed patterns? any good tutorials or quick start guides?

Practice practice practice. Even engineers need to work out to improve their skills I can't tell you how many failed sed commands I've written before I got the ones that worked exactly as I wanted.

[sneakyimp]

Thanks for taking care to clarify your examples.