Use sed to modify a line in a file, *without* duplicating thefile

Ch. Konig wrote:

Hi there
>
I just have to modify a single line in a file. I know how to do that
using sed like:
sed "s/searchpattern/newexpression/" oldfile newfile
>
My "problem" is that I always have to duplicate that file, i.e. I have
to follow it by sth like
/bin/mv -f newfile oldfile
>
Firstly, my files can be quite large, so that can become heavy.
Secondly, I sometimes have to make several changes, so either I make
many files (a new temp file for each change) or I do a mv -f after
each change. (I know I can do several changes with one sed command,
but it's in a script, so I find it easier to do them individually.)
>
I feel there must be a more direct way to do that, but I haven't found
it. Thanks a lot for any advice.

GNU sed has an option -i, which lets you modify the file "in place" (of
course it creates a temp file, but it's all handled by sed and you don't
have to do anything.
Then, using a single sed script to do many changes at once, albeit seemingly
more difficult (which it actually isn't), is much more efficient than
running sed many times, each time making a pass over the entire file.

--
echo 0|sed 's909=#3u)o19;s0#0ooo)];s()(0bu}=(;s#}#.1m"?0^2{#;
s)")9v2@3%"9$);so%op]t(p$e#!o;sz(z^+.z;su+ur!z"au;sxzxd?_{h)cx;:b;
s/\(\(.\).\)\(\(\)*\)\(\(.\).\)\(\(\)\6.*\2.*\)/\5\3\1\7/;
tb'|awk '{while((i+=2)<=length($1)-18)a=a substr($1,i,1);print a}'

Ch. Konig wrote:

Now that's a tease. I thought sed would be obscure enough, but you
through ed at me. Google is astonishingly silent on the whole ed
script front, or there are too many Ed's (people of that name) out
there diluting the results. Back to the subject.
>
I played around with writing some edscripts. The problem with output
from 'diff -e' is that it's line based, while I'm looking for a
pattern (in my case a variable name) to replace (resp. to change its
assigned value). The second bother is that ed needs scripts with
several lines (true?) while I want to keep it a one-liner.
>
What works now is this:
>
(echo ",s/par1/par1\ =\ 7.00e+04/"; echo w) | ed -s mod2
>
This will look for where par1 is assigned (there's random number of
spaces in between and some assigned value after) and replace it with
an assignment of some value that I choose (7.00e+04 in this case).
>
Not extremely elegant, but it does the trick and modifies the file in-
place.

Please note that, for global modifications (ie, the "," address), ed scans
the whole file for each command, so if you have many changes to do it's very
inefficient and almost equivalent to running sed many times over the same
file. As an extreme example, consider a ~1000000 lines file, where you want
to perform the following substitutions:

s/foo/bar/g
s/baz/blah/g

With sed:

$ time sed -i 's/foo/bar/g;s/baz/blah/g' bigfile

real 0m14.437s
user 0m5.700s
sys 0m8.730s


With ed:

$ time (echo ',s/foo/bar/g'; echo ',s/baz/blah/g'; echo 'w') | ed -s bigfile

real 0m31.734s
user 0m12.600s
sys 0m19.050s

You should get similar result in terms of relative timings.

Furthermore, ed's man page states that, unlike in sed, "it is an error if
the substitution fails on every addressed line", so you get "?" output in
that case:

$ cat file2
1
2
3
$ (echo ',s/foo/bar/g'; echo ',s/baz/blah/g'; echo 'w') | ed -s file2
?
?
$

(not mentioning that, at least on my system, ed DES create a temp file
anyway, just like sed -i, although ed immediately unlinks the file so that
it's not visible).

a side note, awk seems to be *really*, *really* damn faster than either
sed or ed, at least in this particular test case:

$ time awk '{gsub(/foo/,"bar");gsub(/baz/,"blah")}1' bigfile newfile

real 0m2.923s
user 0m2.480s
sys 0m0.430s

--
echo 0|sed 's909=#3u)o19;s0#0ooo)];s()(0bu}=(;s#}#.1m"?0^2{#;
s)")9v2@3%"9$);so%op]t(p$e#!o;sz(z^+.z;su+ur!z"au;sxzxd?_{h)cx;:b;
s/\(\(.\).\)\(\(\)*\)\(\(.\).\)\(\(\)\6.*\2.*\)/\5\3\1\7/;
tb'|awk '{while((i+=2)<=length($1)-18)a=a substr($1,i,1);print a}'

Ch. Konig wrote:
>
Can I use awk "in place"?

No, not in the way you think of.

My first trials (having 'bigfile bigfile' at the end of your script)
swalled up the whole file.

Never do that! Any program started in a shell by

any_program bigfile bigfile

will let the shell rewrite bigfile before any_program is started
to operate on bigfile which is provided as argument.

And: Does that now even matter considering even ed makes an
(invisible) copy? I.e. does any change require enough space for
duplication?

Not sure what you ask here.

It's certainly possible to make a binary or textual "patch" on a
file on disk without copying the file. But you are asking for a
substitution where in general the length of the replacement string
will differ from the replaced string. The safe approach is copying
the file before you work on it or creating a new file and moving
it to the name of the original file in case no error occurred

any_program bigfile tmpfile && mv tmpfile bigfile

You can, of course, put that code pattern in a wrapper script so
that you don't "see" the implicit temporary.

Janis

>
CK

Ch. Konig wrote:

>$ time awk '{gsub(/foo/,"bar");gsub(/baz/,"blah")}1' bigfile newfile
>>
>real 0m2.923s
>user 0m2.480s
>sys 0m0.430s
>>
>
Can I use awk "in place"?
>
My first trials (having 'bigfile bigfile' at the end of your script)
swalled up the whole file.

No, you can't do that with the shell, unless - *not recommended* - you play
nasty tricks - like for instance unlinking the original file and writing to
it through another descriptor. An example of such questionable practice is
doing something like

(rm file; sed file) < file

Note, however, that that command still does NT truly edit "in place". There
are still two files involved.

Generally speaking, you can however "slurp" the whole file in memory, do
what you want with the data, and then write back the whole lot to a file
with the same name. This of course comes at the price of using much memory.

With awk, you can add some code to do that, but note that the resulting
script is slower than the original (although still faster than sed):

$ time awk '{gsub(/foo/,"bar");gsub(/baz/,"blah");a[NR]=$0}
END {close(FILENAME);for(i=1;i<=NR;i++) print a[i]>FILENAME}' bigfile

real 0m8.471s
user 0m7.690s
sys 0m0.710s

And: Does that now even matter considering even ed makes an
(invisible) copy? I.e. does any change require enough space for
duplication?

Yes. The main problems with true "in place" modification are:

- what happens if something goes wrong and the operation is interrupted
halfway? (for instance in the sed example above)
- what happens if you replace a string with a longer/shorter string?

Many editors (both interactive and non-interactive), to simulate "in place"
editing, create a temporary file.

--
echo 0|sed 's909=#3u)o19;s0#0ooo)];s()(0bu}=(;s#}#.1m"?0^2{#;
s)")9v2@3%"9$);so%op]t(p$e#!o;sz(z^+.z;su+ur!z"au;sxzxd?_{h)cx;:b;
s/\(\(.\).\)\(\(\)*\)\(\(.\).\)\(\(\)\6.*\2.*\)/\5\3\1\7/;
tb'|awk '{while((i+=2)<=length($1)-18)a=a substr($1,i,1);print a}'