replacing single line of text

[saad82]

I want to be able to replace a single line in a large text file
(several hundred MB). Using the cookbook's method (below) works but I
think the replace fxn chokes on such a large chunk of text. For now, I
simply want to replace the 1st line (CSV header) in the file but I'd
also like to know a more general solution for any line in the file.
There's got a be quick and dirty (and cheap) way to do this... any
help?

Cookbook's method:
output_file.write(input_file.read().replace(stext, rtext))

Thanks,
Pythonner

 

[Tim Chase]

I want to be able to replace a single line in a large text file

(several hundred MB). Using the cookbook's method (below) works but I
think the replace fxn chokes on such a large chunk of text. For now, I
simply want to replace the 1st line (CSV header) in the file but I'd
also like to know a more general solution for any line in the file.
There's got a be quick and dirty (and cheap) way to do this... any
help?

Cookbook's method:
output_file.write(input_file.read().replace(stext, rtext))

The read() method slurps the entire file into memory...bad in the
case you describe.

What you want is to process the file as a stream:

first = True
out = file("out.txt", "w")
for line in file("foo.txt"):
if first:
out.write(line.replace(stext, rtext))
first = False
else:
out.write(line)


If you're using an older version of Python, I don't know if
file() returns an iterator, so you might have to do something like

for line in file("foo.txt").xreadlines():

which I understand is the same sort of thing. I don't know the
history of this too well--others on the list are likely better
versed in such peculariaties.

Or, for those *nix wonks in the crowd (self included), sed will
do the trick nicely:

sed '1s/stext/rtext/' foo.txt > out.txt

Just a few ideas,

-tkc

 

[Simon Forman]

I want to be able to replace a single line in a large text file
(several hundred MB). Using the cookbook's method (below) works but I
think the replace fxn chokes on such a large chunk of text. For now, I
simply want to replace the 1st line (CSV header) in the file but I'd
also like to know a more general solution for any line in the file.
There's got a be quick and dirty (and cheap) way to do this... any
help?

Cookbook's method:
output_file.write(input_file.read().replace(stext, rtext))

Thanks,
Pythonner

The read() method of a file will "read all data until EOF is reached"
if you don't pass it a size argument. If your file is "several hundred
MB" and your RAM is not, you may have some trouble with this.

I don't know how well the replace() method works on tenths-of-a-GB
strings.

The file object supports line-by-line reads through both the readline()
and readlines() methods, but be aware that the readlines() method will
also try to read ALL the data into memory at once unless you pass it a
size argument.

You can also iterate through the lines in an open file object like so:

for line in input_file:
# Do something with the line here.

so, if you know that you want to replace a whole actual line, you could
do this like so:

for line in input_file:
if line == stext:
output_file.write(rtext)
else:
output_file.write(line)


Check out the docs on the file object for more info:
http://docs.python.org/lib/bltin-file-objects.html


HTH,
~Simon

 

[saad82]

Simon & Tim, very valuable responses. Thank you much!