Simple file writing techiques ...

[cdecarlo]

Hello,

I've often found that I am writing little scripts at the interpretor to
read a text file, perform some conversion, and then write the converted
data back out to a file. I normally accomplish the above task by
reading the lines of the entire file into a list, preforming some
function to that list and then writing the list back out. I want to
write a generic function/module that will free me from repeatedly
typing the same thing, perhaps convertFile(filename, covertFunc) and I
was wondering what would be a more optimal solution for writing the
data,

fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()

.... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...

An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file. As well, I know from
'import this' that there should only be one obvious way to do something
.... however to me it's not so obvious

Any suggestions,

Colin

 

[Will McGugan]

fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()

I'd go for something like...

fout = open('somefile','w')
fout.writelines( line+"\n" for line in convertedData )
fout.close()

Although your first solution should perform about the same.

If you have 2.5, you may prefer this...

with open('somefile','w') as fout:
fout.writelines( line+"\n" for line in convertedData )

... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...

The OS should buffer it for you.


Will McGugan

 

[Ant]

fout = open('somefile','w')

for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()

I shouldn't think it matters too much which of these you use - time
them and see what happens.

An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file.

Sucking in the entire file into a list won't scale well, as a huge log
file could quickly eat all of your available memory. You'd be better
off processing each line as you go, and writing it to a temp file,
renaming the temp file once you have finished. Something like:

in_f = "access.log"
out_f = "access.log.tmp"

infile = open(in_f)
outfile = open(out_f)

for line in infile:
outfile.write(process(line))

infile.close()
outfile.close()

os.remove(in_f)
os.rename(out_f, in_f)

(Not tested, but you get the idea...)

 

[Ganesan Rajagopal]

fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()

... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...

The second option would be definitely faster.

An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file.

Considering that you've already read in the whole file into a list, it's too
late to worry about scalability when writing out . Have you considered
the fileinput module?

Ganesan

 

[Ant]

Whoops:

outfile = open(out_f)

outfile = open(out_f, 'w')

may be better ;-)

 

[BartlebyScrivener]

See also this recent discussion on techniques for making sure you get
the file closed, even if there is an exception.

http://tinyurl.com/fryrv

rd

 

[Simon Forman]

Hello,

I've often found that I am writing little scripts at the interpretor to
read a text file, perform some conversion, and then write the converted
data back out to a file. I normally accomplish the above task by

Any suggestions,

Colin

You should check out the fileinput module.

HTH,
~Simon