Python Fast I/o

[Ayushi Dalmia]

I need to write into a file for a project which will be evaluated on the basis of time. What is the fastest way to write 200 Mb of data, accumulated as a list into a file.

Presently I am using this:

with open('index.txt','w') as f:
f.write("".join(data))
f.close()

where data is a list of strings which I want to dump into the index.txt file

 

[Chris Angelico]

I need to write into a file for a project which will be evaluated on the basis of time. What is the fastest way to write 200 Mb of data, accumulated as a list into a file.

Presently I am using this:

with open('index.txt','w') as f:
f.write("".join(data))
f.close()

with open('index.txt','w') as f:
for hunk in data:
f.write(hunk)

You don't need to f.close() - that's what the 'with' block guarantees.
Iterating over data and writing each block separately means you don't
have to first build up a 200MB string. After that, your performance is
going to be mainly tied to the speed of your disk, not anything that
Python can affect.

ChrisA

 

[Tim Chase]

I need to write into a file for a project which will be evaluated
on the basis of time. What is the fastest way to write 200 Mb of
data, accumulated as a list into a file.

Presently I am using this:

with open('index.txt','w') as f:
f.write("".join(data))
f.close()

where data is a list of strings which I want to dump into the
index.txt file --

Most file-like objects should support a writelines() method which
takes an iterable and should save you the trouble of joining all the
content (and as Chris noted, you don't need the .close() since the
with handles it) so the whole thing would condense to:

with open('index.txt', 'w') as f:
f.writelines(data)

-tkc

 

[Ayushi Dalmia]

with open('index.txt','w') as f:

for hunk in data:

f.write(hunk)



You don't need to f.close() - that's what the 'with' block guarantees.

Iterating over data and writing each block separately means you don't

have to first build up a 200MB string. After that, your performance is

going to be mainly tied to the speed of your disk, not anything that

Python can affect.



ChrisA

Thanks for the tip on the closing of the file. I did not know that with ensures closing of the file after iteration is over.

Which is more fast?
Creating a 200 Mb string and then dumping into a file or dividing the 200 Mb string into chunks and then writing those chunks. Won't writing the chunks call more i/o operation?

 

[Chris Angelico]

Your quoted text is becoming double spaced, because of bugs in the
Google Groups client. Please either edit this before posting, or
switch to a better newsreader, or use the mailing list:

https://mail.python.org/mailman/listinfo/python-list

Thanks!

Which is more fast?
Creating a 200 Mb string and then dumping into a file or dividing the 200 Mb string into chunks and then writing those chunks. Won't writing the chunks call more i/o operation?

When you're writing two hundred megabytes, the number of I/O
operations is dominated by that. You have to write that many sectors,
and nothing can change that. Joining your list of strings before
writing incurs the cost of building up a single 200MB string, but even
that is likely to be insignificant in the scheme of things (if you
have the memory available, it won't take very long compared to the
time it takes to write to the disk). Python will buffer its writes, so
you don't have to worry about the details. It's going to do the right
thing for you; you can concentrate on making your code look right.

ChrisA

 

[Roy Smith]

Which is more fast?
Creating a 200 Mb string and then dumping into a file or dividing the 200 Mb
string into chunks and then writing those chunks. Won't writing the chunks
call more i/o operation?

This sounds like a simple experiment to try. Write it both ways, time
each one, and report your results back to the group.