Best way to write a file n-bytes long

[Tony C]

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()



thanks

 

[Sybren Stuvel]

Tony C enlightened us with:

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

Why not create a string that is 1 KB long and write that n*1024 times?

Sybren

 

[Raymond Hettinger]

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

The C coded string methods are not slow.
In Py2.3, 'x' * n calls str.__mul__ which is implemented
using the C library's memset() function.

It is faster still to use itertools.repeat:

st = itertools.repeat('X', 1048576)

Of course, your particular use case is I/O bound
(meaning that string construction isn't the
cause of your performance issues).

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

Raymond Hettinger

 

[Daniel Klein]

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

Pythons file objects are automatically treated as streams. So you could do
something like this:

st = 'X'
somebignumber = 1048576
fh = open('junk','wb')
for n in xrange(somebignumber):
fh.write(st)
fh.close()

Daniel Klein

 

[Michael Porter]

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()



thanks

Another alternative is to seek to required position and write (at least) one
byte...

reqSize = 1048576
fh = open('junk', 'wb')
fh.seek(reqSize - 1)
fh.write('\0')
fh.close()

Mike.

 

[Greg Brunet]

Another alternative is to seek to required position and write (at least) one
byte...

reqSize = 1048576
fh = open('junk', 'wb')
fh.seek(reqSize - 1)
fh.write('\0')
fh.close()

Mike.

I interpreted his 'arbitrary data' to mean the same thing & came up with
the same solution (which should be the fastest way to do it in C as
well!). Anyway, doing some rough timing also seems to show that
creating the string was only taking about 10% of the time that writing
it is, and that the seek solution is about 10x faster that creating the
string & 100x faster that writing the string! It shows once again that
it pays to find out where the bottleneck is before trying to optimize
the wrong area!

 

[Michael Peuser]

(e-mail address removed) (Tony C) writes:
[...]

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

For little files (less than 2 or 3 Mb I think) your code is the
fastest I can think of. But growing There is a new version which is a
lot faster

All this is valid in a very limited scope only.
- Consider limited RAM (lets say 128 MB) - you will be extremely slowed down
by thrashing.
- Consider on-the-fly compression (Windows NTFS): The 1000 MB testdate will
be reduced to 1 Byte!

Kindly
Michael P