Best way to write a file n-bytes long

Discussion in 'Python' started by Tony C, Aug 26, 2003.

  1. Tony C

    Tony C Guest

    Does Python have a function that is analogous to C's write() or
    fwrite()-

    that is , I want to write a file (of arbitrary data) that is 100K, or
    1MB (or more) bytes long..

    Both write() and fwrite() in C allow the user to specify the size of
    the data to be written.

    Python's write only allows a string to be passed in.

    Sure, I could create a string that is n Megabytes long, and pass it to
    write, but it seems as though there should be a better way ???
    String manipulation is typically considered slow.

    st="X" * 1048576
    fh=open("junk","wb")
    fh.write(st)
    fh.close()



    thanks
    Tony C, Aug 26, 2003
    #1
    1. Advertising

  2. Tony C enlightened us with:
    > Sure, I could create a string that is n Megabytes long, and pass it to
    > write, but it seems as though there should be a better way ???
    > String manipulation is typically considered slow.


    Why not create a string that is 1 KB long and write that n*1024 times?

    Sybren
    --
    The problem with the world is stupidity. Not saying there should be a
    capital punishment for stupidity, but why don't we just take the
    safety labels off of everything and let the problem solve itself?
    Sybren Stuvel, Aug 26, 2003
    #2
    1. Advertising

  3. "Tony C" <> wrote in message
    news:...
    > Does Python have a function that is analogous to C's write() or
    > fwrite()-
    >
    > that is , I want to write a file (of arbitrary data) that is 100K, or
    > 1MB (or more) bytes long..
    >
    > Both write() and fwrite() in C allow the user to specify the size of
    > the data to be written.
    >
    > Python's write only allows a string to be passed in.
    >
    > Sure, I could create a string that is n Megabytes long, and pass it to
    > write, but it seems as though there should be a better way ???
    > String manipulation is typically considered slow.


    The C coded string methods are not slow.
    In Py2.3, 'x' * n calls str.__mul__ which is implemented
    using the C library's memset() function.

    It is faster still to use itertools.repeat:

    st = itertools.repeat('X', 1048576)

    Of course, your particular use case is I/O bound
    (meaning that string construction isn't the
    cause of your performance issues).

    > st="X" * 1048576
    > fh=open("junk","wb")
    > fh.write(st)
    > fh.close()



    Raymond Hettinger
    Raymond Hettinger, Aug 26, 2003
    #3
  4. Tony C

    Daniel Klein Guest

    On 26 Aug 2003 15:04:12 -0700, (Tony C) wrote:

    >Does Python have a function that is analogous to C's write() or
    >fwrite()-
    >
    >that is , I want to write a file (of arbitrary data) that is 100K, or
    >1MB (or more) bytes long..
    >
    >Both write() and fwrite() in C allow the user to specify the size of
    >the data to be written.
    >
    >Python's write only allows a string to be passed in.
    >
    >Sure, I could create a string that is n Megabytes long, and pass it to
    >write, but it seems as though there should be a better way ???
    >String manipulation is typically considered slow.
    >
    >st="X" * 1048576
    >fh=open("junk","wb")
    >fh.write(st)
    >fh.close()


    Pythons file objects are automatically treated as streams. So you could do
    something like this:

    st = 'X'
    somebignumber = 1048576
    fh = open('junk','wb')
    for n in xrange(somebignumber):
    fh.write(st)
    fh.close()

    Daniel Klein
    Daniel Klein, Aug 26, 2003
    #4
  5. "Tony C" <> wrote in message
    news:...
    > Does Python have a function that is analogous to C's write() or
    > fwrite()-
    >
    > that is , I want to write a file (of arbitrary data) that is 100K, or
    > 1MB (or more) bytes long..
    >
    > Both write() and fwrite() in C allow the user to specify the size of
    > the data to be written.
    >
    > Python's write only allows a string to be passed in.
    >
    > Sure, I could create a string that is n Megabytes long, and pass it to
    > write, but it seems as though there should be a better way ???
    > String manipulation is typically considered slow.
    >
    > st="X" * 1048576
    > fh=open("junk","wb")
    > fh.write(st)
    > fh.close()
    >
    >
    >
    > thanks


    Another alternative is to seek to required position and write (at least) one
    byte...

    reqSize = 1048576
    fh = open('junk', 'wb')
    fh.seek(reqSize - 1)
    fh.write('\0')
    fh.close()

    Mike.
    Michael Porter, Aug 27, 2003
    #5
  6. Tony C

    Greg Brunet Guest

    "Michael Porter" <> wrote in message
    news:3f4c8ebc$0$256$...
    >
    > "Tony C" <> wrote in message
    > news:...
    > > Does Python have a function that is analogous to C's write() or
    > > fwrite()-
    > >
    > > that is , I want to write a file (of arbitrary data) that is 100K,

    or
    > > 1MB (or more) bytes long..
    > >
    > > Both write() and fwrite() in C allow the user to specify the size of
    > > the data to be written.


    > Another alternative is to seek to required position and write (at

    least) one
    > byte...
    >
    > reqSize = 1048576
    > fh = open('junk', 'wb')
    > fh.seek(reqSize - 1)
    > fh.write('\0')
    > fh.close()
    >
    > Mike.


    I interpreted his 'arbitrary data' to mean the same thing & came up with
    the same solution (which should be the fastest way to do it in C as
    well!). Anyway, doing some rough timing also seems to show that
    creating the string was only taking about 10% of the time that writing
    it is, and that the seek solution is about 10x faster that creating the
    string & 100x faster that writing the string! It shows once again that
    it pays to find out where the bottleneck is before trying to optimize
    the wrong area!

    --
    Greg
    Greg Brunet, Aug 27, 2003
    #6
  7. "Dialtone" <dialtone#NOSPAM#> schrieb im Newsbeitrag
    news:...
    > (Tony C) writes:
    >

    [...]

    > > st="X" * 1048576
    > > fh=open("junk","wb")
    > > fh.write(st)
    > > fh.close()

    >
    > For little files (less than 2 or 3 Mb I think) your code is the
    > fastest I can think of. But growing There is a new version which is a
    > lot faster


    All this is valid in a very limited scope only.
    - Consider limited RAM (lets say 128 MB) - you will be extremely slowed down
    by thrashing.
    - Consider on-the-fly compression (Windows NTFS): The 1000 MB testdate will
    be reduced to 1 Byte!

    Kindly
    Michael P
    Michael Peuser, Aug 30, 2003
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Batista, Facundo

    RE: Best way to write a file n-bytes long

    Batista, Facundo, Aug 27, 2003, in forum: Python
    Replies:
    0
    Views:
    417
    Batista, Facundo
    Aug 27, 2003
  2. Yandos
    Replies:
    12
    Views:
    5,096
    Pete Becker
    Sep 15, 2005
  3. Replies:
    5
    Views:
    278
    dprody
    Nov 5, 2007
  4. scad
    Replies:
    4
    Views:
    938
    James Kanze
    May 28, 2009
  5. Steven D'Aprano
    Replies:
    8
    Views:
    388
    Helmut Jarausch
    Jan 21, 2010
Loading...

Share This Page