append one file to another

Discussion in 'Python' started by b83503104@yahoo.com, Jul 12, 2005.

  1. Guest

    Hi,

    I want to append one (huge) file to another (huge) file. The current
    way I'm doing it is to do something like:

    infile = open (infilename, 'r')
    filestr = infile.read()
    outfile = open(outfilename, 'a')
    outfile.write(filestr)

    I wonder if there is a more efficient way doing this?
    Thanks.
    , Jul 12, 2005
    #1
    1. Advertising

  2. Am Tue, 12 Jul 2005 06:47:50 -0700 schrieb :

    > Hi,
    >
    > I want to append one (huge) file to another (huge) file. The current
    > way I'm doing it is to do something like:
    >
    > infile = open (infilename, 'r')
    > filestr = infile.read()
    > outfile = open(outfilename, 'a')
    > outfile.write(filestr)
    >
    > I wonder if there is a more efficient way doing this?
    > Thanks.


    I guess (don't know), that this is faster:

    for line in infile:
    outfile.write(line)

    At least if this a file with "lines".

    If it is a binary file, you could read
    N bytes at once: infile.read(N)

    Thomas


    --
    Thomas G├╝ttler, http://www.thomas-guettler.de/
    Thomas Guettler, Jul 12, 2005
    #2
    1. Advertising

  3. Guest

    Thanks for the nice suggestions!

    As a side question, you mentioned opening files in binary mode, in case
    the code needs to run under Windows or cross-platform. What would
    happen otherwise? Is it an issue of big little endian or some other
    issue?
    , Jul 12, 2005
    #3
  4. On Tue, 12 Jul 2005 06:47:50 -0700, wrote:

    > Hi,
    >
    > I want to append one (huge) file to another (huge) file.


    What do you call huge? What you or I think of as huge is not necessarily
    huge to your computer.

    > The current
    > way I'm doing it is to do something like:
    >
    > infile = open (infilename, 'r')
    > filestr = infile.read()
    > outfile = open(outfilename, 'a')
    > outfile.write(filestr)
    >
    > I wonder if there is a more efficient way doing this?


    Why? Is it not working? Is it too slow? Does it crash your computer?

    If you have any expectation that you code needs to run under Windows, or
    cross-platform, or contains binary data, you should open your files in
    binary mode:

    infile = open(infilename, 'rb')
    outfile = open(outfilename, 'ab')

    For raw copying, you should probably use binary mode even if they just
    contain text. Better safe than sorry...

    Then, if you are concerned that the files really are huge, that is, as big
    or bigger than the free memory your computer has, read and write them in
    chunks:

    data = infile.read(64) # 64 bytes at a time is a bit small...
    outfile.write(data)

    Instead of 64 bytes, you should pick a more realistic figure, which will
    depend on how much free memory your computer has. I suppose a megabyte is
    probably reasonable, but you will need to experiment to find out.

    Then when you are done, close the files:

    infile.close()
    outfile.close()

    This is not strictly necessary, but it is good practice. If your program
    dies, the files may not be closed properly and you could end up losing
    data.


    --
    Steven.
    Steven D'Aprano, Jul 12, 2005
    #4
  5. Danny Nodal Guest

    Its been a while since I last coded in Python, so please make sure you test
    it before trying it so you don't clobber your existing file. Although it may
    not be more effecient than what you are doing now or has been suggested
    already, it sure cuts down on the typing.

    open(outfilename,'a').write(open(infilename).read())

    Regards.

    <> wrote in message
    news:...
    > Hi,
    >
    > I want to append one (huge) file to another (huge) file. The current
    > way I'm doing it is to do something like:
    >
    > infile = open (infilename, 'r')
    > filestr = infile.read()
    > outfile = open(outfilename, 'a')
    > outfile.write(filestr)
    >
    > I wonder if there is a more efficient way doing this?
    > Thanks.
    >
    Danny Nodal, Jul 12, 2005
    #5
  6. Dear me, replying to myself twice in one day...

    On Wed, 13 Jul 2005 00:39:14 +1000, Steven D'Aprano wrote:

    > Then, if you are concerned that the files really are huge, that is, as big
    > or bigger than the free memory your computer has, read and write them in
    > chunks:
    >
    > data = infile.read(64) # 64 bytes at a time is a bit small...
    > outfile.write(data)


    Sorry, that should be in a loop:

    data = "anything"
    while data:
    data = infile.read(64) # data will be empty when the file is read
    outfile.write(data)

    --
    Steven.
    Steven D'Aprano, Jul 12, 2005
    #6
  7. On 2005-07-12, <> wrote:

    > As a side question, you mentioned opening files in binary
    > mode, in case the code needs to run under Windows or
    > cross-platform. What would happen otherwise? Is it an issue
    > of big little endian or some other issue?


    The end-of-line characters might get converted -- even if
    they're not really "end-of-line" characters in the file in
    question.

    --
    Grant Edwards grante Yow! My mind is a potato
    at field...
    visi.com
    Grant Edwards, Jul 12, 2005
    #7
  8. On Tue, 12 Jul 2005 07:38:39 -0700, wrote:

    > Thanks for the nice suggestions!
    >
    > As a side question, you mentioned opening files in binary mode, in case
    > the code needs to run under Windows or cross-platform. What would
    > happen otherwise? Is it an issue of big little endian or some other
    > issue?


    No, nothing to do with big and little endian issues. It is all to do with
    the line delimiter, and possibly the end-of-file marker.

    Windows uses '\r\n' as the line delimiter for text files. (Or is it
    '\n\r'? I always forget...)

    Old-style Macintosh used '\r', and (almost) everything else, including new
    Macs running OS X, uses '\n'.

    If you open files in text mode, there can be complications due to the
    different line endings. To be perfectly frank, I only use Python under
    Linux, so I don't have the foggiest idea of just what Bad Things can
    happen. I know it is a big problem when using some FTP programs, which
    have a tendency to destroy binary programs if you upload/download them in
    text mode.

    I just did some experiments here, and can't get anything bad to happen.
    But whatever the problem is, my grand-pappy always told me, open the
    danged file in binary mode and you can't go wrong.

    *wink*

    I have found some discussions here:

    http://python.active-venture.com/tut/node9.html

    "Windows makes a distinction between text and binary files; the
    end-of-line characters in text files are automatically altered slightly
    when data is read or written. This behind-the-scenes modification to file
    data is fine for ASCII text files, but it'll corrupt binary data like that
    in JPEGs or .EXE files. Be very careful to use binary mode when reading
    and writing such files."

    and here:

    http://zephyrfalcon.org/labs/python_pitfalls.html

    This website recommends:

    "Solution: Use the correct flags -- 'r' for text mode (even on Unix), 'rb'
    for binary mode."

    but I've never had any problems using 'rb' for text files under Linux.

    I'm also told that Windows uses ctrl-Z as the end-of-file marker, and if
    it finds that character in the middle of a text file, it will assume the
    file has finished and stop reading. But only in text mode, not binary. I
    don't think that's a problem for Linux.


    --
    Steven.
    Steven D'Aprano, Jul 12, 2005
    #8
  9. John Machin Guest

    wrote:
    > Hi,
    >
    > I want to append one (huge) file to another (huge) file. The current
    > way I'm doing it is to do something like:
    >
    > infile = open (infilename, 'r')
    > filestr = infile.read()
    > outfile = open(outfilename, 'a')
    > outfile.write(filestr)
    >
    > I wonder if there is a more efficient way doing this?


    Don't wonder, like the ancient philosophers; be an empiricist :)


    > Thanks.
    >


    If the files are truly huge, you run the risk of exhausting real memory
    and having to swap.

    Try this:
    Having opened the files,

    for line in infile:
    outfile.write(line)

    Otherwise look at the docs for read the method and check out the "size"
    argument.

    General warnings: (1) If you want to be portable, consider text/binary
    differences. (2) Consider what to do if the last line in <outfilename>
    is not terminated.
    John Machin, Jul 13, 2005
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. June Moore

    Append one array to another array

    June Moore, Oct 28, 2004, in forum: Java
    Replies:
    20
    Views:
    68,043
    HeidelbergMan
    Jul 6, 2007
  2. HYRY
    Replies:
    10
    Views:
    600
    Bruno Desthuilliers
    Sep 26, 2007
  3. Victor Bazarov

    Re: Append one enum to another?

    Victor Bazarov, May 8, 2009, in forum: C++
    Replies:
    7
    Views:
    513
    Pascal J. Bourguignon
    May 15, 2009
  4. Ishmael

    Append one file to another?

    Ishmael, Jun 23, 2009, in forum: C Programming
    Replies:
    5
    Views:
    1,718
    user923005
    Jun 23, 2009
  5. Robert Cohen
    Replies:
    3
    Views:
    270
    Andrew Durstewitz
    Jul 15, 2003
Loading...

Share This Page