fast copying of large files in python

Discussion in 'Python' started by Catherine Moroney, Nov 2, 2011.

  1. Hello,

    I'm working on an application that as part of its processing, needs to
    copy 50 Meg binary files from one NFS mounted disk to another.

    The simple-minded approach of shutil.copyfile is very slow, and I'm
    guessing that this is due to the default 16K buffer size. Using
    shtil.copyfileobj is faster when I set a larger buffer size, but it's
    still slow compared to a "os.system("cp %s %s" % (f1,f2))" call.
    Is this because of the overhead entailed in having to open the files
    in copyfileobj?

    I'm not worried about portability, so should I just do the
    os.system call as described above and be done with it? Is that the
    fastest method in this case? Are there any "pure python" ways of
    getting the same speed as os.system?

    Thanks,

    Catherine
    Catherine Moroney, Nov 2, 2011
    #1
    1. Advertising

  2. Catherine Moroney

    Dave Angel Guest

    On 11/02/2011 03:31 PM, Catherine Moroney wrote:
    > Hello,
    >
    > I'm working on an application that as part of its processing, needs to
    > copy 50 Meg binary files from one NFS mounted disk to another.
    >
    > The simple-minded approach of shutil.copyfile is very slow, and I'm
    > guessing that this is due to the default 16K buffer size. Using
    > shtil.copyfileobj is faster when I set a larger buffer size, but it's
    > still slow compared to a "os.system("cp %s %s" % (f1,f2))" call.
    > Is this because of the overhead entailed in having to open the files
    > in copyfileobj?
    >
    > I'm not worried about portability, so should I just do the
    > os.system call as described above and be done with it? Is that the
    > fastest method in this case? Are there any "pure python" ways of
    > getting the same speed as os.system?
    >
    > Thanks,
    >
    > Catherine


    I suspect the fastest way would be to use scp, and not try to use your
    local mount for either file. There are some options on scp that tell it
    to just connect the (up to) two machines to each other and transfer,
    where it won't even come to your local machine.

    I do that sort of thing when I'm connecting over slow internet (and vpn)
    to two machines that are on the same local subnet.

    --

    DaveA
    Dave Angel, Nov 2, 2011
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. PedroX
    Replies:
    9
    Views:
    1,527
    Bryce K. Nielsen
    Jun 28, 2005
  2. Michele Simionato

    Python is darn fast (was: How fast is Python)

    Michele Simionato, Aug 23, 2003, in forum: Python
    Replies:
    13
    Views:
    552
  3. Yi Xing
    Replies:
    6
    Views:
    458
    Simon Forman
    Jul 26, 2006
  4. Devesh Agrawal
    Replies:
    18
    Views:
    250
  5. Stuart Clarke

    Fast searching of large files

    Stuart Clarke, Jul 1, 2010, in forum: Ruby
    Replies:
    6
    Views:
    193
    Roger Pack
    Jul 1, 2010
Loading...

Share This Page