fast copying of large files in python

  • Thread starter Catherine Moroney
  • Start date
C

Catherine Moroney

Hello,

I'm working on an application that as part of its processing, needs to
copy 50 Meg binary files from one NFS mounted disk to another.

The simple-minded approach of shutil.copyfile is very slow, and I'm
guessing that this is due to the default 16K buffer size. Using
shtil.copyfileobj is faster when I set a larger buffer size, but it's
still slow compared to a "os.system("cp %s %s" % (f1,f2))" call.
Is this because of the overhead entailed in having to open the files
in copyfileobj?

I'm not worried about portability, so should I just do the
os.system call as described above and be done with it? Is that the
fastest method in this case? Are there any "pure python" ways of
getting the same speed as os.system?

Thanks,

Catherine
 
D

Dave Angel

Hello,

I'm working on an application that as part of its processing, needs to
copy 50 Meg binary files from one NFS mounted disk to another.

The simple-minded approach of shutil.copyfile is very slow, and I'm
guessing that this is due to the default 16K buffer size. Using
shtil.copyfileobj is faster when I set a larger buffer size, but it's
still slow compared to a "os.system("cp %s %s" % (f1,f2))" call.
Is this because of the overhead entailed in having to open the files
in copyfileobj?

I'm not worried about portability, so should I just do the
os.system call as described above and be done with it? Is that the
fastest method in this case? Are there any "pure python" ways of
getting the same speed as os.system?

Thanks,

Catherine

I suspect the fastest way would be to use scp, and not try to use your
local mount for either file. There are some options on scp that tell it
to just connect the (up to) two machines to each other and transfer,
where it won't even come to your local machine.

I do that sort of thing when I'm connecting over slow internet (and vpn)
to two machines that are on the same local subnet.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,906
Latest member
SkinfixSkintag

Latest Threads

Top