FileUtils.cp_r is slow?

[Paul Lynch]

I wrote a ruby script to backup files from my hard drive to my flash
drive. For some reason, with one of my flash drives, copying files
with FileUtils.cp_r is about 250 times slower than if I drag and drop
the files from one drive to the other in Windows. I'm suspecting a
buffer size problem. Is there a way to control the buffer size cp_r
selects? If not, what alternatives are there to FileUtils.cp_r? (I
am about to try writing my own, but someone must already have a
solution....)

 

[Robert Klemme]

I wrote a ruby script to backup files from my hard drive to my flash
drive. For some reason, with one of my flash drives, copying files
with FileUtils.cp_r is about 250 times slower than if I drag and drop
the files from one drive to the other in Windows. I'm suspecting a
buffer size problem. Is there a way to control the buffer size cp_r
selects? If not, what alternatives are there to FileUtils.cp_r? (I
am about to try writing my own, but someone must already have a
solution....)

Could it be that Windows is doing the copy in the background while cp_r
doesn't? After all, there is a reason why you must not simply plug off
an USB drive...

Kind regards

robert

 

[Robert Klemme]

I wrote a ruby script to backup files from my hard drive to my flash
drive. For some reason, with one of my flash drives, copying files
with FileUtils.cp_r is about 250 times slower than if I drag and drop
the files from one drive to the other in Windows. I'm suspecting a
buffer size problem. Is there a way to control the buffer size cp_r
selects? If not, what alternatives are there to FileUtils.cp_r? (I
am about to try writing my own, but someone must already have a
solution....)

Could it be that Windows is doing the copy in the background while cp_r
doesn't? After all, there is a reason why you must not simply plug off
an USB drive...

Kind regards

robert

 

[Paul Lynch]

[Note: parts of this message were removed to make it a legal post.]

I did some more testing on this. I found that if I ran FileUtils.cp_r or
FileUtils.cp from within irb, the speed was reasonable (maybe 30s to copy
25MB to my flash drive.) But, if I double-clicked my script to run it, it
took about 30 minutes (roughly-- I didn't wait that long.) I wrote my own
copy method that opens the files in binary mode and uses a 10 MB buffer size
(with the read and write methods) and speed dropped back down to 26s. This
is still 6x slower than Windows drag and drop, but it is tolerable.

Here's my copy code, in case anyone else has this problem:
def self.copy_file(from_file, to_file)
buffer_size = 10*MB
begin
to_s = File.open(to_file, 'wb')
File.open(from_file, 'rb') do |from_s|
while(data = from_s.read(buffer_size))
log("read #{data.length}")
to_s.write(data)
end
end
rescue
log("Caught error: "+$!)
to_s.close if to_s
raise
end
end

 

[Robert Klemme]

Just another idea: if your files reside on two different physical
disks you might get a benefit from separating reading and writing into
two different threads.

Kind regards

robert

 

[Luc Evers]

[Note: parts of this message were removed to make it a legal post.]

For file backup I'm using rsync on a Linux system, its possible to start
this tool from Ruby.
Rsync copy the difference between the files and is very fast.

I don't now rsync exist in Windows (maybe Powershell), but its possible
to run on windows via Cygwin.


Luc.