ftplib error- Large file

[half.italian]

Hi all,

I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?

Traceback (most recent call last):

File "/usr/local/sw/bin/FTPput.py", line 142, in execute
self.ftp.put(file.path)

File "/usr/local/sw/lib/FTPconn.py", line 32, in put
return self.storbinary("STOR %s" % destfile, open(f.path, 'rb'))

File "/usr/lib/python2.2/ftplib.py", line 422, in storbinary
return self.voidresp()

File "/usr/lib/python2.2/ftplib.py", line 222, in voidresp
resp = self.getresp()

File "/usr/lib/python2.2/ftplib.py", line 208, in getresp
resp = self.getmultiline()

File "/usr/lib/python2.2/ftplib.py", line 194, in getmultiline
line = self.getline()

File "/usr/lib/python2.2/ftplib.py", line 181, in getline
line = self.file.readline()

IOError: [Errno 104] Connection reset by peer

I'd appreciate any input or ideas.

~Sean

 

[Gabriel Genellina]

I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?


IOError: [Errno 104] Connection reset by peer

You always have to consider a connection failure - it's just more likely
to happen on a big file.
The REST ftp command is used to restart the transfer from a certain offset.

 

[Hendrik van Rooyen]

Hi all,

I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?

20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?

Are there any "anti jabber" precautions taken anywhere?

you may have to break it up into smaller chunks and glue them together
remotely...

hth - Hendrik

 

[half.italian]

20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?

Are there any "anti jabber" precautions taken anywhere?

you may have to break it up into smaller chunks and glue them together
remotely...

hth - Hendrik

It's different sites (state to state) within the same company, so I'm
sure of the IP addresses. Whats odd is that the transfers don't fail
most of the time. The transfer usually completes properly but
ftplib.FTP.storbinary() throws an IOError at the end of the file
instead of returning the status of the transfer. I've resorted to
just catching and logging the exception and then comparing remote vs.
local md5's to check for integrity, but it seems like if a connection
error occurs, storbinary should handle the error and return an error
code rather than throwing an exception. The 20gb files take about an
hour to transfer, and I've had success with files up to 60gb so far...
once I figured out what ftplib was doing and starting cathing the
IOError.

Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?

~Sean

 

[Facundo Batista]

Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?

There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.

You always can hide it yourself, with try/except...

Regards,

 

[half.italian]

There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.

You always can hide it yourself, with try/except...

Regards,

--
. Facundo
.
Blog:http://www.taniquetil.com.ar/plog/
PyAr:http://www.python.org/ar/

Ok. I guess that makes sense. But what about the other
questions...mainly: Why would it throw an exception even though the
file was properly transferred?

I guess this could be blamed on the FTP server.

~Sean

 

[Facundo Batista]

Ok. I guess that makes sense. But what about the other
questions...mainly: Why would it throw an exception even though the
file was properly transferred?

Je, well, I answered the one I knew about,

Regarding the error... es hard to say.

What happens if you transmit the file using a standard ftp program?

Or better, could you sniff your network to see what is the TCP dialog?

I guess this could be blamed on the FTP server.

Maybe. Maybe not...