Odd/Weird errors with FTPLib

B

Bakes

I am using a simple python script to download my logfiles. This is on
a while loop, the logfile grows rapidly, so it is necessary for python
to start downloading the new script as soon as it has finished the
old.

It works fine (for about 20 minutes), then crashes. I have removed a
couple of excepts, and have narrowed the error down to a 'error_perm:
550 logfile.log: The data is invalid.' error.

Does anyone know what the problem might be regarding this, and what I
might do to fix it?

It's really irritating to have it break every so often, and at the
moment, I have no chance in which to fix the error.
 
C

Chris Rebert

I am using a simple python script to download my logfiles. This is on
a while loop, the logfile grows rapidly, so it is necessary for python
to start downloading the new script as soon as it has finished the
old.

It works fine (for about 20 minutes), then crashes. I have removed a
couple of excepts, and have narrowed the error down to a 'error_perm:
550 logfile.log: The data is invalid.' error.

Does anyone know what the problem might be regarding this, and what I
might do to fix it?

Including an actual code snippet and the full error traceback would help a lot.

According to http://en.wikipedia.org/wiki/List_of_FTP_server_return_codes ,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."

Does the logfile get rotated or something, thus causing it to briefly not exist?

It might also help if you explain how your logfile system works.

Cheers,
Chris
 
B

Bakes

Including an actual code snippet and the full error traceback would help a lot.

According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."

Does the logfile get rotated or something, thus causing it to briefly not exist?

It might also help if you explain how your logfile system works.

Cheers,
Chris
--http://blog.rebertia.com

It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.

The logfile is downloaded using this try/except while loop.


while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to
(re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig
['path']), handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until
updated properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not
closed')
ftp = False
time.sleep(10)


I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
 
M

MRAB

Bakes said:
Including an actual code snippet and the full error traceback would help a lot.

According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."

Does the logfile get rotated or something, thus causing it to briefly not exist?

It might also help if you explain how your logfile system works.

Cheers,
Chris
--http://blog.rebertia.com

It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.

The logfile is downloaded using this try/except while loop.


while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to (re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']), handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until updated properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not closed')
ftp = False
time.sleep(10)


I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?

Bare excepts are almost always a bad idea because they'll catch _all_
exceptions, both those you expect could happen and those you don't.
Catch only those you expect.

For example, if the file 'games_mp.log' doesn't exist then
os.path.getsize('games_mp.log') will raise an exception, and if you
forgot to import the os module then that will raise a NameError
exception.

Anyway, I can't see how you leave the loop; I'd expect something like a
'break' statement.

And as a matter of style, I'd prefer None to False to indicate when
there's no FTP connection (and "if not ftp" instead of "if ftp ==
False").
 
B

Bakes

It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
    while True:
        try:
            if ftp == False:
                self.debug('FTP connection not active, attempting to (re)connect')
                ftp = self.ftpconnect()
            size=os.path.getsize('games_mp.log')
            ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']), handleDownload, rest=size)
            if self.console._paused:
                self.console.unpause()
        except:
            print error
            self.debug('Lost connection to server, pausing until updated properly, Sleeping 10 seconds')
            self.console.pause()
            try:
                ftp.close()
                self.debug('FTP Connection Closed')
            except:
                self.debug('FTP does not appear to be open, so not closed')
            ftp = False
            time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?

Bare excepts are almost always a bad idea because they'll catch _all_
exceptions, both those you expect could happen and those you don't.
Catch only those you expect.

For example, if the file 'games_mp.log' doesn't exist then
os.path.getsize('games_mp.log') will raise an exception, and if you
forgot to import the os module then that will raise a NameError
exception.

Anyway, I can't see how you leave the loop; I'd expect something like a
'break' statement.

And as a matter of style, I'd prefer None to False to indicate when
there's no FTP connection (and "if not ftp" instead of "if ftp ==
False").

I removed the try/except and saw when it failed.

I'll change those things, games_mp.log is guaranteed to be there (file
made in another script), os is imported correctly.

So, what do you think the error could be?
 
M

MRAB

Bakes said:
Bakes said:
I am using a simple python script to download my logfiles. This is on
a while loop, the logfile grows rapidly, so it is necessary for python
to start downloading the new script as soon as it has finished the
old.
It works fine (for about 20 minutes), then crashes. I have removed a
couple of excepts, and have narrowed the error down to a 'error_perm:
550 logfile.log: The data is invalid.' error.
Does anyone know what the problem might be regarding this, and what I
might do to fix it?
Including an actual code snippet and the full error traceback would help a lot.
According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."
Does the logfile get rotated or something, thus causing it to briefly not exist?
It might also help if you explain how your logfile system works.
Cheers,
Chris
--http://blog.rebertia.com
It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
while True:
try:
if ftp == False:
self.debug('FTP connection not active, attempting to (re)connect')
ftp = self.ftpconnect()
size=os.path.getsize('games_mp.log')
ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']), handleDownload, rest=size)
if self.console._paused:
self.console.unpause()
except:
print error
self.debug('Lost connection to server, pausing until updated properly, Sleeping 10 seconds')
self.console.pause()
try:
ftp.close()
self.debug('FTP Connection Closed')
except:
self.debug('FTP does not appear to be open, so not closed')
ftp = False
time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
Bare excepts are almost always a bad idea because they'll catch _all_
exceptions, both those you expect could happen and those you don't.
Catch only those you expect.

For example, if the file 'games_mp.log' doesn't exist then
os.path.getsize('games_mp.log') will raise an exception, and if you
forgot to import the os module then that will raise a NameError
exception.

Anyway, I can't see how you leave the loop; I'd expect something like a
'break' statement.

And as a matter of style, I'd prefer None to False to indicate when
there's no FTP connection (and "if not ftp" instead of "if ftp ==
False").

I removed the try/except and saw when it failed.

I'll change those things, games_mp.log is guaranteed to be there (file
made in another script), os is imported correctly.

So, what do you think the error could be?

How does control leave the while loop?

If you're running on Windows and you're opening the file 'games_mp.log'
with mode 'w' then there'll be the issue of the line-endings. If the log
file on the server uses '\n' for the line endings and you're using
'\r\n' then os.path.getsize('games_mp.log') will return a larger size
then you expect. In that case, could ftp.retrbinary() be complaining
because it's given you the entire file and then you're trying to
download from an offset that's beyond the end?

For example, suppose the file on the server contains just "foo\n" (4
bytes):

1. You open a file locally in text mode ('w').

2. You download the entire file, "foo\n".

3. You write out the data, but because you've opened in text mode it
writes "foo\r\n" to the file. The local file size is now 5 bytes.

4. The loop means that you then try to download from offset 5, which is
beyond the end of the file on the server.

5. Error?
 
B

Bakes

Bakes said:
Bakes wrote:
I am using a simple python script to download my logfiles. This is on
a while loop, the logfile grows rapidly, so it is necessary for python
to start downloading the new script as soon as it has finished the
old.
It works fine (for about 20 minutes), then crashes. I have removed a
couple of excepts, and have narrowed the error down to a 'error_perm:
550 logfile.log: The data is invalid.' error.
Does anyone know what the problem might be regarding this, and what I
might do to fix it?
Including an actual code snippet and the full error traceback would help a lot.
According tohttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes,
error code 550 translates to:
"Requested action not taken. File unavailable (e.g., file not found,
no access)."
Does the logfile get rotated or something, thus causing it to briefly not exist?
It might also help if you explain how your logfile system works.
Cheers,
Chris
--http://blog.rebertia.com
It's a cod4 gameserver logfile, being downloaded for a python bot to
parse.
The logfile is downloaded using this try/except while loop.
    while True:
        try:
            if ftp == False:
                self.debug('FTP connection not active, attempting to (re)connect')
                ftp = self.ftpconnect()
            size=os.path.getsize('games_mp.log')
            ftp.retrbinary('RETR ' + os.path.basename(self.ftpconfig ['path']), handleDownload, rest=size)
            if self.console._paused:
                self.console.unpause()
        except:
            print error
            self.debug('Lost connection to server, pausing until updated properly, Sleeping 10 seconds')
            self.console.pause()
            try:
                ftp.close()
                self.debug('FTP Connection Closed')
            except:
                self.debug('FTP does not appear to be open, so not closed')
            ftp = False
            time.sleep(10)
I can only assume that occasionally, the logfile is being written to
by the gameserver at the same time that it's downloading.
If this was the case, do you think a try: download except: sleep
900msec then download loop would work?
Bare excepts are almost always a bad idea because they'll catch _all_
exceptions, both those you expect could happen and those you don't.
Catch only those you expect.
For example, if the file 'games_mp.log' doesn't exist then
os.path.getsize('games_mp.log') will raise an exception, and if you
forgot to import the os module then that will raise a NameError
exception.
Anyway, I can't see how you leave the loop; I'd expect something like a
'break' statement.
And as a matter of style, I'd prefer None to False to indicate when
there's no FTP connection (and "if not ftp" instead of "if ftp ==
False").
I removed the try/except and saw when it failed.
I'll change those things, games_mp.log is guaranteed to be there (file
made in another script), os is imported correctly.
So, what do you think the error could be?

How does control leave the while loop?

If you're running on Windows and you're opening the file 'games_mp.log'
with mode 'w' then there'll be the issue of the line-endings. If the log
file on the server uses '\n' for the line endings and you're using
'\r\n' then os.path.getsize('games_mp.log') will return a larger size
then you expect. In that case, could ftp.retrbinary() be complaining
because it's given you the entire file and then you're trying to
download from an offset that's beyond the end?

For example, suppose the file on the server contains just "foo\n" (4
bytes):

1. You open a file locally in text mode ('w').

2. You download the entire file, "foo\n".

3. You write out the data, but because you've opened in text mode it
writes "foo\r\n" to the file. The local file size is now 5 bytes.

4. The loop means that you then try to download from offset 5, which is
beyond the end of the file on the server.

5. Error?

It opens it in append mode, and if it errors with that message, the
ending is different. I have come across that problem before, and it
has been fixed (download in binary is the fix, mostly).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

ftplib limitations? 9
FTPlib 2
ftplib acting weird 1
ftplib - Did the whole file get sent? 6
Trouble with ftplib uploading to an FTP server 4
using ftplib 5
Weird newbie question 4
ftplib strange behaviour 2

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top