Multi-threaded FTP Question

[dbandler]

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:

File "C:\Python24\lib\ftplib.py", line 107, in __init__
self.connect(host)
File "C:\Python24\lib\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\lib\ftplib.py", line 208, in getresp
resp = self.getmultiline()
File "C:\Python24\lib\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\lib\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError

Is it possible to have more than five simultaneous ftp connections?

Thanks.

Derek

 

[Jeremy Jones]

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:

File "C:\Python24\lib\ftplib.py", line 107, in __init__
self.connect(host)
File "C:\Python24\lib\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\lib\ftplib.py", line 208, in getresp
resp = self.getmultiline()
File "C:\Python24\lib\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\lib\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError

Is it possible to have more than five simultaneous ftp connections?

Thanks.

Derek

I replied to this about 4 hours ago from my gmail email account (not my
google groups account associated with the same email addres), but
haven't seen it show up, so I apologize if this is a dupe.

Would you mind posting your code? Are you trying to pass the same FTP
connection object to all 5 threads?

- Jeremy M. Jones

 

[dbandler]

There are n instances of ftplib.FTP.

Funny thing is that I tried to run the code twice, concurrently, in two
separate IDLE instances. Each instance had four threads/ftp calls.
Again, only five ftp connections were allowed on my computer. The code
errored out on the sixth attempt.

I wonder if there' s some underlying software being called that has a
limit on the number of ftp connections allowed at any point in time.
I'm tempted to try this on cygwin to see if the behavior is different.

Derek

import ftplib, zipfile, datetime, os
from threading import Thread, Lock


class ftpDownload(Thread):
def __init__(self, year, quarter):
Thread.__init__(self)
self.quarter = str(quarter)
self.year = str(year)
self.filename = ""
self.ftp = ftplib.FTP('xxxxxxx')
self.ftp.login()
self.lock = Lock()

def run(self):
filehandle = open(self.filename,'wb')
self.ftp.retrbinary('RETR /xxxxx/xxxxxxx/' + self.year + '/QTR'
+ self.quarter + '/xxxxxxx.zip', filehandle.write)
filehandle.close()
self.ftp.close()

ftplist = []
for year in range(1990,2000):
for quarter in range(1,5):
current = ftpDownload(year, quarter)
ftplist.append(current)
current.start()

for job in ftplist:
job.join()

 

[Dennis Lee Bieber]

Is it possible to have more than five simultaneous ftp connections?

Could it be that the SERVER is limiting things to 5
concurrent/parallel connections from any single IP?

I know I've encountered sites that only allowed two FTP downloads at
a time...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Jeremy Jones]

On 11 Jul 2006 06:45:42 -0700, (e-mail address removed) declaimed the
following in comp.lang.python:

Could it be that the SERVER is limiting things to 5
concurrent/parallel connections from any single IP?

I know I've encountered sites that only allowed two FTP downloads at
a time...

This is what I was starting to think as well. The only thing that
looked funky with the OP's code was that it looked like he was writing
everything to a filename of "" (unless he's intentionally modified his
code to not show where he's setting that).

- Jeremy M. Jones

 

[olsongt]

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:

File "C:\Python24\lib\ftplib.py", line 107, in __init__
self.connect(host)
File "C:\Python24\lib\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\lib\ftplib.py", line 208, in getresp
resp = self.getmultiline()
File "C:\Python24\lib\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\lib\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError

Is it possible to have more than five simultaneous ftp connections?

Thanks.

Derek

It might be XP SP2's worm protection as well:

http://www.speedguide.net/read_articles.php?id=1497

 

[dbandler]

Thanks so much for your help on this. The server that I'm connecting
to is the culprit. They only allow five connections at a time.

I assumed that it was a code issue. I think that we're conditioned to
expect that the problem is on the software side of things.

-Derek