Multiple FTP download using Muliti thread

[johnny]

I have taken a look at the code that dose one download at time, in
multi threaded manner:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/465531

What I wanted to do is, make it download multiple files at the same
time. I am new to python and have gone over "Dive In To Python"
yesterday. Can some give a idea what I need to do. Once I understand
the big picture, then I can piece together.

 

[Justin Ezequiel]

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open(p, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
host = 'ftp.microsoft.com'

c = ftplib.FTP(host)
c.connect()
try:
c.login()
folder = '/deskapps/kids/'
for n in c.nlst(folder):
if n.lower().endswith('.exe'):
q.put((host, n))
finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'

 

[Fredrik Lundh]

from TaskQueue import TaskQueue

what Python version is this ?

</F>

 

[Justin Ezequiel]

what Python version is this ?

</F>

oops. forgot to note...

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475160

 

[Peter Otten]

oops. forgot to note...

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475160

Also noteworthy: "This recipe was accepted for inclusion in Py2.5.", i. e.
in Python 2.5 you can use Queue instead of TaskQueue.

Peter

 

[johnny]

Where or What folder does the ftp files get downloaded to?

 

[johnny]

When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?

 

[Justin Ezequiel]

When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?

does it work using ftp.microsoft.com?

post your code

 

[johnny]

It works using ftp.microsoft.com. But where does it put the downloaded
files? can I specify a download folder location?

 

[johnny]

It places the ftp downloaded contents on the same folder as the this
ftp python script. How do I set a diffrent download folder location?

 

[Fredrik Lundh]

It places the ftp downloaded contents on the same folder as the this
ftp python script. How do I set a diffrent download folder location?

by prepending a directory name to the filename in the open(p, 'wb') call.

</F>

 

[johnny]

I am getting the following error:

raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Here is the code:

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
#host = 'ftp.microsoft.com'
host = 'mysite.com'
c = ftplib.FTP(host)
c.connect()
try:
#c.login()
c.login("(e-mail address removed)","temppass" )

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
#if n.lower().endswith('.exe'):
# q.put((host, n))
if n.lower().endswith('.jpg'):
q.put((host, n))
elif n.lower().endswith('.jpeg'):
q.put((host, n))

finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'

 

[johnny]

Ok I fixed it. Needed to put in username, and password in the c.login
inside while True loop.

while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')