Multiple threaded download streams?

G

gjzusenet

Hello.
Though Python supports threading, I think it is limited to python code
- as soon as you issue a command that uses an external (C?) module, all
of your python threads hang until this command returns.
Is that true?
I'm using urllib2 to download many files, and I have a double problem:
1. downloading all of them is painfully slow since it's serial - one
has to finish before the next request gets sent.
2. my GUI becomes non responsive during the downloads - major problem!

Is there any way to work around that?
I want to run multiple download streams, in parallel, and while keeping
my program responsive.
Are there alternative modules that I can use for that?

Any ideas?
Thanks a lot!
 
G

Grant Edwards

Though Python supports threading, I think it is limited to
python code - as soon as you issue a command that uses an
external (C?) module, all of your python threads hang until
this command returns. Is that true?

No. Not unless the C modules is broken.
I'm using urllib2 to download many files, and I have a double
problem:

1. downloading all of them is painfully slow since it's serial
- one has to finish before the next request gets sent.

Then don't do it that way.
2. my GUI becomes non responsive during the downloads - major
problem!

Use a separate thread for downloading.
Is there any way to work around that?

Yes. Use threads.
I want to run multiple download streams, in parallel, and while keeping
my program responsive.

Then do it that way.
Are there alternative modules that I can use for that?

And what, exactly, didn't work right when you tried the
threading module?
 
D

Diez B. Roggisch

Use a separate thread for downloading.

Or the twisted select-reactor. No threads needed.

Diez
 
P

Peter Hansen

Diez said:
Or the twisted select-reactor. No threads needed.

Although depending on what the GUI is like, and what platform is
involved, one might still want at least a second thread for the reactor
itself.

-Peter
 
B

Bryan Olson

Diez said:
Or the twisted select-reactor. No threads needed.

He's using urllib2, which does not use Twisted's select-reactor.

Fortunately urllib2 downloads run fine in their own threads; no
rewrite-all-the-code-as-Twisted-state-machines needed.
 
L

Larry Bates

Hello.
Though Python supports threading, I think it is limited to python code
- as soon as you issue a command that uses an external (C?) module, all
of your python threads hang until this command returns.
Is that true?
I'm using urllib2 to download many files, and I have a double problem:
1. downloading all of them is painfully slow since it's serial - one
has to finish before the next request gets sent.
2. my GUI becomes non responsive during the downloads - major problem!

Is there any way to work around that?
I want to run multiple download streams, in parallel, and while keeping
my program responsive.
Are there alternative modules that I can use for that?

Any ideas?
Thanks a lot!
Others have spoken to the specifics of threads, etc. What I wanted
to ask was why you think that parallel downloads will be faster?
Unless you are downloading from multiple and different sites and
your Internet download bandwidth is much greater than any individual
site you wish to download from, you won't speed things up. If you
wish to download several files from the SAME site, I doubt that
threading is going to help. Most likely you are limited by the
upload bandwidth of that site. Now if the site you are downloading
from has some throttling turned on to limit the speed of an
individual download, you could gain some benefit. You can't push
bits through the pipes faster than their upper bandwidth.

Just some thoughts to consider.

-Larry Bates
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top