Multiple threaded download streams?

Discussion in 'Python' started by gjzusenet@gmail.com, Feb 25, 2006.

  1. Guest

    Hello.
    Though Python supports threading, I think it is limited to python code
    - as soon as you issue a command that uses an external (C?) module, all
    of your python threads hang until this command returns.
    Is that true?
    I'm using urllib2 to download many files, and I have a double problem:
    1. downloading all of them is painfully slow since it's serial - one
    has to finish before the next request gets sent.
    2. my GUI becomes non responsive during the downloads - major problem!

    Is there any way to work around that?
    I want to run multiple download streams, in parallel, and while keeping
    my program responsive.
    Are there alternative modules that I can use for that?

    Any ideas?
    Thanks a lot!
     
    , Feb 25, 2006
    #1
    1. Advertising

  2. On 2006-02-25, <> wrote:

    > Though Python supports threading, I think it is limited to
    > python code - as soon as you issue a command that uses an
    > external (C?) module, all of your python threads hang until
    > this command returns. Is that true?


    No. Not unless the C modules is broken.

    > I'm using urllib2 to download many files, and I have a double
    > problem:
    >
    > 1. downloading all of them is painfully slow since it's serial
    > - one has to finish before the next request gets sent.


    Then don't do it that way.

    > 2. my GUI becomes non responsive during the downloads - major
    > problem!


    Use a separate thread for downloading.

    > Is there any way to work around that?


    Yes. Use threads.

    > I want to run multiple download streams, in parallel, and while keeping
    > my program responsive.


    Then do it that way.

    > Are there alternative modules that I can use for that?


    And what, exactly, didn't work right when you tried the
    threading module?

    --
    Grant Edwards grante Yow! I demand IMPUNITY!
    at
    visi.com
     
    Grant Edwards, Feb 25, 2006
    #2
    1. Advertising

  3. > Use a separate thread for downloading.

    Or the twisted select-reactor. No threads needed.

    Diez
     
    Diez B. Roggisch, Feb 25, 2006
    #3
  4. Peter Hansen Guest

    Diez B. Roggisch wrote:
    >>Use a separate thread for downloading.

    >
    > Or the twisted select-reactor. No threads needed.


    Although depending on what the GUI is like, and what platform is
    involved, one might still want at least a second thread for the reactor
    itself.

    -Peter
     
    Peter Hansen, Feb 25, 2006
    #4
  5. Bryan Olson Guest

    Diez B. Roggisch wrote:
    >> Use a separate thread for downloading.

    >
    > Or the twisted select-reactor. No threads needed.


    He's using urllib2, which does not use Twisted's select-reactor.

    Fortunately urllib2 downloads run fine in their own threads; no
    rewrite-all-the-code-as-Twisted-state-machines needed.


    --
    --Bryan
     
    Bryan Olson, Feb 25, 2006
    #5
  6. Larry Bates Guest

    wrote:
    > Hello.
    > Though Python supports threading, I think it is limited to python code
    > - as soon as you issue a command that uses an external (C?) module, all
    > of your python threads hang until this command returns.
    > Is that true?
    > I'm using urllib2 to download many files, and I have a double problem:
    > 1. downloading all of them is painfully slow since it's serial - one
    > has to finish before the next request gets sent.
    > 2. my GUI becomes non responsive during the downloads - major problem!
    >
    > Is there any way to work around that?
    > I want to run multiple download streams, in parallel, and while keeping
    > my program responsive.
    > Are there alternative modules that I can use for that?
    >
    > Any ideas?
    > Thanks a lot!
    >

    Others have spoken to the specifics of threads, etc. What I wanted
    to ask was why you think that parallel downloads will be faster?
    Unless you are downloading from multiple and different sites and
    your Internet download bandwidth is much greater than any individual
    site you wish to download from, you won't speed things up. If you
    wish to download several files from the SAME site, I doubt that
    threading is going to help. Most likely you are limited by the
    upload bandwidth of that site. Now if the site you are downloading
    from has some throttling turned on to limit the speed of an
    individual download, you could gain some benefit. You can't push
    bits through the pipes faster than their upper bandwidth.

    Just some thoughts to consider.

    -Larry Bates
     
    Larry Bates, Feb 25, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Adam Atlas
    Replies:
    1
    Views:
    296
    Adam Atlas
    Aug 24, 2005
  2. Robert Brewer
    Replies:
    0
    Views:
    409
    Robert Brewer
    Aug 24, 2005
  3. Chandru
    Replies:
    2
    Views:
    324
    Chandru
    Apr 16, 2007
  4. Paul Hemans

    Managing a multiple threaded service

    Paul Hemans, Jun 17, 2009, in forum: Python
    Replies:
    1
    Views:
    251
    Mark Hammond
    Jun 17, 2009
  5. dotnetnewbie
    Replies:
    1
    Views:
    144
Loading...

Share This Page