asynchronous downloading

Discussion in 'Python' started by Plumo, Feb 23, 2012.

  1. Plumo

    Plumo Guest

    I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet).

    (I would use gevent under different circumstances, but currently need to stick to standard libraries.)

    I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/

    What do you recommend?
    And why is there poor support for asynchronous execution?

    Richard
     
    Plumo, Feb 23, 2012
    #1
    1. Advertising

  2. Justin Ezequiel, Feb 23, 2012
    #2
    1. Advertising

  3. Plumo

    Mark Hammond Guest

    On 23/02/2012 5:58 PM, Plumo wrote:
    > I want to download content asynchronously. This would be
    > straightforward to do threaded or across processes, but difficult
    > asynchronously so people seem to rely on external libraries (twisted
    > / gevent / eventlet).


    Exactly - the fact it's difficult is why those tools compete.

    > (I would use gevent under different circumstances, but currently need
    > to stick to standard libraries.)


    As above - use threads or processes - they are fine for relatively
    modest tasks. If your needs go beyond modest, I'd reevaluate your need
    to stick with just the stdlib - even demanding *sync* http apps often
    wind up using modules outside the stdlib. Look into virtualenv etc if
    permission to install packages is the issue.

    Batteries included free, but turbo-chargers are an extra ;)

    Mark
     
    Mark Hammond, Feb 23, 2012
    #3
  4. Plumo

    Paul Rubin Guest

    Plumo <> writes:
    > What do you recommend?


    Threads.

    > And why is there poor support for asynchronous execution?


    The freenode #python crowd seems to hate threads and prefer twisted,
    which seems to have the features you want and probably handles very
    large #'s of connections better than POSIX threads do. But I find the
    whole event-driven model to be an annoying abstraction inversion and
    threads to be simpler, so I've stayed with threads. I keep hearing
    boogieman stories about the evil hazards of race conditions etc. but
    none of that stuff has ever happened to me (yet) as far as I can tell.
    The main thing is to avoid sharing mutable data between threads to the
    extent that you can. Keep the threads isolated from each other except
    for communication through Queues and not too much can go wrong. The
    last program I wrote had around 20 threads and one or two condition
    variables and I don't think any significant bugs resulted from that.

    FWIW, the Erlang language is built around the above concept, it uses
    super-lightweight userland threads so it can handle millions of them
    concurrently, and it's used successfully for ultra-high-reliability
    phone switches and similar applications that are not allowed to fail, so
    it must be doing something right.

    There are a few schemes like Camaelia (sp?) implementing concurrency
    with Python generators or coroutines, but I think they're not widely
    used, and Python coroutines are kind of crippled because they don't
    carry any stack below their entry point.
     
    Paul Rubin, Feb 23, 2012
    #4
  5. >> I want to download content asynchronously. This would be
    >> straightforward to do threaded or across processes, but difficult
    >> asynchronously so people seem to rely on external libraries (twisted
    >> / gevent / eventlet).

    >
    >
    > Exactly - the fact it's difficult is why those tools compete.


    It is difficult in Python because the async libraries do not offer
    much. Straightforward in some other languages.

    Do you know why there is little support for asynchronous execution in
    the standard libraries?
    For large scale downloading I found thread pools do not scale well.

    Richard
     
    Richard Baron Penman, Feb 23, 2012
    #5
  6. Il 23 febbraio 2012 07:58, Plumo <> ha scritto:
    > I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet).
    >
    > (I would use gevent under different circumstances, but currently need to stick to standard libraries.)
    >
    > I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/
    >
    > What do you recommend?
    > And why is there poor support for asynchronous execution?
    >
    > Richard
    > --
    > http://mail.python.org/mailman/listinfo/python-list


    If you want to stick with asyncore try to take a look at this:
    https://gist.github.com/1519999

    > And why is there poor support for asynchronous execution?


    I'd say that's true for stdlib only (asyncore/asynchat).
    There are plenty of choices amongst third party modules though.
    To say one, I particularly like tornado which is simple and powerful:
    http://www.tornadoweb.org/documentation/httpclient.html

    --- Giampaolo
    http://code.google.com/p/pyftpdlib/
    http://code.google.com/p/psutil/
    http://code.google.com/p/pysendfile/
     
    Giampaolo Rodolà, Feb 23, 2012
    #6
  7. Plumo

    Plumo Guest

    My current implementation works fine below a few hundred threads. But each thread takes up a lot of memory so does not scale well.

    I have been looking at Erlang for that reason, but found it is missing useful libraries in other areas.
     
    Plumo, Feb 24, 2012
    #7
  8. Plumo

    Plumo Guest

    that example is excellent - best use of asynchat I have seen so far.

    I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
     
    Plumo, Feb 24, 2012
    #8
  9. Plumo

    Plumo Guest

    that example is excellent - best use of asynchat I have seen so far.

    I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
     
    Plumo, Feb 24, 2012
    #9
  10. Il 24 febbraio 2012 02:10, Plumo <> ha scritto:
    > that example is excellent - best use of asynchat I have seen so far.
    >
    > I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.


    Well, actually I do/did.
    Point with asyncore/asynchat is that it's original design is so flawed
    and simplicistic it doesn't allow actual customization without
    breaking compatibility.
    See for example:
    http://bugs.python.org/issue6692


    --- Giampaolo
    http://code.google.com/p/pyftpdlib/
    http://code.google.com/p/psutil/
    http://code.google.com/p/pysendfile/
     
    Giampaolo Rodolà, Feb 24, 2012
    #10
  11. >> I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
    >
    > Well, actually I do/did.


    ah OK. I had read this comment from a few years back:
    "IIRC, there was a threat to remove asyncore because there were no
    maintainers, no one was fixing bugs, no one was improving it, and no
    one was really using it"


    > Point with asyncore/asynchat is that it's original design is so flawed
    > and simplicistic it doesn't allow actual customization without
    > breaking compatibility.


    Python3 uses the same API - was there not enough interest to improve it?

    Richard
     
    Richard Baron Penman, Feb 25, 2012
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amir

    asynchronous design

    Amir, Oct 10, 2003, in forum: VHDL
    Replies:
    3
    Views:
    683
  2. Jamie
    Replies:
    13
    Views:
    7,723
    Renaud Pacalet
    Oct 23, 2003
  3. Jamie
    Replies:
    4
    Views:
    1,079
    Thomas Stanka
    Oct 22, 2003
  4. Georges Konstantinidis

    asynchronous counter an Xilinx FPGA for a newbie

    Georges Konstantinidis, Jan 30, 2004, in forum: VHDL
    Replies:
    12
    Views:
    6,014
  5. HUANG Huan
    Replies:
    2
    Views:
    715
    Dave Higton
    Feb 24, 2004
Loading...

Share This Page