futures - a new package for asynchronous execution

Discussion in 'Python' started by Brian Quinlan, Nov 6, 2009.

  1. Hey all,

    I recently implemented a package that I'd like to have include in the
    Python 3.x standard library (and maybe Python 2.x) and I'd love to
    have the feedback of this list.

    The basic idea is to implement an asynchronous execution method
    patterned heavily on java.util.concurrent (but less lame because
    Python has functions as first-class objects). Here is a fairly
    advanced example:

    import futures
    import functools
    import urllib.request

    URLS = [
    'http://www.foxnews.com/',
    'http://www.cnn.com/',
    'http://europe.wsj.com/',
    'http://www.bbc.co.uk/',
    'http://some-made-up-domain.com/']

    def load_url(url, timeout):
    return urllib.request.urlopen(url, timeout=timeout).read()

    # Use a thread pool with 5 threads to download the URLs. Using a pool
    # of processes would involve changing the initialization to:
    # with futures.ProcessPoolExecutor(max_processes=5) as executor
    with futures.ThreadPoolExecutor(max_threads=5) as executor:
    future_list = executor.run_to_futures(
    [functools.partial(load_url, url, 30) for url in URLS])

    # Check the results of each future.
    for url, future in zip(URLS, future_list):
    if future.exception() is not None:
    print('%r generated an exception: %s' % (url,
    future.exception()))
    else:
    print('%r page is %d bytes' % (url, len(future.result())))

    In this example, executor.run_to_futures() returns only when every url
    has been retrieved but it is possible to return immediately, on the
    first completion or on the first failure depending on the desired work
    pattern.

    The complete docs are here:
    http://sweetapp.com/futures/

    A draft PEP is here:
    http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt

    And the code is here:
    http://pypi.python.org/pypi/futures3/

    All feedback appreciated!

    Cheers,
    Brian
     
    Brian Quinlan, Nov 6, 2009
    #1
    1. Advertising

  2. Brian Quinlan

    Aahz Guest

    In article <>,
    Brian Quinlan <> wrote:
    >
    >I recently implemented a package that I'd like to have include in the
    >Python 3.x standard library (and maybe Python 2.x) and I'd love to
    >have the feedback of this list.


    Any recently implemented library has an extremely high bar before it gets
    adopted into the standard library. You should concentrate on persuading
    people to use your library; for example, you should make regular feature
    and bugfix releases and announce them on c.l.py.announce.
    --
    Aahz () <*> http://www.pythoncraft.com/

    [on old computer technologies and programmers] "Fancy tail fins on a
    brand new '59 Cadillac didn't mean throwing out a whole generation of
    mechanics who started with model As." --Andrew Dalke
     
    Aahz, Nov 10, 2009
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    543
    Thomas Schodt
    May 11, 2005
  2. JJ
    Replies:
    1
    Views:
    520
    Steven Cheng[MSFT]
    Oct 15, 2007
  3. Paddy
    Replies:
    1
    Views:
    333
    Terry Reedy
    Feb 22, 2011
  4. Nick Brown
    Replies:
    6
    Views:
    116
    Robert Klemme
    Feb 28, 2010
  5. Asynchronous perl execution

    , Apr 11, 2007, in forum: Perl Misc
    Replies:
    2
    Views:
    95
Loading...

Share This Page