Re: [Twisted-Python] Re-working a synchronous iterator to use Twisted

Discussion in 'Python' started by Jean-Paul Calderone, Jun 17, 2008.

  1. On Tue, 17 Jun 2008 23:10:48 +0200, Terry Jones <> wrote:
    >For the record, here's a followup to my own posting, with working code.
    >The earlier untested code was a bit of a mess. The below runs fine.
    >
    >In case it wasn't clear before, you're pulling "results" (e.g., from a
    >search engine) in off the web. Each results pages comes with an indicator
    >to tell you whether there are more results. I wanted to write a function
    >(see processResults below) that, when called, would call the process
    >function below on each result, all done asynchronously.
    >
    >This solution feels cumbersome, but it does work (aka prints the expected
    >output).
    >
    >Comments welcome (just don't tell me to use version control :))
    >


    I suspect the implementation could be slightly simplified, but it doesn't
    look too bad as it is now (and I'm feeling slightly too lazy to back that
    hunch up with code). I did want to point out that you're missing one
    tiny detail.

    > [snip]
    >
    ># ASYNCHRONOUS calling
    >def processResults(uri):
    > def cb((resultIterator, deferred)):
    > for result in resultIterator:
    > process(result)
    > if deferred is not None:
    > deferred.addCallback(cb)
    > return getResults(uri).addCallback(cb)
    >


    Here, the returned Deferred will fire as soon as `cb´ has a result. The
    return value of `cb´ is always `None´ though, so `cb´ will have a result
    synchronously in all cases. This is incorrect for the case where there
    are more results coming. Your example produced the correct output anyway,
    since all of your Deferreds are created already having results. If you
    had an "asynchronous" Deferred, you'd see your "finished" message before
    process had been called on all results.

    There are two possible solutions to this. The simpler one is to return
    `deferred´ from `cb´. The problem this has is that it builds up a chain
    of unbounded length which may ultimately encounter a limitation in the
    implementation of Deferred, hitting the Python stack depth limit and then
    failing with a RuntimeError.

    The only slightly more complex one is to create a new Deferred in
    `processResults´ and return it. Then, fire it inside `cb´ when `deferred´
    is None. You also need to be slightly careful to hook up the errback
    chain in this case, so that if there's some problem with getting an
    iterator the app-facing Deferred gets errbacked.

    Jean-Paul
    Jean-Paul Calderone, Jun 17, 2008
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rickert
    Replies:
    0
    Views:
    643
    Rickert
    Oct 6, 2011
  2. Replies:
    2
    Views:
    99
  3. Glyph
    Replies:
    0
    Views:
    165
    Glyph
    Dec 27, 2012
  4. Glyph
    Replies:
    0
    Views:
    140
    Glyph
    Apr 3, 2013
  5. Glyph
    Replies:
    0
    Views:
    40
    Glyph
    May 12, 2014
Loading...

Share This Page