Alternatives to LWP::Parallel

Discussion in 'Perl Misc' started by Peter, Oct 24, 2009.

  1. Peter

    Peter Guest

    I'm in the process of writing a script that will take a whole bunch of
    URL's and get their HTTP status. I was going to use
    LWP::parallel::UserAgent to handle parallel HTTP requests. However, I
    found that this module does not work with later version of libwww and
    it looks like this module is no longer maintained.

    So I started looking around at other methods/modules I could use.
    There are certainly quite a few to choose from and some are quite
    complex for my needs.

    Some of the modules I looked at were:

    IO::Lambda::HTTP (this looked the most robust/well maintained, but has
    a learning curve to it)
    POE::Component::Client::HTTP (as above)
    AnyEvent::HTTP (this looked promising, but I couldn't get any results
    with it)
    HTTP::Client::parallel (doesn't appear to have any function to limit
    number of connections)
    HTTP::Async (a little buggy and didn't seem to produce expected
    results).

    One essential feature I need is the ability to throttle/limit the
    number of simultaneous connections or number of connections/requests
    per minute.

    I'd appreciate any advice as to what method/module might be best for
    me. The other governing factor is that I need it to run with
    ActivePerl on Linux and Windows.
     
    Peter, Oct 24, 2009
    #1
    1. Advertising

  2. Peter

    J. Gleixner Guest

    Peter wrote:
    > I'm in the process of writing a script that will take a whole bunch of
    > URL's and get their HTTP status. I was going to use
    > LWP::parallel::UserAgent to handle parallel HTTP requests. However, I
    > found that this module does not work with later version of libwww and
    > it looks like this module is no longer maintained.
    >
    > So I started looking around at other methods/modules I could use.
    > There are certainly quite a few to choose from and some are quite
    > complex for my needs.
    >
    > Some of the modules I looked at were:


    Why not simply Parallel::ForkManager and LWP::UserAgent?

    [...]
     
    J. Gleixner, Oct 26, 2009
    #2
    1. Advertising

  3. Peter

    Peter Guest

    On 27 Oct, 06:02, "J. Gleixner" <>
    wrote:
    > Peter wrote:
    > > I'm in the process of writing a script that will take a whole bunch of
    > > URL's and get their HTTP status.  I was going to use
    > > LWP::parallel::UserAgent to handle parallel HTTP requests.  However, I
    > > found that this module does not work with later version of libwww and
    > > it looks like this module is no longer maintained.

    >
    > > So I started looking around at other methods/modules I could use.
    > > There are certainly quite a few to choose from and some are quite
    > > complex for my needs.

    >
    > > Some of the modules I looked at were:

    >
    > Why not simply Parallel::ForkManager and LWP::UserAgent?
    >
    > [...]



    Ah, now that looks to be just what I need. I don't know how I managed
    to miss that one. Thanks.
     
    Peter, Oct 26, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    538
  2. Zeke Koos
    Replies:
    0
    Views:
    204
    Zeke Koos
    Sep 15, 2003
  3. Zeke Koos
    Replies:
    0
    Views:
    94
    Zeke Koos
    Sep 16, 2003
  4. Alex
    Replies:
    0
    Views:
    218
  5. Leif Wessman
    Replies:
    0
    Views:
    96
    Leif Wessman
    Jun 16, 2004
Loading...

Share This Page