Parallel::UserAgent with Redirect / How to get the original URL??

F

fritz-bayer

Hi,

I'm using the Parrallel::UserAgent to send several HTTP requests at
once. After that I circle through the result and process the content.

I have redirection turned on, so the agent follows redirects. The
problem I have is retrieving the original urls of the requests.

The method:

my $redirectedUrl = $response->request->url;

will return the redirected url instead of the original one.

How can I retrieve the orginal url of the request, before it was
redirected. Of course I registered the url before, but I do not know,
which url caused the redirect.

Anybody who knows, where the original url can be found? Or is it
discarded?

Fritz
 
X

xhoster

Hi,

I'm using the Parrallel::UserAgent to send several HTTP requests at
once. After that I circle through the result and process the content.

I have redirection turned on, so the agent follows redirects. The
problem I have is retrieving the original urls of the requests.

The method:

my $redirectedUrl = $response->request->url;

will return the redirected url instead of the original one.

It would help if you showed us how you get $response. I can only guess
you got it from the hash returned by wait?
How can I retrieve the orginal url of the request, before it was
redirected. Of course I registered the url before, but I do not know,
which url caused the redirect.

Presumably you know the stringified value of the reference of the request
object which caused the redirect (or where else did response come from?).
So you just need to store the URL of keyed to the same stringified
reference.

Of you could navigate the LWP::parallel::UserAgent::Entry object:

$response->response->previous->request->url();

Xho
 
J

John Bokma

Hi,

I'm using the Parrallel::UserAgent to send several HTTP requests at
once. After that I circle through the result and process the content.

I have redirection turned on, so the agent follows redirects. The
problem I have is retrieving the original urls of the requests.

wait returns a hash reference, the keys are the stringified requests (a
request is a reference) you registered. So when you register a request,
add the stringified request (reference) to a hash table, and use the url
as a value:

my %look_up;
my $request = ....

$pua->register( $request )

$look_up{ $request } = $url;

my $entries = $pua->wait;
for my $key ( keys %$entries ) {

:
print "The original URL is: $look_up{ $key }\n";
:
}

(untested & from top of my head)

Yeah, it's a bit clumsy IMO. No idea if there is an easier way.
 
F

fritz-bayer

John said:
wait returns a hash reference, the keys are the stringified requests (a
request is a reference) you registered. So when you register a request,
add the stringified request (reference) to a hash table, and use the url
as a value:

my %look_up;
my $request = ....

$pua->register( $request )

$look_up{ $request } = $url;

my $entries = $pua->wait;
for my $key ( keys %$entries ) {

:
print "The original URL is: $look_up{ $key }\n";
:
}

(untested & from top of my head)

Yeah, it's a bit clumsy IMO. No idea if there is an easier way.

Thanks a lot guys! And yes I'm retrieving them from the hash, which
wait returns...
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top