Making urllib2's POST 302 handle same as Perl LWP's behaviour

[Karra]

I am doing a POST to a webserver and get a 302 Found response
(redirect). urllib2's default behaviour is to do a GET on the new url
from the Location: URI in the 302 response.

This is different from what I have found with LWP::UserAgent-

request() in perl. After much searching I understand there is a view

that automatic redirection for a 302 in response to a POST is not in
conformance to the relevant RFCs. Therefore, I believe urllib2's
behaviour appears to be non-conformant (as, I believe are many
browsers).

Now, regardless of what is the "correct" approach to handling the 302,
there is some information in the returned html of the 302 which I am
losing because of the subsequent GET. I tried to raise a HTTPError
from redirect_request() but that just kills the connection with the
server. I tried returning None, same result.

Can someone point me to how I can get the default LWP:UserAgent
behaviour of handling this scenario using urllib2?

 

[Karra]

Can someone point me to how I can get the default LWP:UserAgent
behaviour of handling this scenario using urllib2?

Out of frustration, I decided to give 'mechanize' a try. It came as an
awesome surprise that mechanize implements the exact api of urllib2 -
meaning all I had to do was a query-replace of urllib2 to mechanize,
and lo-and-behold I got what I wanted!

Which is to say, the behaviour of mechanize matches Perl's
LWP::UserAgent as afar as handling of POST redirects go.

 

[Octavian Rasnita]

I have tried:

In httpd.conf:

Redirect /one/ http://localhost/two/
Redirect /two/ http://localhost/three/
redirect /three/ http://www.google.com/

Then I made a POST request with LWP::UserAgent to /one:

use LWP::UserAgent;
print LWP::UserAgent->new->post('http://localhost/one/')->as_string;

And the result was:
HTTP/1.1 302 Found
Connection: close
Date: Wed, 29 Dec 2010 13:58:37 GMT
Location: http://localhost/two/
Server: Apache/2.2.15 (Win32) mod_perl/2.0.4-dev Perl/v5.10.1
....

Exactly as you described.

Then I made a POST request with WWW::Mechanize:

use WWW::Mechanize;
print WWW::Mechanize->new->post('http://localhost/one/')->as_string;

And the result was:
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Wed, 29 Dec 2010 14:00:44 GMT
Server: gws

So it seems that LWP::UserAgent works similarly with mechanize and WWW::Mechanize similarly to urllib. Nice.

Octavian

----- Original Message -----
From: "Karra" <[email protected]>
Newsgroups: comp.lang.python
To: <[email protected]>
Sent: Wednesday, December 29, 2010 2:35 PM
Subject: Re: Making urllib2's POST 302 handle same as Perl LWP's behaviour

Can someone point me to how I can get the default LWP:UserAgent
behaviour of handling this scenario using urllib2?

Out of frustration, I decided to give 'mechanize' a try. It came as an
awesome surprise that mechanize implements the exact api of urllib2 -
meaning all I had to do was a query-replace of urllib2 to mechanize,
and lo-and-behold I got what I wanted!

Which is to say, the behaviour of mechanize matches Perl's
LWP::UserAgent as afar as handling of POST redirects go.