getting all HTTP headers from urllib2 Request?

[cgoldberg]

I have a Python web client that uses urllib2. It is easy enough to
add my own HTTP headers to the outgoing requests. I just create a
dictionary of the headers I want to add, and pass it to the Request
initializer.

These custom headers are not all that gets sent. urllib2 attaches
headers also. You can view the headers that urrlib2 adds by looking
at unredirected_hdrs.

However, these aren't the only HTTP headers that get sent on the
wire. Other standard HTTP headers get added to outgoing requests as
well as the custom ones I explicitly add and urllib2 adds. When I
sniff the request using Wireshark, I see headers besides the ones I
added myself.

My question is how do a I get access to *all* of these headers? I
want to log every request (including the full set of HTTP headers that
get sent) that gets sent from my program, and can't figure out how.
any pointers?

In a nutshell: How do I get *all* the outgoing headers from an HTTP
request created by urllib2?


- Corey Goldberg

 

[zdxerr]

I didn't try it, but the Request Class from urllib2 has a method
called, header_items(). That could be what your looking for.

 

[cgoldberg]

I didn't try it, but the Request Class from urllib2 has a method

called, header_items(). That could be what your looking for.

yes, that method only shows you all the headers added by urllib2.

there are other headers that are produced by httplib under the covers
that are added to the outgoing http request. That is what I am trying
to get at.

-Corey

 

[Gabriel Genellina]

yes, that method only shows you all the headers added by urllib2.

there are other headers that are produced by httplib under the covers
that are added to the outgoing http request. That is what I am trying
to get at.

Looking at the httplib sources, the only headers it may add are Host,
Accept-Encoding: identity, and Content-Length.

 

[cgoldberg]

Looking at the httplib sources, the only headers it may add are Host,  

Accept-Encoding: identity, and Content-Length.

those are exactly the headers I want to capture.
do you know how to get a hold of them from a request using urllib2.

-Corey

 

[cgoldberg]

Looking at the httplib sources, the only headers it may add are Host,  

now that I think of it, if it is only 3 headers, I can just override
them explicitly from urllib2 and then log that.

thanks a lot for looking into the httplib source!

-Corey