getting all HTTP headers from urllib2 Request?

Discussion in 'Python' started by cgoldberg, Mar 3, 2009.

  1. cgoldberg

    cgoldberg Guest

    I have a Python web client that uses urllib2. It is easy enough to
    add my own HTTP headers to the outgoing requests. I just create a
    dictionary of the headers I want to add, and pass it to the Request
    initializer.

    These custom headers are not all that gets sent. urllib2 attaches
    headers also. You can view the headers that urrlib2 adds by looking
    at unredirected_hdrs.

    However, these aren't the only HTTP headers that get sent on the
    wire. Other standard HTTP headers get added to outgoing requests as
    well as the custom ones I explicitly add and urllib2 adds. When I
    sniff the request using Wireshark, I see headers besides the ones I
    added myself.

    My question is how do a I get access to *all* of these headers? I
    want to log every request (including the full set of HTTP headers that
    get sent) that gets sent from my program, and can't figure out how.
    any pointers?

    In a nutshell: How do I get *all* the outgoing headers from an HTTP
    request created by urllib2?


    - Corey Goldberg
     
    cgoldberg, Mar 3, 2009
    #1
    1. Advertising

  2. cgoldberg

    Guest

    I didn't try it, but the Request Class from urllib2 has a method
    called, header_items(). That could be what your looking for.

    On Mar 3, 1:38 am, cgoldberg <> wrote:
    > I have a Python web client that uses urllib2.  It is easy enough to
    > add my own HTTP headers to the outgoing requests. I just create a
    > dictionary of the headers I want to add, and pass it to the Request
    > initializer.
    >
    > These custom headers are not all that gets sent.  urllib2 attaches
    > headers also.  You can view the headers that urrlib2 adds by looking
    > at unredirected_hdrs.
    >
    > However, these aren't the only HTTP headers that get sent on the
    > wire.  Other standard HTTP headers get added to outgoing requests as
    > well as the custom ones I explicitly add and urllib2 adds. When I
    > sniff the request using Wireshark, I see headers besides the ones I
    > added myself.
    >
    > My question is how do a I get access to *all* of these headers?  I
    > want to log every request (including the full set of HTTP headers that
    > get sent) that gets sent from my program, and can't figure out how.
    > any pointers?
    >
    > In a nutshell: How do I get *all* the outgoing headers from an HTTP
    > request created by urllib2?
    >
    > - Corey Goldberg
     
    , Mar 3, 2009
    #2
    1. Advertising

  3. cgoldberg

    cgoldberg Guest

    > I didn't try it, but the Request Class from urllib2 has a method
    > called, header_items(). That could be what your looking for.



    yes, that method only shows you all the headers added by urllib2.

    there are other headers that are produced by httplib under the covers
    that are added to the outgoing http request. That is what I am trying
    to get at.

    -Corey
     
    cgoldberg, Mar 3, 2009
    #3
  4. En Tue, 03 Mar 2009 13:44:12 -0200, cgoldberg <>
    escribió:

    >> I didn't try it, but the Request Class from urllib2 has a method
    >> called, header_items(). That could be what your looking for.

    >
    > yes, that method only shows you all the headers added by urllib2.
    >
    > there are other headers that are produced by httplib under the covers
    > that are added to the outgoing http request. That is what I am trying
    > to get at.


    Looking at the httplib sources, the only headers it may add are Host,
    Accept-Encoding: identity, and Content-Length.

    --
    Gabriel Genellina
     
    Gabriel Genellina, Mar 3, 2009
    #4
  5. cgoldberg

    cgoldberg Guest

    > Looking at the httplib sources, the only headers it may add are Host,  
    > Accept-Encoding: identity, and Content-Length.



    those are exactly the headers I want to capture.
    do you know how to get a hold of them from a request using urllib2.

    -Corey
     
    cgoldberg, Mar 3, 2009
    #5
  6. cgoldberg

    cgoldberg Guest

    > > Looking at the httplib sources, the only headers it may add are Host,  
    > > Accept-Encoding: identity, and Content-Length.



    now that I think of it, if it is only 3 headers, I can just override
    them explicitly from urllib2 and then log that.

    thanks a lot for looking into the httplib source!

    -Corey
     
    cgoldberg, Mar 3, 2009
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. nRk
    Replies:
    1
    Views:
    1,001
    Steven D'Aprano
    Feb 12, 2009
  2. Arthur Yousif

    Getting all http headers

    Arthur Yousif, Aug 4, 2003, in forum: ASP .Net Security
    Replies:
    1
    Views:
    140
    Arthur Yousif
    Aug 4, 2003
  3. Ian
    Replies:
    2
    Views:
    1,958
  4. Arthur

    Getting all http headers

    Arthur, Aug 3, 2003, in forum: ASP .Net Web Services
    Replies:
    0
    Views:
    121
    Arthur
    Aug 3, 2003
  5. icarreon
    Replies:
    0
    Views:
    1,739
    icarreon
    Jan 10, 2012
Loading...

Share This Page