view http info when browsing

Discussion in 'Python' started by ben, Sep 6, 2004.

  1. ben

    ben Guest

    I'm trying to write a web client script in python to log onto a web
    page and pull some information off of it. The page has quite a few
    behind the scenes http things going on that are making it difficult to
    write the python script. I think if I could see the raw http data
    that comes to my browser life would be much easier. Does anybody know
    of programs that would run alongside my browser and show the http data
    that is send back and forth?

    I've looked at plugins like httpwatch and iewatch. httpwatch looks
    good but it costs quite a bit of money, and the trial is not fully
    functional. iewatch has a free trial, but they don't show cookie
    info, which is important.

    Does anybody know a good way of view the raw http?

    Thanks for your help,

    Ben
     
    ben, Sep 6, 2004
    #1
    1. Advertising

  2. In article <>, ben wrote:
    > I'm trying to write a web client script in python to log onto a web
    > page and pull some information off of it. The page has quite a few
    > behind the scenes http things going on that are making it difficult to
    > write the python script. I think if I could see the raw http data
    > that comes to my browser life would be much easier. Does anybody know
    > of programs that would run alongside my browser and show the http data
    > that is send back and forth?
    >
    > I've looked at plugins like httpwatch and iewatch. httpwatch looks
    > good but it costs quite a bit of money, and the trial is not fully
    > functional. iewatch has a free trial, but they don't show cookie
    > info, which is important.


    from urllib import urlopen
    r = urlopen("http://www.python.org/")
    print r.headers

    You can also take a look at mozilla firefox and download the excellent
    webdeveloper extension. It also allows you to view headers and lots of other
    stuff.

    Hth,

    PterK


    --
    Peter van Kampen
    pterk -- at -- datatailors.com
     
    Peter van Kampen, Sep 6, 2004
    #2
    1. Advertising

  3. ben

    Paul McGuire Guest

    "ben" <> wrote in message
    news:...
    > I'm trying to write a web client script in python to log onto a web
    > page and pull some information off of it. The page has quite a few
    > behind the scenes http things going on that are making it difficult to
    > write the python script. I think if I could see the raw http data
    > that comes to my browser life would be much easier. Does anybody know
    > of programs that would run alongside my browser and show the http data
    > that is send back and forth?
    >
    > I've looked at plugins like httpwatch and iewatch. httpwatch looks
    > good but it costs quite a bit of money, and the trial is not fully
    > functional. iewatch has a free trial, but they don't show cookie
    > info, which is important.
    >
    > Does anybody know a good way of view the raw http?
    >
    > Thanks for your help,
    >
    > Ben


    Try HTTPLook at www.httpsniffer.com . We've used it some on our current
    project, to monitor SOAP message contents. I don't know about cookie
    viewing, but it's free for 15-day (full featured) trial, less than $30 if
    you want to keep it.

    -- Paul
     
    Paul McGuire, Sep 6, 2004
    #3
  4. ben

    Paul Rubin Guest

    (ben) writes:
    > Does anybody know a good way of view the raw http?


    I always just use a proxy server for that. www.junkbuster.org has a
    free and convenient one that has options for logging the http stream.
     
    Paul Rubin, Sep 6, 2004
    #4
  5. (ben) wrote in message news:<>...
    > I'm trying to write a web client script in python to log onto a web
    > page and pull some information off of it. The page has quite a few
    > behind the scenes http things going on that are making it difficult to
    > write the python script. I think if I could see the raw http data
    > that comes to my browser life would be much easier. Does anybody know
    > of programs that would run alongside my browser and show the http data
    > that is send back and forth?
    >
    > I've looked at plugins like httpwatch and iewatch. httpwatch looks
    > good but it costs quite a bit of money, and the trial is not fully
    > functional. iewatch has a free trial, but they don't show cookie
    > info, which is important.
    >
    > Does anybody know a good way of view the raw http?
    >
    > Thanks for your help,
    >
    > Ben



    TCPwatch should do what you want. It's a proxy serverso you'll have to
    point your browser at that - it shows headers and page content
    etc.....

    If you're handling cookies then ClientCookie (or cookielib) could well
    do what you want without you having to worry about it (if you're using
    python why do you need a browser - why can't you just print the
    headers ?)

    Regards,

    Fuzzy

    http://www.voidspace.org.uk/atlantibots/pythonutils.html
     
    Michael Foord, Sep 6, 2004
    #5
  6. ben

    Lad Guest

    (ben) wrote in message news:<>...
    > I'm trying to write a web client script in python to log onto a web
    > page and pull some information off of it. The page has quite a few
    > behind the scenes http things going on that are making it difficult to
    > write the python script. I think if I could see the raw http data
    > that comes to my browser life would be much easier. Does anybody know
    > of programs that would run alongside my browser and show the http data
    > that is send back and forth?
    >
    > I've looked at plugins like httpwatch and iewatch. httpwatch looks
    > good but it costs quite a bit of money, and the trial is not fully
    > functional. iewatch has a free trial, but they don't show cookie
    > info, which is important.
    >
    > Does anybody know a good way of view the raw http?
    >
    > Thanks for your help,
    >
    > Ben


    I downloaded a "sniffer" in Python. Let me know if you are interested
     
    Lad, Sep 6, 2004
    #6
  7. ben

    [ EuGeNe ] Guest

    give www.debugbar.com a bash

    the trial version is fully functional and it is cheap
     
    [ EuGeNe ], Sep 6, 2004
    #7
  8. ben

    John J. Lee Guest

    (ben) writes:

    > I'm trying to write a web client script in python to log onto a web
    > page and pull some information off of it. The page has quite a few
    > behind the scenes http things going on that are making it difficult to
    > write the python script. I think if I could see the raw http data
    > that comes to my browser life would be much easier. Does anybody know
    > of programs that would run alongside my browser and show the http data
    > that is send back and forth?
    >
    > I've looked at plugins like httpwatch and iewatch. httpwatch looks
    > good but it costs quite a bit of money, and the trial is not fully
    > functional. iewatch has a free trial, but they don't show cookie
    > info, which is important.
    >
    > Does anybody know a good way of view the raw http?

    [...]

    from http://wwwsearch.sourceforge.net/bits/GeneralFAQ.html :

    | I want to see what my web browser is doing, but standard network
    | sniffers like ethereal or netcat (nc) don't work for HTTPS. How do I
    | sniff HTTPS traffic?
    |
    | Three good options:
    |
    | Mozilla plugin: livehttpheaders.
    | ieHTTPHeaders does the same for MSIE.
    | Use lynx -trace, and filter out the junk with a script.
    |
    | I'm told you can also use a proxy like proxomitron (never tried it
    | myself). There's also a commercial MSIE plugin.

    URLs:

    http://www.ethereal.com/

    http://livehttpheaders.mozdev.org/

    http://www.blunck.info/iehttpheaders.html

    http://lynx.browser.org/

    http://www.proxomitron.info/

    http://www.simtec.ltd.uk/


    There are plugins that show what session & permanent cookies your
    browser has stored, too. Both Mozilla and Konqueror (& Safari and
    Opera too, I guess?) do this as standard, but for IE:

    http://www.codeproject.com/shell/cookiespy.asp?df=100&forumid=1989&exp=0&select=21591


    John
     
    John J. Lee, Sep 8, 2004
    #8
  9. ben

    Michael Guest

    Ben,

    I am using iewatch (iewatch.com) quite a bit and it works great in
    showing me the cookie information, both for request cookies and
    response cookies (look for the Cookie and Set-Cookie headers). There
    is a fully functional 21 day trial version available and it only costs
    a fraction of what httpwatch is charging.

    Michael

    (ben) wrote in message news:<>...
    > I'm trying to write a web client script in python to log onto a web
    > page and pull some information off of it. The page has quite a few
    > behind the scenes http things going on that are making it difficult to
    > write the python script. I think if I could see the raw http data
    > that comes to my browser life would be much easier. Does anybody know
    > of programs that would run alongside my browser and show the http data
    > that is send back and forth?
    >
    > I've looked at plugins like httpwatch and iewatch. httpwatch looks
    > good but it costs quite a bit of money, and the trial is not fully
    > functional. iewatch has a free trial, but they don't show cookie
    > info, which is important.
    >
    > Does anybody know a good way of view the raw http?
    >
    > Thanks for your help,
    >
    > Ben
     
    Michael, Sep 9, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Vasanth
    Replies:
    0
    Views:
    565
    Vasanth
    Jun 28, 2004
  2. Andrea Williams

    Clear the Render info & change to different info

    Andrea Williams, Oct 27, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    1,428
    Ken Dopierala Jr.
    Oct 27, 2004
  3. William
    Replies:
    2
    Views:
    407
    Bob Kranson
    Sep 12, 2003
  4. Greg
    Replies:
    0
    Views:
    400
  5. Parthiv Joshi
    Replies:
    1
    Views:
    729
    Samuel L Matzen
    Jul 6, 2004
Loading...

Share This Page