view http info when browsing

[ben]

I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

Does anybody know a good way of view the raw http?

Thanks for your help,

Ben

 

[Peter van Kampen]

I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

from urllib import urlopen
r = urlopen("http://www.python.org/")
print r.headers

You can also take a look at mozilla firefox and download the excellent
webdeveloper extension. It also allows you to view headers and lots of other
stuff.

Hth,

PterK

 

[Paul McGuire]

I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

Does anybody know a good way of view the raw http?

Thanks for your help,

Ben

Try HTTPLook at www.httpsniffer.com . We've used it some on our current
project, to monitor SOAP message contents. I don't know about cookie
viewing, but it's free for 15-day (full featured) trial, less than $30 if
you want to keep it.

-- Paul

 

[Paul Rubin]

Does anybody know a good way of view the raw http?

I always just use a proxy server for that. www.junkbuster.org has a
free and convenient one that has options for logging the http stream.

 

[Michael Foord]

I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

Does anybody know a good way of view the raw http?

Thanks for your help,

Ben

TCPwatch should do what you want. It's a proxy serverso you'll have to
point your browser at that - it shows headers and page content
etc.....

If you're handling cookies then ClientCookie (or cookielib) could well
do what you want without you having to worry about it (if you're using
python why do you need a browser - why can't you just print the
headers ?)

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Lad]

I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

Does anybody know a good way of view the raw http?

Thanks for your help,

Ben

I downloaded a "sniffer" in Python. Let me know if you are interested

 

[[ EuGeNe ]]

give www.debugbar.com a bash

the trial version is fully functional and it is cheap

 

[John J. Lee]

I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

Does anybody know a good way of view the raw http?

[...]

from http://wwwsearch.sourceforge.net/bits/GeneralFAQ.html :

| I want to see what my web browser is doing, but standard network
| sniffers like ethereal or netcat (nc) don't work for HTTPS. How do I
| sniff HTTPS traffic?
|
| Three good options:
|
| Mozilla plugin: livehttpheaders.
| ieHTTPHeaders does the same for MSIE.
| Use lynx -trace, and filter out the junk with a script.
|
| I'm told you can also use a proxy like proxomitron (never tried it
| myself). There's also a commercial MSIE plugin.

URLs:

http://www.ethereal.com/

http://livehttpheaders.mozdev.org/

http://www.blunck.info/iehttpheaders.html

http://lynx.browser.org/

http://www.proxomitron.info/

http://www.simtec.ltd.uk/


There are plugins that show what session & permanent cookies your
browser has stored, too. Both Mozilla and Konqueror (& Safari and
Opera too, I guess?) do this as standard, but for IE:

http://www.codeproject.com/shell/cookiespy.asp?df=100&forumid=1989&exp=0&select=21591


John

 

[Michael]

Ben,

I am using iewatch (iewatch.com) quite a bit and it works great in
showing me the cookie information, both for request cookies and
response cookies (look for the Cookie and Set-Cookie headers). There
is a fully functional 21 day trial version available and it only costs
a fraction of what httpwatch is charging.

Michael