HTTP Listener??

[Rusty Hill]

Is there some tool that I can use to profile all the activity HTML, cookies,
headers, etc. that are sent back and forth to a site when I access it using
my web browser? Essentially I want to navigate a site manually in my
browser and get enough information so I can program something to crawl the
site in an automated fashion but need to know what headers and stuff to send
in my request. Thank you.

 

[Daniel Cigic]

IE Developer Toolbar

 

[Nicholas Paldino [.NET/C# MVP]]

Rusty,

There are a few programs out there that will let you do this (I'm sure
there is even a firefox extension if that is your browser of choice).

For any kind of web browser though, you can use something like Fiddler
(which is free). All you have to do is run the program, and then set your
proxy settings on your web browser to point to the port and ip that the
interceptor is running on (usually the local host).

 

[Samuel R. Neff]

Charles will record an HTTP session (set of requests) and then replay
them. That sounds like what you're asking for, right?

http://www.xk72.com/charles/

I haven't used that aspec of Charles but in general it's a wonderful
app.

Sam

 

[RedGrittyBrick]

Is there some tool that I can use to profile all the activity HTML, cookies,
headers, etc. that are sent back and forth to a site when I access it using
my web browser? Essentially I want to navigate a site manually in my
browser and get enough information so I can program something to crawl the
site in an automated fashion but need to know what headers and stuff to send
in my request. Thank you.

Wireshark?

 

[Steve C. Orr [MCSD, MVP, CSM, ASP Insider]]

I suggest Fiddler.
It's free and good.
http://www.fiddler2.com