Screen scraper again

[_eee_]

I recently posted a query about screen scraping, but haven't turned up
any leads yet. Here's what I need to do:

The first screen is retrieved via HttpWebRequest/Response. Easy
enough, as no parameters are required. But then I need to fill in
some fields from that initial screen and POST it back to the website
(to get back info on a specific subject).

It seems easier to do a GET, but unfortunately I need to do this via
POST.

Surely this has been done a lot, right? Any clues on where to look?

Are there other newsgroups that would be more appropriate for this
question?

 

[Eric Lawrence [MSFT]]

Are you saying you're having a problem doing the POST? The following
function posts to a website and returns a response stream. The body of the
post is passed as a string (sPOST). If you need help figuring out what
should be in the post body, spend some time watching what IE passes up using
a tool like www.fiddlertool.com provides.

public static Stream DoHTTPPost(string sURL, string sLang, string sPOST)
{
HttpWebRequest oRequest = (HttpWebRequest)WebRequest.Create(sURL);
oRequest.UserAgent= "Picker/1.0";
oRequest.Headers.Add("Accept-Language", sLang);
oRequest.Method = "POST";
oRequest.ContentType="text/xml";
StreamWriter myWriter = null;
Stream strmPost = oRequest.GetRequestStream();
try
{
myWriter = new StreamWriter(strmPost, System.Text.Encoding.UTF8);
myWriter.Write(sPOST);
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
finally
{
myWriter.Close();
}

return oRequest.GetResponse().GetResponseStream();
}


--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.

 

[_eee_]

Are you saying you're having a problem doing the POST? The following
function posts to a website and returns a response stream.
....

That was exactly what I was looking for, Eric!
Fiddlertool is also very useful.

Now, where did you find out how to do that?
Any books that cover it?

 

[Eric Lawrence [MSFT]]

I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book I
used constantly when I was writing Fiddler in C#.

Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.

 

[_eee_]

I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book I
used constantly when I was writing Fiddler in C#.

Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

You WROTE Fiddler? I should have put that together.
VERY nice program, Eric.

I'll look for the book.

 

[Eric Lawrence [MSFT]]

Thanks!

Please feel free to let me know (via the "Contact" link at fiddlertool.com)
if you have any suggestions for future Fiddler enhancements.

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.

 

[_eee_]

Thanks!

Please feel free to let me know (via the "Contact" link at fiddlertool.com)
if you have any suggestions for future Fiddler enhancements.

Eric Lawrence
Program Manager
Assistance and Worldwide Services

I shall, Eric. It looks like the immediate problem is solved,
and Fiddler was very helpful for that. I'll probably be using it more
in future Asp.net projects. (Others on this group should take a look)