Screen scraper again

Discussion in 'ASP .Net' started by _eee_, Feb 27, 2004.

  1. _eee_

    _eee_ Guest

    I recently posted a query about screen scraping, but haven't turned up
    any leads yet. Here's what I need to do:

    The first screen is retrieved via HttpWebRequest/Response. Easy
    enough, as no parameters are required. But then I need to fill in
    some fields from that initial screen and POST it back to the website
    (to get back info on a specific subject).

    It seems easier to do a GET, but unfortunately I need to do this via
    POST.

    Surely this has been done a lot, right? Any clues on where to look?

    Are there other newsgroups that would be more appropriate for this
    question?
     
    _eee_, Feb 27, 2004
    #1
    1. Advertising

  2. Are you saying you're having a problem doing the POST? The following
    function posts to a website and returns a response stream. The body of the
    post is passed as a string (sPOST). If you need help figuring out what
    should be in the post body, spend some time watching what IE passes up using
    a tool like www.fiddlertool.com provides.

    public static Stream DoHTTPPost(string sURL, string sLang, string sPOST)
    {
    HttpWebRequest oRequest = (HttpWebRequest)WebRequest.Create(sURL);
    oRequest.UserAgent= "Picker/1.0";
    oRequest.Headers.Add("Accept-Language", sLang);
    oRequest.Method = "POST";
    oRequest.ContentType="text/xml";
    StreamWriter myWriter = null;
    Stream strmPost = oRequest.GetRequestStream();
    try
    {
    myWriter = new StreamWriter(strmPost, System.Text.Encoding.UTF8);
    myWriter.Write(sPOST);
    }
    catch (Exception e)
    {
    MessageBox.Show(e.Message);
    }
    finally
    {
    myWriter.Close();
    }

    return oRequest.GetResponse().GetResponseStream();
    }


    --
    Thanks,

    Eric Lawrence
    Program Manager
    Assistance and Worldwide Services

    This posting is provided "AS IS" with no warranties, and confers no rights.


    "_eee_" <> wrote in message
    news:...
    > I recently posted a query about screen scraping, but haven't turned up
    > any leads yet. Here's what I need to do:
    >
    > The first screen is retrieved via HttpWebRequest/Response. Easy
    > enough, as no parameters are required. But then I need to fill in
    > some fields from that initial screen and POST it back to the website
    > (to get back info on a specific subject).
    >
    > It seems easier to do a GET, but unfortunately I need to do this via
    > POST.
    >
    > Surely this has been done a lot, right? Any clues on where to look?
    >
    > Are there other newsgroups that would be more appropriate for this
    > question?
    >
     
    Eric Lawrence [MSFT], Feb 27, 2004
    #2
    1. Advertising

  3. _eee_

    _eee_ Guest

    On Thu, 26 Feb 2004 18:49:07 -0800, "Eric Lawrence [MSFT]"
    <> wrote:

    >Are you saying you're having a problem doing the POST? The following
    >function posts to a website and returns a response stream.
    >....


    That was exactly what I was looking for, Eric!
    Fiddlertool is also very useful.

    Now, where did you find out how to do that?
    Any books that cover it?
     
    _eee_, Feb 27, 2004
    #3
  4. I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book I
    used constantly when I was writing Fiddler in C#.

    Thanks,

    Eric Lawrence
    Program Manager
    Assistance and Worldwide Services

    This posting is provided "AS IS" with no warranties, and confers no rights.

    "_eee_" <> wrote in message
    news:...
    >
    > On Thu, 26 Feb 2004 18:49:07 -0800, "Eric Lawrence [MSFT]"
    > <> wrote:
    >
    > >Are you saying you're having a problem doing the POST? The following
    > >function posts to a website and returns a response stream.
    > >....

    >
    > That was exactly what I was looking for, Eric!
    > Fiddlertool is also very useful.
    >
    > Now, where did you find out how to do that?
    > Any books that cover it?
    >
    >
     
    Eric Lawrence [MSFT], Feb 27, 2004
    #4
  5. _eee_

    _eee_ Guest

    On Fri, 27 Feb 2004 13:36:06 -0800, "Eric Lawrence [MSFT]"
    <> wrote:

    >I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book I
    >used constantly when I was writing Fiddler in C#.
    >
    >Thanks,
    >
    >Eric Lawrence
    >Program Manager
    >Assistance and Worldwide Services


    You WROTE Fiddler? I should have put that together.
    VERY nice program, Eric.

    I'll look for the book.
     
    _eee_, Feb 27, 2004
    #5
  6. Thanks!

    Please feel free to let me know (via the "Contact" link at fiddlertool.com)
    if you have any suggestions for future Fiddler enhancements.

    Eric Lawrence
    Program Manager
    Assistance and Worldwide Services

    This posting is provided "AS IS" with no warranties, and confers no rights.

    "_eee_" <> wrote in message
    news:...
    > On Fri, 27 Feb 2004 13:36:06 -0800, "Eric Lawrence [MSFT]"
    > <> wrote:
    >
    > >I recommend O'Reilly's "HTTP: The Definitive Reference", which is the

    book I
    > >used constantly when I was writing Fiddler in C#.
    > >
    > >Thanks,
    > >
    > >Eric Lawrence
    > >Program Manager
    > >Assistance and Worldwide Services

    >
    > You WROTE Fiddler? I should have put that together.
    > VERY nice program, Eric.
    >
    > I'll look for the book.
    >
     
    Eric Lawrence [MSFT], Feb 28, 2004
    #6
  7. _eee_

    _eee_ Guest

    On Sat, 28 Feb 2004 00:02:26 -0800, "Eric Lawrence [MSFT]"
    <> wrote:

    >Thanks!
    >
    >Please feel free to let me know (via the "Contact" link at fiddlertool.com)
    >if you have any suggestions for future Fiddler enhancements.
    >
    >Eric Lawrence
    >Program Manager
    >Assistance and Worldwide Services


    I shall, Eric. It looks like the immediate problem is solved,
    and Fiddler was very helpful for that. I'll probably be using it more
    in future Asp.net projects. (Others on this group should take a look)
     
    _eee_, Feb 28, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. _eee_

    Screen Scraper

    _eee_, Feb 25, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    597
    _eee_
    Feb 25, 2004
  2. Dave Monroe

    Screen Scraper with Java API

    Dave Monroe, Oct 17, 2003, in forum: Java
    Replies:
    1
    Views:
    689
    Richard Reynolds
    Oct 17, 2003
  3. James Stroud

    Python Screen Scraper

    James Stroud, Apr 24, 2007, in forum: Python
    Replies:
    7
    Views:
    580
    skotjs
    Apr 25, 2007
  4. my first screen scraper

    , Dec 2, 2007, in forum: Python
    Replies:
    0
    Views:
    521
  5. doog
    Replies:
    8
    Views:
    150
Loading...

Share This Page