Screen Scraper

Discussion in 'ASP .Net' started by _eee_, Feb 25, 2004.

  1. _eee_

    _eee_ Guest

    Does anyone know of a simple code module that can do screen scraping,
    including simulating user-entered pushbuttons, etc.

    I can get the first screen on a website with HttpWebRequest, but I
    need to follow up to that by simulating user entries, then get
    subsequent response screens to that entered data.

    I figure someone has done this elegantly.
    _eee_, Feb 25, 2004
    #1
    1. Advertising

  2. This works for me:

    objRequest =
    System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
    asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") & "&Rpt="
    & sRpt & "&P=" & Session("Permission") & "&T=" &
    cmbTerritory.SelectedItem.Value & "&O=" & cmbGroup.SelectedItem.Value &
    "&R=" & cmbRegion.SelectedItem.Value & "&Chan=" & Chan & "&sSoldto=" &
    ssoldto & "&sShipto=" & J6shipto & "&J7Periodselect=" & J7Period & "&J7Opt="
    & J7Option & "&LC=" & J6lawcon & "&msoldto=" & msoldto & "&BA=" & xBA &
    "&RG=" & xRG & "&FC=" & xFC & "&SC=" & xSC)

    'objRequest.Credentials = New NetworkCredential("sql", "sql", "REMINGTON")

    objRequest.Credentials = New NetworkCredential("username", "password",
    "domain")

    objResponse = objRequest.GetResponse()

    Dim sr As New StreamReader(objResponse.GetResponseStream())

    result = sr.ReadToEnd()

    iLength = Len(result)

    ' commented out to debug 03/27/03

    ''''If InStr(result, "No records matched") > 0 Then

    '''' result = "<br>No Records Matched <br>"

    ''''Else

    '''' If sRpt = "Samples" Then

    '''' Dim sb As New StringBuilder()

    '''' sb.Append("<table border=""9"" ")

    '''' iBegin = InStr(result, sb.ToString)

    '''' Else

    '''' iBegin = InStr(result, "<td><table border=")

    '''' End If

    '''' 'iBegin = InStr(result, "<td><table border=")

    '''' '<table border="9" align="left"

    '''' Dim iend = InStr(result, "<p><small>The report ran")

    '''' result = Mid(result, iBegin, (iLength - ((iLength - iend) + iBegin)))

    '''' result = result.Replace("<td><table border=""9"" align=""left""",
    "<td><table border=""9"" align=""center""")

    '''' 'result = "<align=""center"">" & result

    ''''End If

    MyPage.Text = result

    Button1.Visible = False



    The commented code is out for debugging purposes, but I call an asp page
    that parses out the stuff I call it with, and calls a stored procedure.



    SC

    "_eee_" <> wrote in message
    news:...
    > Does anyone know of a simple code module that can do screen scraping,
    > including simulating user-entered pushbuttons, etc.
    >
    > I can get the first screen on a website with HttpWebRequest, but I
    > need to follow up to that by simulating user entries, then get
    > subsequent response screens to that entered data.
    >
    > I figure someone has done this elegantly.
    Steve Chatham, Feb 25, 2004
    #2
    1. Advertising

  3. This assumes that the request will accept parameters using a "GET" commend.
    If the site you are capturing uses "POST", you may need to use the Headers
    collection of the HttpWebRequest object. Never done this myself, so this is
    about as much info as I can give you.

    "Steve Chatham" <> wrote in message
    news:Ot3BI59#...
    > This works for me:
    >
    > objRequest =
    >

    System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
    > asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") &

    "&Rpt="
    > & sRpt & "&P=" & Session("Permission") & "&T=" &
    > cmbTerritory.SelectedItem.Value & "&O=" & cmbGroup.SelectedItem.Value &
    > "&R=" & cmbRegion.SelectedItem.Value & "&Chan=" & Chan & "&sSoldto=" &
    > ssoldto & "&sShipto=" & J6shipto & "&J7Periodselect=" & J7Period &

    "&J7Opt="
    > & J7Option & "&LC=" & J6lawcon & "&msoldto=" & msoldto & "&BA=" & xBA &
    > "&RG=" & xRG & "&FC=" & xFC & "&SC=" & xSC)
    >
    > 'objRequest.Credentials = New NetworkCredential("sql", "sql", "REMINGTON")
    >
    > objRequest.Credentials = New NetworkCredential("username", "password",
    > "domain")
    >
    > objResponse = objRequest.GetResponse()
    >
    > Dim sr As New StreamReader(objResponse.GetResponseStream())
    >
    > result = sr.ReadToEnd()
    >
    > iLength = Len(result)
    >
    > ' commented out to debug 03/27/03
    >
    > ''''If InStr(result, "No records matched") > 0 Then
    >
    > '''' result = "<br>No Records Matched <br>"
    >
    > ''''Else
    >
    > '''' If sRpt = "Samples" Then
    >
    > '''' Dim sb As New StringBuilder()
    >
    > '''' sb.Append("<table border=""9"" ")
    >
    > '''' iBegin = InStr(result, sb.ToString)
    >
    > '''' Else
    >
    > '''' iBegin = InStr(result, "<td><table border=")
    >
    > '''' End If
    >
    > '''' 'iBegin = InStr(result, "<td><table border=")
    >
    > '''' '<table border="9" align="left"
    >
    > '''' Dim iend = InStr(result, "<p><small>The report ran")
    >
    > '''' result = Mid(result, iBegin, (iLength - ((iLength - iend) + iBegin)))
    >
    > '''' result = result.Replace("<td><table border=""9"" align=""left""",
    > "<td><table border=""9"" align=""center""")
    >
    > '''' 'result = "<align=""center"">" & result
    >
    > ''''End If
    >
    > MyPage.Text = result
    >
    > Button1.Visible = False
    >
    >
    >
    > The commented code is out for debugging purposes, but I call an asp page
    > that parses out the stuff I call it with, and calls a stored procedure.
    >
    >
    >
    > SC
    >
    > "_eee_" <> wrote in message
    > news:...
    > > Does anyone know of a simple code module that can do screen scraping,
    > > including simulating user-entered pushbuttons, etc.
    > >
    > > I can get the first screen on a website with HttpWebRequest, but I
    > > need to follow up to that by simulating user entries, then get
    > > subsequent response screens to that entered data.
    > >
    > > I figure someone has done this elegantly.

    >
    >
    Peter Rilling, Feb 25, 2004
    #3
  4. _eee_

    _eee_ Guest

    >> "_eee_" <> wrote in message
    >> > Does anyone know of a simple code module that can do screen scraping,
    >> > including simulating user-entered pushbuttons, etc.
    >> >
    >> > I can get the first screen on a website with HttpWebRequest, but I
    >> > need to follow up to that by simulating user entries, then get
    >> > subsequent response screens to that entered data.


    >"Steve Chatham" <> wrote in message
    >news:Ot3BI59#...
    >> objRequest =
    >>System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
    >> asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") &


    [code snipped. Thanks, Steve!]

    <> wrote:
    >This assumes that the request will accept parameters using a "GET" commend.
    >If the site you are capturing uses "POST", you may need to use the Headers
    >collection of the HttpWebRequest object. Never done this myself, so this is
    >about as much info as I can give you.


    Thanks to Steve for the posted code. But Peter is correct--I do need
    to do this with 'Post'. I should have mentioned that.

    I didn't think it would be an uncommon thing to do. Otherwise, how is
    anyone scraping data from a site that requires entering text or
    pushing a button?
    _eee_, Feb 25, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. _eee_

    Screen scraper again

    _eee_, Feb 27, 2004, in forum: ASP .Net
    Replies:
    6
    Views:
    442
    _eee_
    Feb 28, 2004
  2. Dave Monroe

    Screen Scraper with Java API

    Dave Monroe, Oct 17, 2003, in forum: Java
    Replies:
    1
    Views:
    679
    Richard Reynolds
    Oct 17, 2003
  3. James Stroud

    Python Screen Scraper

    James Stroud, Apr 24, 2007, in forum: Python
    Replies:
    7
    Views:
    569
    skotjs
    Apr 25, 2007
  4. my first screen scraper

    , Dec 2, 2007, in forum: Python
    Replies:
    0
    Views:
    488
  5. doog
    Replies:
    8
    Views:
    147
Loading...

Share This Page