Screen Scraper

[_eee_]

Does anyone know of a simple code module that can do screen scraping,
including simulating user-entered pushbuttons, etc.

I can get the first screen on a website with HttpWebRequest, but I
need to follow up to that by simulating user entries, then get
subsequent response screens to that entered data.

I figure someone has done this elegantly.

 

[Steve Chatham]

This works for me:

objRequest =
System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") & "&Rpt="
& sRpt & "&P=" & Session("Permission") & "&T=" &
cmbTerritory.SelectedItem.Value & "&O=" & cmbGroup.SelectedItem.Value &
"&R=" & cmbRegion.SelectedItem.Value & "&Chan=" & Chan & "&sSoldto=" &
ssoldto & "&sShipto=" & J6shipto & "&J7Periodselect=" & J7Period & "&J7Opt="
& J7Option & "&LC=" & J6lawcon & "&msoldto=" & msoldto & "&BA=" & xBA &
"&RG=" & xRG & "&FC=" & xFC & "&SC=" & xSC)

'objRequest.Credentials = New NetworkCredential("sql", "sql", "REMINGTON")

objRequest.Credentials = New NetworkCredential("username", "password",
"domain")

objResponse = objRequest.GetResponse()

Dim sr As New StreamReader(objResponse.GetResponseStream())

result = sr.ReadToEnd()

iLength = Len(result)

' commented out to debug 03/27/03

''''If InStr(result, "No records matched") > 0 Then

'''' result = "<br>No Records Matched <br>"

''''Else

'''' If sRpt = "Samples" Then

'''' Dim sb As New StringBuilder()

'''' sb.Append("<table border=""9"" ")

'''' iBegin = InStr(result, sb.ToString)

'''' Else

'''' iBegin = InStr(result, "<td><table border=")

'''' End If

'''' 'iBegin = InStr(result, "<td><table border=")

'''' '<table border="9" align="left"

'''' Dim iend = InStr(result, "<p><small>The report ran")

'''' result = Mid(result, iBegin, (iLength - ((iLength - iend) + iBegin)))

'''' result = result.Replace("<td><table border=""9"" align=""left""",
"<td><table border=""9"" align=""center""")

'''' 'result = "<align=""center"">" & result

''''End If

MyPage.Text = result

Button1.Visible = False



The commented code is out for debugging purposes, but I call an asp page
that parses out the stuff I call it with, and calls a stored procedure.



SC

 

[Peter Rilling]

This assumes that the request will accept parameters using a "GET" commend.
If the site you are capturing uses "POST", you may need to use the Headers
collection of the HttpWebRequest object. Never done this myself, so this is
about as much info as I can give you.

 

[_eee_]

[code snipped. Thanks, Steve!]

This assumes that the request will accept parameters using a "GET" commend.
If the site you are capturing uses "POST", you may need to use the Headers
collection of the HttpWebRequest object. Never done this myself, so this is
about as much info as I can give you.

Thanks to Steve for the posted code. But Peter is correct--I do need
to do this with 'Post'. I should have mentioned that.

I didn't think it would be an uncommon thing to do. Otherwise, how is
anyone scraping data from a site that requires entering text or
pushing a button?