Scraping data from a web form submit

[Brent]

Hi,
I want to build an app (C#, windows app, or web app, shouldn't make a
difference) that submits an address to the canada post website's address
lookup and then scrape the postal code out of the resulting page. Here is
their page. http://www.canadapost.ca/tools/pcl/bin/advanced-e.asp It uses a
form post.

I would like a basic rundown of how to submit my own info into their results
page (post), and then scrape out the info I need.
Thanks,
Brent

 

[bruce barker]

1) use webclient class to call the website.
2) hit the page and view source to see what the input control names are
3) post the page with the values you want.
4) write code to parse the returning html for the values you want. a few reg
expressions ought to work.

-- bruce (sqlwork.com)

 

[Scott Allen]

in addition to Bruce's info, there is a tool (Fiddler) which is a
debugging proxy - it will capture all the HTTP traffic and allow you
to analyze it. This can be a great help when figuring out what you
need to POST.

http://www.fiddlertool.com/fiddler/

 

[Brent]

Thanks for the help, I got it now. Here is the full thing of code that I
used...
System.Collections.Specialized.NameValueCollection values = new System.

Collections.Specialized.NameValueCollection();

values.Add("street_number","113");

values.Add("street_name","MCINNES");

values.Add("prov","AB");

WebClient webclient = new WebClient();

webclient.Headers.Add("Content-Type","application/x-www-form-urlencoded");

byte[] responseArray = webclient.UploadValues(

"http://www.canadapost.ca/tools/pcl/bin/cp_search_response-e.asp","POST",

values);

string response = System.Text.Encoding.ASCII.GetString(responseArray);

this.message_textBox.Text=
response.Substring(response.IndexOf("AB",response.

IndexOf("PCLFormattedAddress"))+4,7);

 

[clintonG]

Aside from your current approach if I were you I would also ask
canadapost.ca if they intend to expose the data using web services
as every time their page changes you may have to rebuild your scraper.

 

[randi2160]

Need Help

Hi, i need assistance in scraping data and placing it into a database.
I am looking for an automative way of Going to a webpage, Scraping it for the contact information like Name, address, email and website address .... bascially scraping yellow page data and post it in my own database

1. Go to ex: pluralism.org/
2. On the page that list Hindumandirs by state simply go to those pages and each link and scrape the name of mandir, address of mandir and contac information. Image for each correspodning post will be cool too.
3. Take that data and insert it into my database

Can someone give me a code to do that?

 

[Richard.Williams]

Scraping name, address, telephone info from web pages

Hi Randi2160:

There is a very applicable discussion on this topic at google biterscripting group

groups.google.com/group/biterscripting/browse_thread/thread/c2d3e7d953b7dc10

The scripts posted there do exactly what you need. There are two scripts

• Script page.txt
  Extracts info from one yellow page, puts info in an excel file. You can change this script to save the extracted info in text file, generate address labels, store in a database, etc.
  
• Script pageloop.txt
  Goes thru pages of yellowpages one by one, calls the page.txt script to extract info on one page at a time.

Overall, you can scrape info from hundreds of yellow pages in close to an hour. Make sure you have proper authorization from the owner of the site from which you are scraping information.

Start a new thread at that google group if you need further help.

Richard

 

[techsmart]

Try a software

Hey Randi,
Scripting and coding is ok. But if you want to automate your screen scraping need it's better you use ready software.
Automation Anywhere is good package. You can download the trial form its web page. Google Automation Anywhere, go to its download page and get the free trial.

Happy scraping!

 

[Developer2010]

Extract Data from a Website

Hi,

I am new to screen scrapping!

I want to make a selection from a dropdown box with screen scrapping, and then based upon your selection, the data that are then display need to be extracted to a excel file.

Can anyone help?