R
Robert Martinez
I've seen a lot about screen scraping with .NET, mostly in VB.net. I have
been able to convert most of it over, but it is still just very basic stuff.
Can someone help direct me toward some good info / samples on the following:
I want to be able to do 3 things:
1) Set up a module in IBUYSPY Portal (like in the right or left pane) that
simply just scrapes headlines from certain sites.
2) Allow users to set up scraping of certain sites that they want -- I
realize this is VERY hard to do, but the following I thought might help:
3) Possibly have a worker process that would download the source of a site
(regardless of the language it's written in) and save it as an HTML stream
(I have this part down -- there are tons of examples out there), then parse
out the content wanted. However, as content changes on a regular basis on
some sites, including structure, is there a way to automate this? I know I
couldn't automate the parsing, but I thought maybe an ADMIN function that
simply downloads sites and a) checks to see if structure has changed, if
not, then no problem; or b) if a site structure has changed, then dump the
HTML stream in a file and send off a notification to the ADMIN to go and
look at it to set up proper parsing.
Any insight would be much appreciated.
been able to convert most of it over, but it is still just very basic stuff.
Can someone help direct me toward some good info / samples on the following:
I want to be able to do 3 things:
1) Set up a module in IBUYSPY Portal (like in the right or left pane) that
simply just scrapes headlines from certain sites.
2) Allow users to set up scraping of certain sites that they want -- I
realize this is VERY hard to do, but the following I thought might help:
3) Possibly have a worker process that would download the source of a site
(regardless of the language it's written in) and save it as an HTML stream
(I have this part down -- there are tons of examples out there), then parse
out the content wanted. However, as content changes on a regular basis on
some sites, including structure, is there a way to automate this? I know I
couldn't automate the parsing, but I thought maybe an ADMIN function that
simply downloads sites and a) checks to see if structure has changed, if
not, then no problem; or b) if a site structure has changed, then dump the
HTML stream in a file and send off a notification to the ADMIN to go and
look at it to set up proper parsing.
Any insight would be much appreciated.