Screen Scraping HTTPS

[Michael C#]

Hi all,

I'm writing a web service that uses HttpWebRequest/HttpWebResponse to screen
scrape. It works fine for regular HTTP sites, but for an HTTPS site (no
username/password) it gets stuck... I think the certificate is causing it
to fail. Here's the error message I get: "The underlying connection was
closed: Could not establish trust relationship with remote server." Any
ideas? Thanks

 

[Michael C#]

Scratch that, I found a solution. Thanks.

 

[Jon Maz]

Can you post your solution? That way you'll help anyone else who has the
same issue.

 

[Michael C#]

Yeah sure. The problem was that the site name and certificate name didn't
match. It's a government site so they were using the certificate of their
'parent department'. This was causing the Certificate Validation to fail.
To get around it, I created a Class that implements the ICertificatePolicy
interface. In the CheckValidationResult public function of the Class, I
returned True. Finally I assigned an instance of the Class to the
ServicePointManager.CertificatePolicy property.

One word of caution: this pretty much beats any security via certificates
completely into the ground. To get some level of security back, you can
code the CheckValidationResult function to check values of the Certificate
when it's passed in, like certificate.GetName(), GetEffectiveDateString(),
GetExpirationDateString(), etc.

 

[hellrazor]

Scratch that, I found a solution. Thanks.

You should have said "Scrape that"

 

[Michael C#]

LOL. There's one in every crowd, right?

You should have said "Scrape that"

 

[J. Michael Terenin]

Michael,
Could you revisit this thread discussion and perhaps show an example of your
code ? I am attempting the same thing, using Fiddler as a debugger, and am
noticing a similar breakdown when I try an access a web page over SSL (https)
.. I actually found your solution on another blog site, but it still didn't
help me resolve my block.

Much thanks !