Screen Scraping HTTPS

Discussion in 'ASP .Net Web Services' started by Michael C#, Jun 2, 2005.

  1. Michael C#

    Michael C# Guest

    Hi all,

    I'm writing a web service that uses HttpWebRequest/HttpWebResponse to screen
    scrape. It works fine for regular HTTP sites, but for an HTTPS site (no
    username/password) it gets stuck... I think the certificate is causing it
    to fail. Here's the error message I get: "The underlying connection was
    closed: Could not establish trust relationship with remote server." Any
    ideas? Thanks
    Michael C#, Jun 2, 2005
    #1
    1. Advertising

  2. Michael C#

    Michael C# Guest

    Scratch that, I found a solution. Thanks.

    "Michael C#" <> wrote in message
    news:...
    > Hi all,
    >
    > I'm writing a web service that uses HttpWebRequest/HttpWebResponse to
    > screen scrape. It works fine for regular HTTP sites, but for an HTTPS
    > site (no username/password) it gets stuck... I think the certificate is
    > causing it to fail. Here's the error message I get: "The underlying
    > connection was closed: Could not establish trust relationship with remote
    > server." Any ideas? Thanks
    >
    >
    Michael C#, Jun 2, 2005
    #2
    1. Advertising

  3. Michael C#

    Jon Maz Guest

    Can you post your solution? That way you'll help anyone else who has the
    same issue.
    Jon Maz, Jun 6, 2005
    #3
  4. Michael C#

    Michael C# Guest

    Yeah sure. The problem was that the site name and certificate name didn't
    match. It's a government site so they were using the certificate of their
    'parent department'. This was causing the Certificate Validation to fail.
    To get around it, I created a Class that implements the ICertificatePolicy
    interface. In the CheckValidationResult public function of the Class, I
    returned True. Finally I assigned an instance of the Class to the
    ServicePointManager.CertificatePolicy property.

    One word of caution: this pretty much beats any security via certificates
    completely into the ground. To get some level of security back, you can
    code the CheckValidationResult function to check values of the Certificate
    when it's passed in, like certificate.GetName(), GetEffectiveDateString(),
    GetExpirationDateString(), etc.

    "Jon Maz" <> wrote in message
    news:...
    > Can you post your solution? That way you'll help anyone else who has the
    > same issue.
    >
    >
    Michael C#, Jun 7, 2005
    #4
  5. Michael C#

    hellrazor Guest

    "Michael C#" <> wrote in
    news::

    > Scratch that, I found a solution. Thanks.


    You should have said "Scrape that" ;) :)
    hellrazor, Jun 10, 2005
    #5
  6. Michael C#

    Michael C# Guest

    LOL. There's one in every crowd, right? :)

    "hellrazor" <> wrote in message
    news:Xns96717024654D7jorgeanotherworldcom@207.46.248.16...
    > "Michael C#" <> wrote in
    > news::
    >
    >> Scratch that, I found a solution. Thanks.

    >
    > You should have said "Scrape that" ;) :)
    Michael C#, Jun 10, 2005
    #6
  7. Michael,
    Could you revisit this thread discussion and perhaps show an example of your
    code ? I am attempting the same thing, using Fiddler as a debugger, and am
    noticing a similar breakdown when I try an access a web page over SSL (https)
    .. I actually found your solution on another blog site, but it still didn't
    help me resolve my block.

    Much thanks !
    J. Michael Terenin, Dec 18, 2005
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robert Martinez

    Screen Scraping C#

    Robert Martinez, Aug 26, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    2,744
    Robert Martinez
    Aug 26, 2003
  2. George Durzi

    HTML Screen Scraping Q

    George Durzi, Feb 25, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    506
    George Durzi
    Feb 25, 2004
  3. Jim Giblin

    Screen scraping in ASP.NET

    Jim Giblin, Aug 16, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    808
    Jens Christian Mikkelsen
    Aug 16, 2004
  4. David Jones

    Web Scraping/Site Scraping

    David Jones, Jul 11, 2004, in forum: Python
    Replies:
    4
    Views:
    502
    Andrew Bennetts
    Jul 13, 2004
  5. Replies:
    2
    Views:
    330
    John Nagle
    Apr 21, 2007
Loading...

Share This Page