Automating access to a HTTPS website

Discussion in 'ASP .Net' started by Stefan Kiryazov (MCAD), Apr 8, 2006.

  1. Hi all,

    I'm facing the following problem. I am trying to programmatically
    retrieve data from a website that needs a login through HTTPS.
    Originally, as I am not familiar with HTTPS and SSL, i tried to capture
    web browser requests with a HTTP sniffer, and then send the same
    requests with my application, using System.Net.Sockets.Socket.

    And, as you probably have guessed, it didnt work, because, as I just
    came to know, the client and the server perform a handshake over TCP,
    thats not HTTP based and thus can't be captured by my HTTP sniffer.

    So, any ideas are greatly appreciated, because I have no idea what
    shall I do. Start reading the HTTPS and SSL spec perhaps? This is a
    way, but I am like the idea of finding an easier solution, maybe a
    framework for accessing and parsing web sites? Or utility for taking
    over a browser and employing it to do my work? Anything else?

    Please excuse me if I am asking questions that have been around for
    ages, but currently my searches haven't led me to relevant answers.
    Thanks everybody
     
    Stefan Kiryazov (MCAD), Apr 8, 2006
    #1
    1. Advertising

  2. Hi Stefan,

    You should be able to do this with the Webclient class:

    http://msdn2.microsoft.com/en-us/library/system.net.webclient_members(VS.80).aspx

    It allows you to set the credentials as it requests the resources:

    http://msdn2.microsoft.com/en-us/library/system.net.webclient.credentials(VS.80).aspx

    Ken
    Microsoft MVP [ASP.NET]


    "Stefan Kiryazov (MCAD)" <> wrote in message
    news:...
    > Hi all,
    >
    > I'm facing the following problem. I am trying to programmatically
    > retrieve data from a website that needs a login through HTTPS.
    > Originally, as I am not familiar with HTTPS and SSL, i tried to capture
    > web browser requests with a HTTP sniffer, and then send the same
    > requests with my application, using System.Net.Sockets.Socket.
    >
    > And, as you probably have guessed, it didnt work, because, as I just
    > came to know, the client and the server perform a handshake over TCP,
    > thats not HTTP based and thus can't be captured by my HTTP sniffer.
    >
    > So, any ideas are greatly appreciated, because I have no idea what
    > shall I do. Start reading the HTTPS and SSL spec perhaps? This is a
    > way, but I am like the idea of finding an easier solution, maybe a
    > framework for accessing and parsing web sites? Or utility for taking
    > over a browser and employing it to do my work? Anything else?
    >
    > Please excuse me if I am asking questions that have been around for
    > ages, but currently my searches haven't led me to relevant answers.
    > Thanks everybody
    >
     
    Ken Cox - Microsoft MVP, Apr 9, 2006
    #2
    1. Advertising

  3. Stefan Kiryazov (MCAD)

    Joerg Jooss Guest

    Thus wrote Stefan,

    > Hi all,
    >
    > I'm facing the following problem. I am trying to programmatically
    > retrieve data from a website that needs a login through HTTPS.
    > Originally, as I am not familiar with HTTPS and SSL, i tried to
    > capture web browser requests with a HTTP sniffer, and then send the
    > same requests with my application, using System.Net.Sockets.Socket.
    >
    > And, as you probably have guessed, it didnt work, because, as I just
    > came to know, the client and the server perform a handshake over TCP,
    > thats not HTTP based and thus can't be captured by my HTTP sniffer.


    Good idea, flawed execution ;-)

    In order to capture HTTPs traffic, it's easiest to use a browser-side plug-in
    that works on the request after decryption, e.g. LiveHttpHeaders.

    For the implementation in .NET, you can use HttpWebRequest/Response or WebClient
    from the System.Net namespace. You might want to check out the HtmlAgilityPack
    to process HTML content.

    Cheers,
    --
    Joerg Jooss
     
    Joerg Jooss, Apr 10, 2006
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. David Armour

    HTTPS and automating client sessions

    David Armour, Feb 9, 2004, in forum: Java
    Replies:
    2
    Views:
    475
    David Armour
    Feb 9, 2004
  2. Axel
    Replies:
    8
    Views:
    1,260
    Adrienne Boswell
    Apr 27, 2009
  3. jotto
    Replies:
    4
    Views:
    441
    jotto
    Oct 2, 2006
  4. Naveen Dhanuka
    Replies:
    1
    Views:
    318
  5. Paul E. Schoen

    Automating website maintenance

    Paul E. Schoen, Aug 19, 2010, in forum: Javascript
    Replies:
    6
    Views:
    153
    Paul E. Schoen
    Aug 21, 2010
Loading...

Share This Page