Automating access to a HTTPS website

[Stefan Kiryazov (MCAD)]

Hi all,

I'm facing the following problem. I am trying to programmatically
retrieve data from a website that needs a login through HTTPS.
Originally, as I am not familiar with HTTPS and SSL, i tried to capture
web browser requests with a HTTP sniffer, and then send the same
requests with my application, using System.Net.Sockets.Socket.

And, as you probably have guessed, it didnt work, because, as I just
came to know, the client and the server perform a handshake over TCP,
thats not HTTP based and thus can't be captured by my HTTP sniffer.

So, any ideas are greatly appreciated, because I have no idea what
shall I do. Start reading the HTTPS and SSL spec perhaps? This is a
way, but I am like the idea of finding an easier solution, maybe a
framework for accessing and parsing web sites? Or utility for taking
over a browser and employing it to do my work? Anything else?

Please excuse me if I am asking questions that have been around for
ages, but currently my searches haven't led me to relevant answers.
Thanks everybody

 

[Ken Cox - Microsoft MVP]

Hi Stefan,

You should be able to do this with the Webclient class:

http://msdn2.microsoft.com/en-us/library/system.net.webclient_members(VS.80).aspx

It allows you to set the credentials as it requests the resources:

http://msdn2.microsoft.com/en-us/library/system.net.webclient.credentials(VS.80).aspx

Ken
Microsoft MVP [ASP.NET]

 

[Joerg Jooss]

Thus wrote Stefan,

Hi all,

I'm facing the following problem. I am trying to programmatically
retrieve data from a website that needs a login through HTTPS.
Originally, as I am not familiar with HTTPS and SSL, i tried to
capture web browser requests with a HTTP sniffer, and then send the
same requests with my application, using System.Net.Sockets.Socket.

And, as you probably have guessed, it didnt work, because, as I just
came to know, the client and the server perform a handshake over TCP,
thats not HTTP based and thus can't be captured by my HTTP sniffer.

Good idea, flawed execution ;-)

In order to capture HTTPs traffic, it's easiest to use a browser-side plug-in
that works on the request after decryption, e.g. LiveHttpHeaders.

For the implementation in .NET, you can use HttpWebRequest/Response or WebClient
from the System.Net namespace. You might want to check out the HtmlAgilityPack
to process HTML content.

Cheers,