https proxy

P

Paul Sweeney

Does anyone know of a working (python) https proxy which allows viewing of
unencrypted data being sent from my browser to an https site?

I've worked my way through most on the list at
http://xhaus.com/alan/python/proxies.html, but while many claim to support
https, if you actually point your browser at the proxies, they work fine for
http, but not for https pages.

TIA

Paul Sweeney
 
S

Simon Dahlbacka

Paul said:
Does anyone know of a working (python) https proxy which allows viewing of
unencrypted data being sent from my browser to an https site?

hmm, I thought the _purpose_ of using https was to make it relatively
impossible to view the unencrypted data being the "man in the middle"..
 
P

Paul Sweeney

Simon said:
hmm, I thought the _purpose_ of using https was to make it relatively
impossible to view the unencrypted data being the "man in the middle"..

It's certainly not impossible, there are tools like Paros for java which do
the job, the browser sets up an http connection with the proxy (using the
proxy's built in certificate), and the proxy then sets up an https
connection with the destination server, but the data is unencrypted in the
proxy before being re-encrypted to send to the destination server.

What is (virtually) impossible is to intercept and do a "man in the middle"
attack on an existing connect. I don't want to intercept stuff on the net,
just see what the browser on my machine is sending/receiving

HTH :)
 
P

Peter Hansen

Paul said:
It's certainly not impossible, there are tools like Paros for java which do
the job, the browser sets up an http connection with the proxy (using the
proxy's built in certificate), and the proxy then sets up an https
connection with the destination server, but the data is unencrypted in the
proxy before being re-encrypted to send to the destination server.

What is (virtually) impossible is to intercept and do a "man in the middle"
attack on an existing connect. I don't want to intercept stuff on the net,
just see what the browser on my machine is sending/receiving

It sounds like you want either to see the raw data stream (the
encrypted stuff), or you want to see the unencrypted data that
the browser would be sending if it weren't using https. It's
still unclear. If the latter, why not use Paros, since you seem
to know about it and how it works?

(The reason your request is unclear is because your first message
talks about seeing the "unencrypted data being sent from [your]
browser to an https site" and yet obviously, as you know, there
is no unencrypted data going to the https site... But since you
certainly know this, it makes it unclear just which you are
requesting.)

-Peter
 
P

Paul Sweeney

Peter said:
Paul said:
It's certainly not impossible, there are tools like Paros for java which do
the job, the browser sets up an http connection with the proxy (using the
proxy's built in certificate), and the proxy then sets up an https
connection with the destination server, but the data is unencrypted in the
proxy before being re-encrypted to send to the destination server.

What is (virtually) impossible is to intercept and do a "man in the middle"
attack on an existing connect. I don't want to intercept stuff on the net,
just see what the browser on my machine is sending/receiving

It sounds like you want either to see the raw data stream (the
encrypted stuff), or you want to see the unencrypted data that
the browser would be sending if it weren't using https. It's
still unclear. If the latter, why not use Paros, since you seem
to know about it and how it works?

(The reason your request is unclear is because your first message
talks about seeing the "unencrypted data being sent from [your]
browser to an https site" and yet obviously, as you know, there
is no unencrypted data going to the https site... But since you
certainly know this, it makes it unclear just which you are
requesting.)

-Peter

Ok, apologies, let me try to explain it more clearly.

I wish to access an https:// site using my browser and click around it. I
wish to see the http(s) GET urls and http(s) POST urls and data which are
being sent to the https server. If the site were an http:// site I would
use a tool like ethereal to see what was going on, but this data is sent
encrypted from the browser due to the https connection, so I'll just see the
encrypted data. So I need a tool to view the unencrypted form of the data
which is being sent encrypted (sorry for being unclear about the "unecrypted
data" in my last post, I hope this is better).

I have and could use a tool like Paros, but it is java and (need I say
more?):

a) I love Python :)-D) ...
b) ... seriously, I'd like to be able to play with the source code to record
the gets and posts for later replay in a python based retriever tool.

Thanks for your interest in this thread.

Paul Sweeney
 
P

Peter Hansen

Paul said:
I wish to access an https:// site using my browser and click around it. I
wish to see the http(s) GET urls and http(s) POST urls and data which are
being sent to the https server. If the site were an http:// site I would
use a tool like ethereal to see what was going on, but this data is sent
encrypted from the browser due to the https connection, so I'll just see the
encrypted data. So I need a tool to view the unencrypted form of the data
which is being sent encrypted (sorry for being unclear about the "unecrypted
data" in my last post, I hope this is better).

Based on what you say here, I think the answer is back to being
"impossible". The encryption is going on in the browser. If
the browser thinks it's talking HTTPS, then it will encrypt
stuff and talk HTTPS. Nothing between the browser end of the
socket and the server end of the socket will be able to
decrypt it.

If you had a proxy that used its own certificate and established
an https connection when the client tried to connect with http
to something, then you could at least start the whole thing off
without encrypted data on the browser side. As soon as you tried
to click on an embedded https link, however, the browser would
try to establish an encrypted link, and the proxy wouldn't work
any more.

What about just writing a tool using urllib2 and basically
simulating the browser activity in Python? Then you can
bypass the whole issue, and have programmatic access to the
unencrypted data before or after it hits the encrypted socket
(depending on the direction of travel).
I have and could use a tool like Paros, but it is java and (need I say
more?):

a) I love Python :)-D) ...
b) ... seriously, I'd like to be able to play with the source code to record
the gets and posts for later replay in a python based retriever tool.

I love Python too, but that doesn't stop me from using a more
effective tool when the need arises. Surely something like Paros,
if it can do what you need (and I don't see how it can), also
has the ability to save the data to a file for later perusal,
even if the utility for retrieving it were in Python.

-Peter
 
D

Dietmar Schwertberger

Paul Sweeney said:
Does anyone know of a working (python) https proxy which allows viewing of
unencrypted data being sent from my browser to an https site?

I've worked my way through most on the list at
http://xhaus.com/alan/python/proxies.html, but while many claim to support
https, if you actually point your browser at the proxies, they work fine for
http, but not for https pages.

I wrote a http proxy server some time ago to use Netscape/Firefox for
homebanking through the modem connection of my RISC OS computer (I don't
like connecting my Windows PC to the internet....).
I don't think a simple proxy server will help you at all as it just waits
for the browser connecting, receives a "CONNECT ..." request, builds the
requested connection and then just passes on the data between the browser
and the website until the connection is closed from either side ...

Anyway it's available from my homepage; it's in the RISC OS additions
archive - the name is SSLRelay.



Regards,

Dietmar
 
P

Paul Rubin

Paul Sweeney said:
I wish to access an https:// site using my browser and click around
it. I wish to see the http(s) GET urls and http(s) POST urls and
data which are being sent to the https server. If the site were an
http:// site I would use a tool like ethereal to see what was going
on, but this data is sent encrypted from the browser due to the
https connection, so I'll just see the encrypted data. So I need a
tool to view the unencrypted form of the data which is being sent
encrypted (sorry for being unclear about the "unecrypted data" in my
last post, I hope this is better).

You could do that but I don't know of already-existing Python code
that does it. Basically it would be a man-in-the-middle attack against
the browser/server pair:

browser---(SSL)---proxy---(SSL)---server

There'd be two separate SSL sessions and the proxy would of course
have to present its own server certificate to the browser, which you'd
configure the browser to accept. The server application could take
various measures to detect this subterfuge and stop it from happening,
but I doubt any normal server apps actually bother to do that. It's
not a true security break, since the browser user has to accept the
proxy certificate, thereby giving "permission" for the interception.
 
J

John Hazen

* Peter Hansen said:
Based on what you say here, I think the answer is back to being
"impossible". The encryption is going on in the browser. If
the browser thinks it's talking HTTPS, then it will encrypt
stuff and talk HTTPS. Nothing between the browser end of the
socket and the server end of the socket will be able to
decrypt it.

Although, if this is just for testing, I think you could trick the
browser by running a proxy on a different machine that accepts an HTTPS
connection, and creates another HTTPS connection with the destination
server. Then run your own DNS server that hands back bogus answers to
every request resolving to the proxy's IP address.

The browser will talk HTTPS to the proxy, thinking it's talking to the
target server. Of course, this will trigger the "certificate doesn't
match IP address" type of warning, but I think most browsers will let
you continue after warning you. (If not, you could probably still get
around it with the gross hack of having the DNS server communicate with
the proxy, so the proxy could generate on the fly a certificate for the
server name that the browser is trying to reach. This would get you the
"certificate can't be verified" type of error, which is definitely one
that you can continue after acknowledging.)

So, if it's just for testing and research, that might work. Obviously,
it's not a viable setup if you're trying to use it to snoop on the
conversations of unsuspecting users (which I'm not accusing you of, but
it's a much harder problem).

-John
 
P

Paul Rubin

Although, if this is just for testing, I think you could trick the
browser by running a proxy on a different machine that accepts an HTTPS
connection, and creates another HTTPS connection with the destination
server. Then run your own DNS server that hands back bogus answers to
every request resolving to the proxy's IP address.

You can use /etc/hosts for that. No need for running your own DNS.
But why bother? Just configure the browser send requests through the
proxy.
The browser will talk HTTPS to the proxy, thinking it's talking to the
target server. Of course, this will trigger the "certificate doesn't
match IP address" type of warning, but I think most browsers will let
you continue after warning you.

Yes, or you can generate a cert with the right hostname in it, and
install it in the browser as a trusted cert.

To the OP, are you trying to debug a server-side app? If you're not
trying to send actual confidential data through the SSL connection and
if you can configure the server, it's probably simplest to just open a
non-SSL port, unless you're trying to debug something SSL-specific.
 
P

Paul Sweeney

Paul said:
You can use /etc/hosts for that. No need for running your own DNS.
But why bother? Just configure the browser send requests through the
proxy.

agreed, no need to involve DNS, just set as proxy in browser connection
settings
Yes, or you can generate a cert with the right hostname in it, and
install it in the browser as a trusted cert.

This is not a problem since it is ok to have to click on the certificate
warning box that will come up in the browser.
To the OP, are you trying to debug a server-side app? If you're not
trying to send actual confidential data through the SSL connection and
if you can configure the server, it's probably simplest to just open a
non-SSL port, unless you're trying to debug something SSL-specific.

Unfortunately I don't have access to the target server, so this is not an
option (see you other post to this thread for a more description of the
problem)

Thanks
 
P

Paul Sweeney

Paul said:
You could do that but I don't know of already-existing Python code
that does it.

Rats, that would seem to be the consensus on this, the nearest is the http
proxies which allow https to tunnel through them leaving the data encrypted.
At least they would do if this feature of any of them actually worked ;-)
Basically it would be a man-in-the-middle attack against
the browser/server pair:

browser---(SSL)---proxy---(SSL)---server

There'd be two separate SSL sessions and the proxy would of course
have to present its own server certificate to the browser, which you'd
configure the browser to accept. The server application could take
various measures to detect this subterfuge and stop it from happening,
but I doubt any normal server apps actually bother to do that. It's
not a true security break, since the browser user has to accept the
proxy certificate, thereby giving "permission" for the interception.

A perfect description of the problem and possible solutions, thanks Paul,
glad you could make this out from my descriptions :-D

This is exactly what the java Paros tool does, but I'd like to have a Python
version which I can customise to do the things I need automatically. I'm
working on an automatic testing framework written in python which can
perform scripted actions on a target system, and want an easy way to
determine the urls of GETs and the urls (including query strong) and data of
POSTS performed under https connections. In order to replay them later. I'm
doing it with the target server owners permission, but don't have access to
fiddle with server configuration settings etc.

Just to emphasise again, I'm only trying to see the data that my browser is
sending/receiving, *not* intercept someone elses connection.

Any posts for suggested python software gratefully received. If I end up
writing one of these, I'll try to get agreement from my employer for it to
be open sourced since there seems to be an opening for such a tool, even if
there isn't much of a market for it!
 
P

Paul Sweeney

Dietmar said:
I wrote a http proxy server some time ago to use Netscape/Firefox for
homebanking through the modem connection of my RISC OS computer (I don't
like connecting my Windows PC to the internet....).
I don't think a simple proxy server will help you at all as it just waits
for the browser connecting, receives a "CONNECT ..." request, builds the
requested connection and then just passes on the data between the browser
and the website until the connection is closed from either side ...

Anyway it's available from my homepage; it's in the RISC OS additions
archive - the name is SSLRelay.

Thanks for posting a response Dietmar.

It sounds like this works the same as the http proxies in my OP so it won't
unencrypt the data to make it viewable, but then it doesn't need to for your
app :)

In case someone else is reading this and it sounds useful to them, I
couldn't figure the url of your website, so you may want to repost with it.

Paul
 
A

Alan Kennedy

[Paul Sweeney]

[Paul Rubin]
You could do that but I don't know of already-existing Python code
that does it. Basically it would be a man-in-the-middle attack against
the browser/server pair:

browser---(SSL)---proxy---(SSL)---server

There'd be two separate SSL sessions and the proxy would of course
have to present its own server certificate to the browser, which you'd
configure the browser to accept. The server application could take
various measures to detect this subterfuge and stop it from happening,
but I doubt any normal server apps actually bother to do that. It's
not a true security break, since the browser user has to accept the
proxy certificate, thereby giving "permission" for the interception.

Paul S,

Perhaps you could achieve what you want with Apache and mod_proxy?

http://httpd.apache.org/docs/mod/mod_proxy.html

Use (a HTTPS-configured) Apache to decode the requests with mod_proxy,
and simply forward all requests to another plain (HTTP) python proxy
of your choice. The ProxyRemote or ProxyPass directives could be used
for that.

On looking at the ProxyRemote documentation, it states "only 'http' is
supported by this module.'

However, a quick search turns up the following snippet, i.e. that
Apache mod_ssl provides HTTPS support for mod_proxy.

From the horses mouth, i.e. Ralf Engelschall: "when you want to
forward to a HTTPS target you need HTTPS support in mod_proxy.
That's only possible with mod_ssl (which enhanced mod_proxy for
HTTPS). Then you can do all: Accept HTTP and HTTPS and connect
to HTTP and HTTPS targets, i.e. create gateways in all
combinations."

http://www.lists.aldigital.co.uk/apache-ssl/msg02860.html

HTH,
 
D

Dietmar Schwertberger

Paul Sweeney said:
Thanks for posting a response Dietmar.

It sounds like this works the same as the http proxies in my OP so it won't
unencrypt the data to make it viewable, but then it doesn't need to for your
app :)

In case someone else is reading this and it sounds useful to them, I
couldn't figure the url of your website, so you may want to repost with it.
Oh yes, it's http://www.schwertberger.de .


Regards,

Dietmar
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top