Instrumented web proxy

[Andrew McLean]

I would like to write a web (http) proxy which I can instrument to
automatically extract information from certain web sites as I browse
them. Specifically, I would want to process URLs that match a particular
regexp. For those URLs I would have code that parsed the content and
logged some of it.

Think of it as web scraping under manual control.

I found this list of Python web proxies

http://www.xhaus.com/alan/python/proxies.html

Tiny HTTP Proxy in Python looks promising as it's nominally simple (not
many lines of code)

http://www.okisoft.co.jp/esc/python/proxy/

It does what it's supposed to, but I'm a bit at a loss as where to
intercept the traffic. I suspect it should be quite straightforward, but
I'm finding the code a bit opaque.

Any suggestions?

Andrew

 

[Miki]

Hello Andrew,

Tiny HTTP Proxy in Python looks promising as it's nominally simple (not
many lines of code)

http://www.okisoft.co.jp/esc/python/proxy/

It does what it's supposed to, but I'm a bit at a loss as where to
intercept the traffic. I suspect it should be quite straightforward, but
I'm finding the code a bit opaque.

Any suggestions?

From a quick look at the code, you need to either hook to do_GET where
you have the URL (see the urlunparse line).
If you want the actual content of the page, you'll need to hook to
_read_write (data = i.recv(8192)).

HTH,

 

[Paul Rubin]

I would like to write a web (http) proxy which I can instrument to
automatically extract information from certain web sites as I browse
them. Specifically, I would want to process URLs that match a
particular regexp. For those URLs I would have code that parsed the
content and logged some of it.

Think of it as web scraping under manual control.

I've used Proxy 3 for this, a very cool program with powerful
capabilities for on the fly html rewriting.

http://theory.stanford.edu/~amitp/proxy.html

 

[Andrew McLean]

I've used Proxy 3 for this, a very cool program with powerful
capabilities for on the fly html rewriting.

http://theory.stanford.edu/~amitp/proxy.html

This looks very useful. Unfortunately I can't seem to get it to run
under Windows (specifically Vista) using Python 1.5.2, 2.2.3 or 2.5.2.
I'll try Linux if I get a chance.