Controlling source IP address within urllib2

D

Dan

Does anybody know how to control the source IP address (IPv4) when
using the urllib2 library? I have a Linux box with several IP
addresses in the same subnet, and I want to simulate several
individuals within that subnet accessing web pages independently. I
need the functionality of urllib2 because there will be redirects and
other HTTP-type functions to implement. It would be nice if I could
create (and bind) sockets myself and then tell the urllib functions to
use those sockets. Perhaps there is some sort of "back-door" way of
doing this??? Any hints are appreciated!
-Dan
 
J

John J. Lee

Dan said:
Does anybody know how to control the source IP address (IPv4) when
using the urllib2 library? I have a Linux box with several IP
addresses in the same subnet, and I want to simulate several
individuals within that subnet accessing web pages independently. I
need the functionality of urllib2 because there will be redirects and
other HTTP-type functions to implement. It would be nice if I could
create (and bind) sockets myself and then tell the urllib functions to
use those sockets. Perhaps there is some sort of "back-door" way of
doing this??? Any hints are appreciated!

There's no built-in support, but I expect it's very easy to do. I
suppose the easiest way would be to derive from
httplib.HTTPConnection, overriding .connect() to call .bind() on the
socket. Call it BindingHTTPConnection, then define:

class BindingHTTPHandler(urllib2.HTTPHandler):
def http_open(self, req):
return self.do_open(BindingHTTPConnection, req)


Same goes for https:

if hasattr(httplib, 'HTTPS'):
class BindingHTTPSHandler(urllib2.HTTPSHandler):
def https_open(self, req):
return self.do_open(BindingHTTPSConnection, req)


Personally, I probably wouldn't use build_opener(), but since the
above classes derive from the corresponding urllib2 classes, you can
do:

opener = urllib2.build_opener(BindingHTTPHandler, BindingHTTPSHandler)
response = opener.open('http://www.example.com/')


John
 
D

Dan

John,
Thanks for your input. I can kind of see the light in this, but I'm
having difficulty knowing where the "do_open" method comes from. Also,
I'll need to follow redirects, so I assume then I would add a
HTTPRedirectHandler instance to the urllib2.build_opener. (?) Thanks
again for your help.
-Dan
 
J

John J. Lee

Dan said:
John,
Thanks for your input. I can kind of see the light in this, but I'm
having difficulty knowing where the "do_open" method comes from. Also,
AbstractHTTPHandler


I'll need to follow redirects, so I assume then I would add a
HTTPRedirectHandler instance to the urllib2.build_opener. (?) Thanks

No. It does that by default.


John
 
D

Dan

John,
Thanks again for your help!
I think that the do_open function in AbstractHTTPHandler does not
return the correct object type as required by the opener. When I
include the code you recommended, the implementation comes back with
the message, "urlopen error unknown url type: http". Strange, because
I would think that overriding the "http_open" function in the handler
would have signaled that this function is capable of handling http. If
I call the HTTPHandler base class "http_open" function from within the
derived class, all works okay, but of course, I don't get to use the
source IP address I wanted to use.
I'll keep trying and let you know what I find.
-Dan
 
D

Dan

Just FYI ... I finally got this to work. Unfortunately, I was unable
to use the urllib2 library. The embedded Linux team here informed me
that the urllib2 library will not be available on the machine I wanted
to deploy on, so I had to go back to using the urllib and httplib
libraries. After trying several things, I finally gave up and hacked
the library code directly. It turned out to be quite easy. I just had
to wedge in a "self.sock.bind()" call in front of the
"self.sock.connect()" call within the HTTPConnection class inside the
httplib library. (I used the bind() function to nail down the source
IP address of the client.) Of course, I had to weave in some code that
allowed me to pass the client IP address through the URLopener class in
the urllib library. Everything seems to work so far.

John, thanks again for your help. You pointed me in the right
direction.

-Dan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top