Controlling source IP address within urllib2

Discussion in 'Python' started by Dan, Jun 4, 2005.

  1. Dan

    Dan Guest

    Does anybody know how to control the source IP address (IPv4) when
    using the urllib2 library? I have a Linux box with several IP
    addresses in the same subnet, and I want to simulate several
    individuals within that subnet accessing web pages independently. I
    need the functionality of urllib2 because there will be redirects and
    other HTTP-type functions to implement. It would be nice if I could
    create (and bind) sockets myself and then tell the urllib functions to
    use those sockets. Perhaps there is some sort of "back-door" way of
    doing this??? Any hints are appreciated!
    -Dan
    Dan, Jun 4, 2005
    #1
    1. Advertising

  2. Dan

    John J. Lee Guest

    "Dan" <> writes:

    > Does anybody know how to control the source IP address (IPv4) when
    > using the urllib2 library? I have a Linux box with several IP
    > addresses in the same subnet, and I want to simulate several
    > individuals within that subnet accessing web pages independently. I
    > need the functionality of urllib2 because there will be redirects and
    > other HTTP-type functions to implement. It would be nice if I could
    > create (and bind) sockets myself and then tell the urllib functions to
    > use those sockets. Perhaps there is some sort of "back-door" way of
    > doing this??? Any hints are appreciated!


    There's no built-in support, but I expect it's very easy to do. I
    suppose the easiest way would be to derive from
    httplib.HTTPConnection, overriding .connect() to call .bind() on the
    socket. Call it BindingHTTPConnection, then define:

    class BindingHTTPHandler(urllib2.HTTPHandler):
    def http_open(self, req):
    return self.do_open(BindingHTTPConnection, req)


    Same goes for https:

    if hasattr(httplib, 'HTTPS'):
    class BindingHTTPSHandler(urllib2.HTTPSHandler):
    def https_open(self, req):
    return self.do_open(BindingHTTPSConnection, req)


    Personally, I probably wouldn't use build_opener(), but since the
    above classes derive from the corresponding urllib2 classes, you can
    do:

    opener = urllib2.build_opener(BindingHTTPHandler, BindingHTTPSHandler)
    response = opener.open('http://www.example.com/')


    John
    John J. Lee, Jun 4, 2005
    #2
    1. Advertising

  3. Dan

    Dan Guest

    John,
    Thanks for your input. I can kind of see the light in this, but I'm
    having difficulty knowing where the "do_open" method comes from. Also,
    I'll need to follow redirects, so I assume then I would add a
    HTTPRedirectHandler instance to the urllib2.build_opener. (?) Thanks
    again for your help.
    -Dan
    Dan, Jun 6, 2005
    #3
  4. Dan

    John J. Lee Guest

    "Dan" <> writes:

    > John,
    > Thanks for your input. I can kind of see the light in this, but I'm
    > having difficulty knowing where the "do_open" method comes from. Also,


    AbstractHTTPHandler


    > I'll need to follow redirects, so I assume then I would add a
    > HTTPRedirectHandler instance to the urllib2.build_opener. (?) Thanks


    No. It does that by default.


    John
    John J. Lee, Jun 8, 2005
    #4
  5. Dan

    Dan Guest

    John,
    Thanks again for your help!
    I think that the do_open function in AbstractHTTPHandler does not
    return the correct object type as required by the opener. When I
    include the code you recommended, the implementation comes back with
    the message, "urlopen error unknown url type: http". Strange, because
    I would think that overriding the "http_open" function in the handler
    would have signaled that this function is capable of handling http. If
    I call the HTTPHandler base class "http_open" function from within the
    derived class, all works okay, but of course, I don't get to use the
    source IP address I wanted to use.
    I'll keep trying and let you know what I find.
    -Dan
    Dan, Jun 8, 2005
    #5
  6. Dan

    Dan Guest

    Just FYI ... I finally got this to work. Unfortunately, I was unable
    to use the urllib2 library. The embedded Linux team here informed me
    that the urllib2 library will not be available on the machine I wanted
    to deploy on, so I had to go back to using the urllib and httplib
    libraries. After trying several things, I finally gave up and hacked
    the library code directly. It turned out to be quite easy. I just had
    to wedge in a "self.sock.bind()" call in front of the
    "self.sock.connect()" call within the HTTPConnection class inside the
    httplib library. (I used the bind() function to nail down the source
    IP address of the client.) Of course, I had to weave in some code that
    allowed me to pass the client IP address through the URLopener class in
    the urllib library. Everything seems to work so far.

    John, thanks again for your help. You pointed me in the right
    direction.

    -Dan
    Dan, Jun 10, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. headware

    controlling redirection within frames

    headware, Aug 5, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    323
    headware
    Aug 5, 2003
  2. Josef Cihal
    Replies:
    0
    Views:
    727
    Josef Cihal
    Sep 5, 2005
  3. Replies:
    1
    Views:
    964
    Rene Pijlman
    Mar 10, 2006
  4. =?Utf-8?B?RGFuaWVs?=
    Replies:
    1
    Views:
    316
    Alexey Smirnov
    May 10, 2007
  5. slmnhq
    Replies:
    0
    Views:
    649
    slmnhq
    Feb 9, 2010
Loading...

Share This Page