scope of socket.setdefaulttimeout?

Discussion in 'Python' started by Russell Warren, Sep 29, 2005.

  1. Does anyone know the scope of the socket.setdefaulttimeout call? Is it
    a cross-process/system setting or does it stay local in the application
    in which it is called?

    I've been testing this and it seems to stay in the application scope,
    but the paranoid side of me thinks I may be missing something... any
    confirmation would be helpful.
     
    Russell Warren, Sep 29, 2005
    #1
    1. Advertising

  2. Russell Warren

    Steve Holden Guest

    Russell Warren wrote:
    > Does anyone know the scope of the socket.setdefaulttimeout call? Is it
    > a cross-process/system setting or does it stay local in the application
    > in which it is called?
    >
    > I've been testing this and it seems to stay in the application scope,
    > but the paranoid side of me thinks I may be missing something... any
    > confirmation would be helpful.
    >

    Yes, it's an application setting, you aren't changing things for anyone
    else.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC www.holdenweb.com
    PyCon TX 2006 www.python.org/pycon/
     
    Steve Holden, Sep 30, 2005
    #2
    1. Advertising

  3. Threads and socket.setdefaulttimeout

    It appears that the timeout setting is contained within a process
    (thanks for the confirmation), but I've realized that the function
    doesn't play friendly with threads. If I have multiple threads using
    sockets and one (or more) is using timeouts, one thread affects the
    other and you get unpredictable behavior sometimes. I included a short
    script at the end of this that demonstrates the threading problem.

    I'm trying to get around this by forcing all locations that want to set
    a timeout to use a 'safe' call immediately prior to socket creation
    that locks out setting the timeout again until the lock is released.
    Something like this:

    try:
    SafeSetSocketTimeout(Timeout_s)
    #lock currently acquired to prevent other threads sneaking in here
    CreateSocket()
    finally:
    ReleaseSocketTimeoutSettingLock()
    UseSocket()

    However - this is getting increasingly painful because I don't have
    easy access to all of the socket creations where I'd like to do this.
    The biggest pain right now is that I'm using xmlrpclib which has some
    seriously/frustratingly heavy use of __ prefixes that makes getting
    inside to do this at socket creation near impossible (at least I think
    so). Right now the best I can do is surround the xmlrpclib calls with
    this (effectively putting the lock release after the UseSocket), but
    then other threads get hung up for the duration of the call or timeout,
    rather than just the simple socket creation.

    It would be nice if the timeout were implemented as an argument in the
    socket constructor rather than having this global method. Is there a
    reason for this? I tried sifting through the cvs source and got lost -
    couldn't even find the call definition for socket(family, type, proto)
    and gave up...

    Does anybody have any idea of another way to do what I need (indpendent
    socket timeouts per thread), or have suggestions on how to break into
    xmlrpclib (actually down into httplib) to do the methdo I was trying?

    Related question: Is there some global way that I'm unaware of to make
    it so that some few lines of code are atomic/uninterruptable and no
    other thread can sneak in between?

    All suggestions appreciated! Hopefully I'm just missing something
    obvious.

    Russ

    #--- This script confirms that settimeout's affect is across threads
    import threading, xmlrpclib, socket

    def st():
    socket.setdefaulttimeout(0.1)

    try:
    proxy = xmlrpclib.ServerProxy("http://localhost:10000")
    print proxy.NonExistentCallThatShouldTimeout()
    except Exception, E: print "Exception caught: %s" % (E,)

    cbThread = threading.Thread(target = st)
    cbThread.start()

    try:
    print proxy.NonExistentCallThatShouldTimeout()
    except Exception, E: print "Exception caught: %s" % (E,)

    #Output is:
    #Exception caught: (10061, 'Connection refused')
    #Exception caught: timed out
     
    Russell Warren, Oct 12, 2005
    #3
  4. Russell Warren

    Steve Holden Guest

    Re: Threads and socket.setdefaulttimeout

    Russell Warren wrote:
    > It appears that the timeout setting is contained within a process
    > (thanks for the confirmation), but I've realized that the function
    > doesn't play friendly with threads. If I have multiple threads using
    > sockets and one (or more) is using timeouts, one thread affects the
    > other and you get unpredictable behavior sometimes. I included a short
    > script at the end of this that demonstrates the threading problem.
    >

    When you say "one thread affects another", I see that your example uses
    the same function for both threads. IMHO it's much better to override
    the thread's run() method than to provide a callable at thread creating
    time. That way you can be sure each thread's execution is firmly in the
    context of the particular thread instance's namespace.

    having said all this, I don't think that's your issue.

    > I'm trying to get around this by forcing all locations that want to set
    > a timeout to use a 'safe' call immediately prior to socket creation
    > that locks out setting the timeout again until the lock is released.
    > Something like this:
    >
    > try:
    > SafeSetSocketTimeout(Timeout_s)
    > #lock currently acquired to prevent other threads sneaking in here
    > CreateSocket()
    > finally:
    > ReleaseSocketTimeoutSettingLock()
    > UseSocket()
    >

    This seems extremely contorted, and I'm pretty sure we can find a better
    way.

    > However - this is getting increasingly painful because I don't have
    > easy access to all of the socket creations where I'd like to do this.
    > The biggest pain right now is that I'm using xmlrpclib which has some
    > seriously/frustratingly heavy use of __ prefixes that makes getting
    > inside to do this at socket creation near impossible (at least I think
    > so). Right now the best I can do is surround the xmlrpclib calls with
    > this (effectively putting the lock release after the UseSocket), but
    > then other threads get hung up for the duration of the call or timeout,
    > rather than just the simple socket creation.
    >

    The threads' network calls should be yielding process control during
    their timeout period to allow other runnable threads to proceed. That's
    how it's always worked for me, anyway, and I've written an email sender
    with 200 parallel threads.

    > It would be nice if the timeout were implemented as an argument in the
    > socket constructor rather than having this global method. Is there a
    > reason for this? I tried sifting through the cvs source and got lost -
    > couldn't even find the call definition for socket(family, type, proto)
    > and gave up...
    >

    You are aware, I presume, that you can set a timeout on each socket
    individually using its settimeout() method?

    > Does anybody have any idea of another way to do what I need (indpendent
    > socket timeouts per thread), or have suggestions on how to break into
    > xmlrpclib (actually down into httplib) to do the methdo I was trying?
    >

    See above. However, this *does* require you to have access to the
    sockets, which is tricky if they are buried deep in some opaque object's
    methods.

    > Related question: Is there some global way that I'm unaware of to make
    > it so that some few lines of code are atomic/uninterruptable and no
    > other thread can sneak in between?
    >

    There are locks! I suspect what you need is a threading.Rlock object,
    that a thread has to hold to be able to modify the (global) default
    timeout. This isn't a full solution to your problem, though, as you have
    correctly deduced.

    The problem arises when a single method call creates a socket and then
    tries to do something with it (like connect to a remote server), which
    *is* problematic. I have to say that I haven't ever used different
    timeout values for the sockets in different parallel threads, so this is
    a new problem to me.

    > All suggestions appreciated! Hopefully I'm just missing something
    > obvious.
    >
    > Russ
    >
    > #--- This script confirms that settimeout's affect is across threads
    > import threading, xmlrpclib, socket
    >
    > def st():
    > socket.setdefaulttimeout(0.1)
    >
    > try:
    > proxy = xmlrpclib.ServerProxy("http://localhost:10000")
    > print proxy.NonExistentCallThatShouldTimeout()
    > except Exception, E: print "Exception caught: %s" % (E,)
    >
    > cbThread = threading.Thread(target = st)
    > cbThread.start()
    >
    > try:
    > print proxy.NonExistentCallThatShouldTimeout()
    > except Exception, E: print "Exception caught: %s" % (E,)
    >
    > #Output is:
    > #Exception caught: (10061, 'Connection refused')
    > #Exception caught: timed out
    >

    Here (unless I'm missing something obvious) it seems that your worker
    thread terminates immediately after setting the default timeout, and
    both of the proxy calls are made from the main thread, so I'm not
    particularly surprised at the results, given the global nature of the
    default socket timeout.

    Maybe someone else can think of something that will help.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC www.holdenweb.com
    PyCon TX 2006 www.python.org/pycon/
     
    Steve Holden, Oct 12, 2005
    #4
  5. Re: Threads and socket.setdefaulttimeout

    Thanks for the detailed repsone... sorry for the lag in responding to
    it.

    After reading and further thought, the only reason I was using
    setdefaulttimeout in the first place (rather then using a direct
    settimeout on the socket) was because it seemed like the only way (and
    easy) of getting access to the seemingly deeply buried socket being
    used by xmlrpclib. That was prior to me using threads of course. I
    then started trying to make this solution work with thread, but it is
    now too convoluted as you say. Now I think the best solution is likely
    to redirect my efforts at getting access to the socket used by
    xmlrpclib so that I can set it's timeout directly. I'm still unclear
    how to do this cleanly, though.

    Getting to some of your comments.

    > When you say "one thread affects another", I see that your example uses
    > the same function for both threads. IMHO it's much better to override
    > the thread's run() method than to provide a callable at thread creating
    > time. That way you can be sure each thread's execution is firmly in the
    > context of the particular thread instance's namespace.
    >
    > having said all this, I don't think that's your issue.


    Correct - the bottom code is nothing to do with my code and was only to
    quickly prove that it was cross-thread.

    > This seems extremely contorted, and I'm pretty sure we can find a better
    > way.


    Couldn't agree more!

    > The threads' network calls should be yielding process control during
    > their timeout period to allow other runnable threads to proceed. That's


    Yep. This is not causing me any problem.

    > You are aware, I presume, that you can set a timeout on each socket
    > individually using its settimeout() method?


    Yes, but I momentarily had forgot about it... as mentioned I ended up
    making the since-bad choice of using setdefaulttimeout to get timeouts
    set on the inaccessible sockets. Then I carried it too far...

    > See above. However, this *does* require you to have access to the
    > sockets, which is tricky if they are buried deep in some opaque object's
    > methods.


    Any help on how to crack the safe would be appreciated.

    > There are locks! I suspect what you need is a threading.Rlock object,
    > that a thread has to hold to be able to modify the (global) default
    > timeout. This isn't a full solution to your problem, though, as you have
    > correctly deduced.


    Not quite what I was after I don't think since potentially interfering
    code needs to check the lock (via acquire) to avoid conflict. What I
    guess I mean is something general for the process saying "never ever
    interrupt this block og code by running code on another thread,
    regardless of whether the other thread(s) check a lock". Thinking more
    about it it seems unreasonable so I'll drop the question.

    Russ
     
    Russell Warren, Oct 19, 2005
    #5
  6. Russell Warren

    Steve Holden Guest

    Re: Threads and socket.setdefaulttimeout

    Russell Warren wrote:
    > Thanks for the detailed repsone... sorry for the lag in responding to
    > it.

    [discussion of problems with timeouts on threaded code's sockets]
    >
    > Not quite what I was after I don't think since potentially interfering
    > code needs to check the lock (via acquire) to avoid conflict. What I
    > guess I mean is something general for the process saying "never ever
    > interrupt this block og code by running code on another thread,
    > regardless of whether the other thread(s) check a lock". Thinking more
    > about it it seems unreasonable so I'll drop the question.
    >


    Well, I'm about out of ideas, but c.l.py is a very inventive group, so
    maybe someone else will be able to contribute a bright thought. Anyone?

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC www.holdenweb.com
    PyCon TX 2006 www.python.org/pycon/
     
    Steve Holden, Oct 19, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. adrian
    Replies:
    4
    Views:
    728
    John J. Lee
    Apr 1, 2005
  2. Sheila King

    socket setdefaulttimeout

    Sheila King, Aug 13, 2005, in forum: Python
    Replies:
    10
    Views:
    865
    Steve Holden
    Aug 17, 2005
  3. rtilley

    socket.setdefaulttimeout()

    rtilley, Oct 8, 2005, in forum: Python
    Replies:
    1
    Views:
    482
    Fredrik Lundh
    Oct 9, 2005
  4. John Nagle
    Replies:
    1
    Views:
    366
    Steve Holden
    Apr 24, 2007
  5. Laszlo Nagy
    Replies:
    1
    Views:
    4,854
    Mark Wooding
    Jan 27, 2009
Loading...

Share This Page