Socket being garbage collected too early

Discussion in 'Python' started by Scott Robinson, Dec 16, 2004.

  1. I have been having trouble with the garbage collector and sockets.
    Unfortunately, google keeps telling me that the problem is the garbage
    collector ignoring dead (closed?) sockets instead of removing live
    ones. My problem is


    x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    do_stuff(x.sock)


    def do_stuff(sock):
    sock_list.append(sock)


    once do_stuff finishes, x.sock disappears, and I can only believe it
    is being garbage collected. I'd like to hear the standard means for
    avoiding this issue (gc appears to have only the interface to declare
    something garbage, not to declare something not garbage).

    Scott Robinson
    Scott Robinson, Dec 16, 2004
    #1
    1. Advertising

  2. Scott Robinson

    Steve Holden Guest

    Scott Robinson wrote:

    > I have been having trouble with the garbage collector and sockets.
    > Unfortunately, google keeps telling me that the problem is the garbage
    > collector ignoring dead (closed?) sockets instead of removing live
    > ones. My problem is
    >
    >
    > x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    > do_stuff(x.sock)
    >
    >
    > def do_stuff(sock):
    > sock_list.append(sock)
    >
    >
    > once do_stuff finishes, x.sock disappears, and I can only believe it
    > is being garbage collected. I'd like to hear the standard means for
    > avoiding this issue (gc appears to have only the interface to declare
    > something garbage, not to declare something not garbage).
    >
    > Scott Robinson
    >

    Unfortunately, assuming it's being garbage collected might turn out to
    be incorrect. What evidence do you have that the socket "disappears"? Do
    you get a segmentation ault, or what?

    If the socket simply fails to work that would be a different case
    altogether, but it seems to me that we need a bit more evodence that the
    anecdotal stuff you've provided so far.

    Quite apart from anything else, by the way, the code you posted appears
    to use a global sock_list. A reference b y that list would in any case
    stop the socket from being garbage collected (quite apart from the fact
    that the socket module itself will do so as long as the socket is open).

    So, could we see an error message, or some other evidence of what is
    going on. For example, after the call to do_stuff(), what do you see if you

    print sock_list

    for example. I think your initial hypothesis is insufficient.

    regards
    Steve
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
    Steve Holden, Dec 17, 2004
    #2
    1. Advertising

  3. Scott Robinson

    Mike Meyer Guest

    Scott Robinson <> writes:

    > I have been having trouble with the garbage collector and sockets.
    > Unfortunately, google keeps telling me that the problem is the garbage
    > collector ignoring dead (closed?) sockets instead of removing live
    > ones. My problem is
    >
    >
    > x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    > do_stuff(x.sock)
    >
    >
    > def do_stuff(sock):
    > sock_list.append(sock)
    >
    >
    > once do_stuff finishes, x.sock disappears, and I can only believe it
    > is being garbage collected. I'd like to hear the standard means for
    > avoiding this issue (gc appears to have only the interface to declare
    > something garbage, not to declare something not garbage).


    The code as shown doesn't work:

    >> import socket
    >> def do_stuff(sock):

    .. sock_list.append(sock)
    ..
    >> sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    >> do_stuff(sock)

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 2, in do_stuff
    NameError: global name 'sock_list' is not defined
    >>


    If you add "sock_list = []" just before the def of do_stuff, the code
    will work, and your sockets won't get garbage collected.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
    Mike Meyer, Dec 17, 2004
    #3
  4. Scott Robinson

    David Bolen Guest

    Scott Robinson <> writes:

    > I have been having trouble with the garbage collector and sockets.


    Are you actually getting errors or is this just theoretical?

    > Unfortunately, google keeps telling me that the problem is the garbage
    > collector ignoring dead (closed?) sockets instead of removing live
    > ones. My problem is
    >
    >
    > x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    > do_stuff(x.sock)
    >
    >
    > def do_stuff(sock):
    > sock_list.append(sock)
    >
    > once do_stuff finishes, x.sock disappears, and I can only believe it
    > is being garbage collected.


    Can you clarify this? What do you mean by "x.sock" disappears? Are
    you getting a NameError later when trying to use "x.sock"?

    x.sock is just a name binding, so it is not really involved in garbage
    collection (GC applies to the objects to which names are bound).

    In this case, you need to include much more in the way of code (a
    fully running, but smallest possible, snippet of code would be best),
    since the above can be interpreted many ways. At the least, it's very
    important to include information about the namespace within which
    those two code snippets run if anyone is likely to be able to give you
    a good answer. Also, being very precise about the error condition you
    are experiencing (including actual error messages, tracebacks, etc...)
    is crucial.

    Is 'x' referencing a local or global object, and does that socket code
    occur within a method, a function, or what? Also, in do_stuff, where
    is sock_list defined? Is it local, global?

    If, as written, sock_list is a local name to do_stuff, then that
    binding is going to disappear when do_stuff completes, thus, the list
    to which it is bound will be destroyed, including all references to
    objects that the list may contain. So at that point, when you return
    from do_stuff, the only reference to the socket object will be in
    x.sock. But if 'x' is also local to the function/method where the
    call to do_stuff is, the name binding will be removed when the
    function/method returns, at which point there will be no references to
    the socket object, and yes, it will be destroyed.

    But if sock_list is global, and continues to exist when do_stuff
    completes, then the reference it contains to the socket will keep the
    socket object alive even if you remove the x.sock binding.

    -- David
    David Bolen, Dec 17, 2004
    #4
  5. On 16 Dec 2004 20:38:29 -0500, David Bolen <> wrote:

    >Scott Robinson <> writes:
    >
    >> I have been having trouble with the garbage collector and sockets.

    >
    >Are you actually getting errors or is this just theoretical?
    >
    >> Unfortunately, google keeps telling me that the problem is the garbage
    >> collector ignoring dead (closed?) sockets instead of removing live
    >> ones. My problem is
    >>
    >>
    >> x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    >> do_stuff(x.sock)
    >>
    >>
    >> def do_stuff(sock):
    >> sock_list.append(sock)
    >>
    >> once do_stuff finishes, x.sock disappears, and I can only believe it
    >> is being garbage collected.

    >
    >Can you clarify this? What do you mean by "x.sock" disappears? Are
    >you getting a NameError later when trying to use "x.sock"?
    >
    >x.sock is just a name binding, so it is not really involved in garbage
    >collection (GC applies to the objects to which names are bound).
    >
    >In this case, you need to include much more in the way of code (a
    >fully running, but smallest possible, snippet of code would be best),
    >since the above can be interpreted many ways. At the least, it's very
    >important to include information about the namespace within which
    >those two code snippets run if anyone is likely to be able to give you
    >a good answer. Also, being very precise about the error condition you
    >are experiencing (including actual error messages, tracebacks, etc...)
    >is crucial.
    >
    >Is 'x' referencing a local or global object, and does that socket code
    >occur within a method, a function, or what? Also, in do_stuff, where
    >is sock_list defined? Is it local, global?
    >
    >If, as written, sock_list is a local name to do_stuff, then that
    >binding is going to disappear when do_stuff completes, thus, the list
    >to which it is bound will be destroyed, including all references to
    >objects that the list may contain. So at that point, when you return
    >from do_stuff, the only reference to the socket object will be in
    >x.sock. But if 'x' is also local to the function/method where the
    >call to do_stuff is, the name binding will be removed when the
    >function/method returns, at which point there will be no references to
    >the socket object, and yes, it will be destroyed.
    >
    >But if sock_list is global, and continues to exist when do_stuff
    >completes, then the reference it contains to the socket will keep the
    >socket object alive even if you remove the x.sock binding.
    >
    >-- David

    (so much for Python being executable psuedocode).

    It looks like I was completely wrong. The problem I ran into was not
    checking into baseHTTPserver and looking for self.close_connection=1.

    I am pretty sure this happened to me before, and I rewrote the code to
    avoid sockets being closed after being referenced to a live object,
    but I can't reproduce it.

    Anyway, here is my debugged test rig. It works. Ignore it.


    Scott
    [test program follows]
    import socket, thread, time

    class sock_holder:
    pass

    HOST = '127.0.0.1'
    PORT = 2004

    def echo_server(port):
    print "started at",port
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, port))
    s.listen(1)
    conn, addr = s.accept()
    print 'Connected by', addr
    while 1:
    data = conn.recv(1024)
    if not data: break
    conn.send(data)
    conn.close()

    def wait_and_speak(a):
    time.sleep(5)
    try:
    a.sock.send("this is message 2")
    except:
    print "error on message 2"
    try:
    data = a.sock.recv(1024)
    print data
    except:
    print "error recieving message 2"
    a.sock.close()

    def other_thread(a):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    thread.start_new(echo_server,(PORT,))
    time.sleep(1)
    s.connect((HOST,PORT ))
    s.send('Hello, port 1')
    data = s.recv(1024)
    print data
    a.sock=s

    thread.start_new(wait_and_speak,(a,))

    a=sock_holder()
    thread.start_new(other_thread,(a,))
    time.sleep(10)
    Scott Robinson, Dec 18, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?Sg==?=

    When are static members garbage collected?

    =?Utf-8?B?Sg==?=, Feb 25, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    336
    bruce barker
    Feb 25, 2004
  2. Cheng Thao
    Replies:
    0
    Views:
    375
    Cheng Thao
    Aug 6, 2003
  3. Mark McKay
    Replies:
    5
    Views:
    868
    xarax
    Oct 3, 2003
  4. Kent Johnson

    When is a thread garbage collected?

    Kent Johnson, Mar 16, 2005, in forum: Python
    Replies:
    1
    Views:
    436
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Mar 16, 2005
  5. eden li
    Replies:
    2
    Views:
    103
    Ryan Davis
    Jun 14, 2009
Loading...

Share This Page