Socket being garbage collected too early

[Scott Robinson]

I have been having trouble with the garbage collector and sockets.
Unfortunately, google keeps telling me that the problem is the garbage
collector ignoring dead (closed?) sockets instead of removing live
ones. My problem is


x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
do_stuff(x.sock)


def do_stuff(sock):
sock_list.append(sock)


once do_stuff finishes, x.sock disappears, and I can only believe it
is being garbage collected. I'd like to hear the standard means for
avoiding this issue (gc appears to have only the interface to declare
something garbage, not to declare something not garbage).

Scott Robinson

 

[Steve Holden]

I have been having trouble with the garbage collector and sockets.
Unfortunately, google keeps telling me that the problem is the garbage
collector ignoring dead (closed?) sockets instead of removing live
ones. My problem is


x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
do_stuff(x.sock)


def do_stuff(sock):
sock_list.append(sock)


once do_stuff finishes, x.sock disappears, and I can only believe it
is being garbage collected. I'd like to hear the standard means for
avoiding this issue (gc appears to have only the interface to declare
something garbage, not to declare something not garbage).

Scott Robinson

Unfortunately, assuming it's being garbage collected might turn out to
be incorrect. What evidence do you have that the socket "disappears"? Do
you get a segmentation ault, or what?

If the socket simply fails to work that would be a different case
altogether, but it seems to me that we need a bit more evodence that the
anecdotal stuff you've provided so far.

Quite apart from anything else, by the way, the code you posted appears
to use a global sock_list. A reference b y that list would in any case
stop the socket from being garbage collected (quite apart from the fact
that the socket module itself will do so as long as the socket is open).

So, could we see an error message, or some other evidence of what is
going on. For example, after the call to do_stuff(), what do you see if you

print sock_list

for example. I think your initial hypothesis is insufficient.

regards
Steve

 

[Mike Meyer]

I have been having trouble with the garbage collector and sockets.
Unfortunately, google keeps telling me that the problem is the garbage
collector ignoring dead (closed?) sockets instead of removing live
ones. My problem is


x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
do_stuff(x.sock)


def do_stuff(sock):
sock_list.append(sock)


once do_stuff finishes, x.sock disappears, and I can only believe it
is being garbage collected. I'd like to hear the standard means for
avoiding this issue (gc appears to have only the interface to declare
something garbage, not to declare something not garbage).

The code as shown doesn't work:
.. sock_list.append(sock)
.. Traceback (most recent call last):
File "<stdin>", line 1, in ?

If you add "sock_list = []" just before the def of do_stuff, the code
will work, and your sockets won't get garbage collected.

<mike

 

[David Bolen]

I have been having trouble with the garbage collector and sockets.

Are you actually getting errors or is this just theoretical?

Unfortunately, google keeps telling me that the problem is the garbage
collector ignoring dead (closed?) sockets instead of removing live
ones. My problem is


x.sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
do_stuff(x.sock)


def do_stuff(sock):
sock_list.append(sock)

once do_stuff finishes, x.sock disappears, and I can only believe it
is being garbage collected.

Can you clarify this? What do you mean by "x.sock" disappears? Are
you getting a NameError later when trying to use "x.sock"?

x.sock is just a name binding, so it is not really involved in garbage
collection (GC applies to the objects to which names are bound).

In this case, you need to include much more in the way of code (a
fully running, but smallest possible, snippet of code would be best),
since the above can be interpreted many ways. At the least, it's very
important to include information about the namespace within which
those two code snippets run if anyone is likely to be able to give you
a good answer. Also, being very precise about the error condition you
are experiencing (including actual error messages, tracebacks, etc...)
is crucial.

Is 'x' referencing a local or global object, and does that socket code
occur within a method, a function, or what? Also, in do_stuff, where
is sock_list defined? Is it local, global?

If, as written, sock_list is a local name to do_stuff, then that
binding is going to disappear when do_stuff completes, thus, the list
to which it is bound will be destroyed, including all references to
objects that the list may contain. So at that point, when you return
from do_stuff, the only reference to the socket object will be in
x.sock. But if 'x' is also local to the function/method where the
call to do_stuff is, the name binding will be removed when the
function/method returns, at which point there will be no references to
the socket object, and yes, it will be destroyed.

But if sock_list is global, and continues to exist when do_stuff
completes, then the reference it contains to the socket will keep the
socket object alive even if you remove the x.sock binding.

-- David

 

[Scott Robinson]

Are you actually getting errors or is this just theoretical?


Can you clarify this? What do you mean by "x.sock" disappears? Are
you getting a NameError later when trying to use "x.sock"?

x.sock is just a name binding, so it is not really involved in garbage
collection (GC applies to the objects to which names are bound).

In this case, you need to include much more in the way of code (a
fully running, but smallest possible, snippet of code would be best),
since the above can be interpreted many ways. At the least, it's very
important to include information about the namespace within which
those two code snippets run if anyone is likely to be able to give you
a good answer. Also, being very precise about the error condition you
are experiencing (including actual error messages, tracebacks, etc...)
is crucial.

Is 'x' referencing a local or global object, and does that socket code
occur within a method, a function, or what? Also, in do_stuff, where
is sock_list defined? Is it local, global?

If, as written, sock_list is a local name to do_stuff, then that
binding is going to disappear when do_stuff completes, thus, the list
to which it is bound will be destroyed, including all references to
objects that the list may contain. So at that point, when you return
from do_stuff, the only reference to the socket object will be in
x.sock. But if 'x' is also local to the function/method where the
call to do_stuff is, the name binding will be removed when the
function/method returns, at which point there will be no references to
the socket object, and yes, it will be destroyed.

But if sock_list is global, and continues to exist when do_stuff
completes, then the reference it contains to the socket will keep the
socket object alive even if you remove the x.sock binding.

-- David

(so much for Python being executable psuedocode).

It looks like I was completely wrong. The problem I ran into was not
checking into baseHTTPserver and looking for self.close_connection=1.

I am pretty sure this happened to me before, and I rewrote the code to
avoid sockets being closed after being referenced to a live object,
but I can't reproduce it.

Anyway, here is my debugged test rig. It works. Ignore it.


Scott
[test program follows]
import socket, thread, time

class sock_holder:
pass

HOST = '127.0.0.1'
PORT = 2004

def echo_server(port):
print "started at",port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, port))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(1024)
if not data: break
conn.send(data)
conn.close()

def wait_and_speak(a):
time.sleep(5)
try:
a.sock.send("this is message 2")
except:
print "error on message 2"
try:
data = a.sock.recv(1024)
print data
except:
print "error recieving message 2"
a.sock.close()

def other_thread(a):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
thread.start_new(echo_server,(PORT,))
time.sleep(1)
s.connect((HOST,PORT ))
s.send('Hello, port 1')
data = s.recv(1024)
print data
a.sock=s

thread.start_new(wait_and_speak,(a,))

a=sock_holder()
thread.start_new(other_thread,(a,))
time.sleep(10)