socket and subprocess problem

[goatold]

Hi all,

Here is my problem, see if any one else met this before
In my python code I use subprocess.Popen to run and external program
who will listen to a TCP port. And I also create a socket to connect
to the TCP port that the external program is listening.
I will get 'Connection refused, errno=111' when I try to socket.connect
().
But if I run my subprocess.Popen code and socket code in two separate
python process. Socket will connect just fine. OS is RHEL5 x86_64,
python version is 2.6.1 Pseudo code as below


Class a:
def run()
subprocess.Popen(..)
Class b:
def run():
sock = socket.socket()
sock.connect(..)
#################################
test.py
# socket connect will fail here
a.run()
b.run()
###################################
test1.py
if __name__ = '__main__':
a.run()

test2.py
# socket will connect fine
if __name__ = '__main__':
b.run

 

[Bryan Olson]

In my python code I use subprocess.Popen to run and external program
who will listen to a TCP port. And I also create a socket to connect
to the TCP port that the external program is listening.
I will get 'Connection refused, errno=111' when I try to socket.connect
().

Looks like a race. The first process tries to connect before the
external program gets through socket.listen(). If you think about it,
it's not that surprising.

But if I run my subprocess.Popen code and socket code in two separate
python process. Socket will connect just fine.

It's still a race condition even if the side you want to win almost
always does.

 

[Roy Smith]

In my python code I use subprocess.Popen to run and external program
who will listen to a TCP port. And I also create a socket to connect
to the TCP port that the external program is listening.
I will get 'Connection refused, errno=111' when I try to
socket.connect().

Class a:
def run()
subprocess.Popen(..)
Class b:
def run():
sock = socket.socket()
sock.connect(..)
#################################
test.py
# socket connect will fail here
a.run()
b.run()
###################################
test1.py
if __name__ = '__main__':
a.run()

test2.py
# socket will connect fine
if __name__ = '__main__':
b.run

Sounds like a timing problem. I assume that the process started by a.run()
creates a socket and does a bind/listen/accept sequence on it. The problem
is, there's nothing in your code which guarantees that this happens before
b.run() executes the connect() call.

The cheesy way to test this is to sleep for a second somewhere between
a.run() and b.run(). See if that helps.

If it doesn't, then it's possible the process started by a.run() isn't
doing what it's supposed to do. Try running test1.py, and while it's
running, run netstat to see if you've got something listening on the port
you expect.

 

[goatold]

Guys thanks to point it out.
Yes, it's a race problem. I tried sleep long enough, then I can
connect to the socket. I should add code to try to connect to the
socket for a given time out.

 

[James Mills]

Guys thanks to point it out.
Yes, it's a race problem. I tried sleep long enough, then I can
connect to the socket. I should add code to try to connect to the
socket for a given time out.

This is where event-driven approaches
become really useful

subprocess process:

#1. When my subprocess process has successfully
started notify the parent.
#2. When my subprocess process has successfully
created a listening socket, notify the parent.

parent process:

#1. When our subprocess process has
successfully started a listening socket
initiate a connection.

I could implement a prototype of this
if the OP is interested.

--JamesMills

 

[Bryan Olson]

Guys thanks to point it out.
Yes, it's a race problem. I tried sleep long enough, then I can
connect to the socket. I should add code to try to connect to the
socket for a given time out.

As Roy noted, that's "the cheesy way". Are the kind of programmers who
accept cheesy solutions? Of course not.

The right solution is for the child process to tell the parent when the
port is ready for connections. There are a variety of ways to transfer
that message; the most straightforward is for the child process to write
something to its standard output.

Parent process:

Launch child.
read() child's output, waiting for the ready message.
connect() to the port the child established.

Child process:

Create the socket; bind(); listen().
Write the ready message to stdout.
accept().

There's a subtle yet important point here, that frequently gets lost:


Actually, it's just bind() and listen() that have to happen before the
client can connect(); accept() will not succeed until after a client
connects. In our example the child process is the server, and the
server's readiness alert should go after his call to listen() returns,
but before he calls accept().

 

[Bryan Olson]

subprocess process:

#1. When my subprocess process has successfully
started notify the parent.
#2. When my subprocess process has successfully
created a listening socket, notify the parent.

parent process:

#1. When our subprocess process has
successfully started a listening socket
initiate a connection.

I'd swear James copied my response, except his came first. Even the
formatting came out similar. I hadn't seen his response when I wrote
mine, and wouldn't have bothered posing the same thing again.

 

[James Mills]

I'd swear James copied my response, except his came first. Even the
formatting came out similar. I hadn't seen his response when I wrote mine,
and wouldn't have bothered posing the same thing again.

Great minds think alike huh
You should check out my circuits (1) library!

cheers
James

1: http://trac.softcircuit.net.au/circuits/