socket and subprocess problem

Discussion in 'Python' started by goatold@gmail.com, Dec 16, 2008.

  1. Guest

    Hi all,

    Here is my problem, see if any one else met this before
    In my python code I use subprocess.Popen to run and external program
    who will listen to a TCP port. And I also create a socket to connect
    to the TCP port that the external program is listening.
    I will get 'Connection refused, errno=111' when I try to socket.connect
    ().
    But if I run my subprocess.Popen code and socket code in two separate
    python process. Socket will connect just fine. OS is RHEL5 x86_64,
    python version is 2.6.1 Pseudo code as below


    Class a:
    def run()
    subprocess.Popen(..)
    Class b:
    def run():
    sock = socket.socket()
    sock.connect(..)
    #################################
    test.py
    # socket connect will fail here
    a.run()
    b.run()
    ###################################
    test1.py
    if __name__ = '__main__':
    a.run()

    test2.py
    # socket will connect fine
    if __name__ = '__main__':
    b.run
    , Dec 16, 2008
    #1
    1. Advertising

  2. Bryan Olson Guest

    wrote:
    > In my python code I use subprocess.Popen to run and external program
    > who will listen to a TCP port. And I also create a socket to connect
    > to the TCP port that the external program is listening.
    > I will get 'Connection refused, errno=111' when I try to socket.connect
    > ().


    Looks like a race. The first process tries to connect before the
    external program gets through socket.listen(). If you think about it,
    it's not that surprising.

    > But if I run my subprocess.Popen code and socket code in two separate
    > python process. Socket will connect just fine.


    It's still a race condition even if the side you want to win almost
    always does.


    --
    --Bryan
    Bryan Olson, Dec 16, 2008
    #2
    1. Advertising

  3. Roy Smith Guest

    In article
    <>,
    wrote:

    > In my python code I use subprocess.Popen to run and external program
    > who will listen to a TCP port. And I also create a socket to connect
    > to the TCP port that the external program is listening.
    > I will get 'Connection refused, errno=111' when I try to
    > socket.connect().


    > Class a:
    > def run()
    > subprocess.Popen(..)
    > Class b:
    > def run():
    > sock = socket.socket()
    > sock.connect(..)
    > #################################
    > test.py
    > # socket connect will fail here
    > a.run()
    > b.run()
    > ###################################
    > test1.py
    > if __name__ = '__main__':
    > a.run()
    >
    > test2.py
    > # socket will connect fine
    > if __name__ = '__main__':
    > b.run


    Sounds like a timing problem. I assume that the process started by a.run()
    creates a socket and does a bind/listen/accept sequence on it. The problem
    is, there's nothing in your code which guarantees that this happens before
    b.run() executes the connect() call.

    The cheesy way to test this is to sleep for a second somewhere between
    a.run() and b.run(). See if that helps.

    If it doesn't, then it's possible the process started by a.run() isn't
    doing what it's supposed to do. Try running test1.py, and while it's
    running, run netstat to see if you've got something listening on the port
    you expect.
    Roy Smith, Dec 16, 2008
    #3
  4. Guest

    Guys thanks to point it out.
    Yes, it's a race problem. I tried sleep long enough, then I can
    connect to the socket. I should add code to try to connect to the
    socket for a given time out.

    Roy Smith wrote:
    > In article
    > <>,
    > wrote:
    >
    > > In my python code I use subprocess.Popen to run and external program
    > > who will listen to a TCP port. And I also create a socket to connect
    > > to the TCP port that the external program is listening.
    > > I will get 'Connection refused, errno=111' when I try to
    > > socket.connect().

    >
    > > Class a:
    > > def run()
    > > subprocess.Popen(..)
    > > Class b:
    > > def run():
    > > sock = socket.socket()
    > > sock.connect(..)
    > > #################################
    > > test.py
    > > # socket connect will fail here
    > > a.run()
    > > b.run()
    > > ###################################
    > > test1.py
    > > if __name__ = '__main__':
    > > a.run()
    > >
    > > test2.py
    > > # socket will connect fine
    > > if __name__ = '__main__':
    > > b.run

    >
    > Sounds like a timing problem. I assume that the process started by a.run()
    > creates a socket and does a bind/listen/accept sequence on it. The problem
    > is, there's nothing in your code which guarantees that this happens before
    > b.run() executes the connect() call.
    >
    > The cheesy way to test this is to sleep for a second somewhere between
    > a.run() and b.run(). See if that helps.
    >
    > If it doesn't, then it's possible the process started by a.run() isn't
    > doing what it's supposed to do. Try running test1.py, and while it's
    > running, run netstat to see if you've got something listening on the port
    > you expect.
    , Dec 16, 2008
    #4
  5. James Mills Guest

    On Tue, Dec 16, 2008 at 3:30 PM, <> wrote:
    > Guys thanks to point it out.
    > Yes, it's a race problem. I tried sleep long enough, then I can
    > connect to the socket. I should add code to try to connect to the
    > socket for a given time out.


    This is where event-driven approaches
    become really useful :)

    subprocess process:

    #1. When my subprocess process has successfully
    started notify the parent.
    #2. When my subprocess process has successfully
    created a listening socket, notify the parent.

    parent process:

    #1. When our subprocess process has
    successfully started a listening socket
    initiate a connection.

    I could implement a prototype of this
    if the OP is interested.

    --JamesMills
    James Mills, Dec 16, 2008
    #5
  6. Bryan Olson Guest

    wrote:
    > Guys thanks to point it out.
    > Yes, it's a race problem. I tried sleep long enough, then I can
    > connect to the socket. I should add code to try to connect to the
    > socket for a given time out.


    As Roy noted, that's "the cheesy way". Are the kind of programmers who
    accept cheesy solutions? Of course not.

    The right solution is for the child process to tell the parent when the
    port is ready for connections. There are a variety of ways to transfer
    that message; the most straightforward is for the child process to write
    something to its standard output.

    Parent process:

    Launch child.
    read() child's output, waiting for the ready message.
    connect() to the port the child established.

    Child process:

    Create the socket; bind(); listen().
    Write the ready message to stdout.
    accept().

    There's a subtle yet important point here, that frequently gets lost:

    Roy Smith wrote:
    >> Sounds like a timing problem. I assume that the process started by a.run()
    >> creates a socket and does a bind/listen/accept sequence on it. The problem
    >> is, there's nothing in your code which guarantees that this happens before
    >> b.run() executes the connect() call.


    Actually, it's just bind() and listen() that have to happen before the
    client can connect(); accept() will not succeed until after a client
    connects. In our example the child process is the server, and the
    server's readiness alert should go after his call to listen() returns,
    but before he calls accept().


    --
    --Bryan
    Bryan Olson, Dec 16, 2008
    #6
  7. Bryan Olson Guest

    James Mills wrote:
    > subprocess process:
    >
    > #1. When my subprocess process has successfully
    > started notify the parent.
    > #2. When my subprocess process has successfully
    > created a listening socket, notify the parent.
    >
    > parent process:
    >
    > #1. When our subprocess process has
    > successfully started a listening socket
    > initiate a connection.


    I'd swear James copied my response, except his came first. Even the
    formatting came out similar. I hadn't seen his response when I wrote
    mine, and wouldn't have bothered posing the same thing again.


    --
    --Bryan
    Bryan Olson, Dec 18, 2008
    #7
  8. James Mills Guest

    On Thu, Dec 18, 2008 at 8:00 PM, Bryan Olson <> wrote:
    > I'd swear James copied my response, except his came first. Even the
    > formatting came out similar. I hadn't seen his response when I wrote mine,
    > and wouldn't have bothered posing the same thing again.


    Great minds think alike huh :)
    You should check out my circuits (1) library!

    cheers
    James

    1: http://trac.softcircuit.net.au/circuits/
    James Mills, Dec 18, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Laszlo Nagy
    Replies:
    1
    Views:
    4,761
    Mark Wooding
    Jan 27, 2009
  2. Jean-Paul Calderone
    Replies:
    0
    Views:
    945
    Jean-Paul Calderone
    Jan 27, 2009
  3. Laszlo Nagy
    Replies:
    0
    Views:
    526
    Laszlo Nagy
    Feb 1, 2009
  4. Steve Holden
    Replies:
    0
    Views:
    644
    Steve Holden
    Feb 1, 2009
  5. Steve Holden
    Replies:
    1
    Views:
    699
Loading...

Share This Page