subprocess module and blocking

Discussion in 'Python' started by Robin Becker, Jun 12, 2005.

  1. Robin Becker

    Robin Becker Guest

    I'm using a polling loop in a thread that looks approximately like this

    while 1:
    p = find_a_process()
    rc = p.poll()
    if rc is not None:
    out, err = p.communicate()
    #deal with output etc
    sleep(1)

    the process p is opened using

    p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, cwd=cwd)

    stdin is actually never written to.

    I notice that under both win32 and freebsd that things are fine provided
    that the subprocess doesn't write too much to stdout/stderr. However,
    the subprocess seems to lock often if too much is written (under freebsd
    I see a process state of POLL). I assume that the subprocess is filling
    up the pipe and then failing to wake up again. I had expected that
    subprocess would take care of this for me, but possibly I'm being
    utterly clueless and stupid. What should I do to avoid blocking in the
    subprocess?
    --
    Robin Becker
    Robin Becker, Jun 12, 2005
    #1
    1. Advertising

  2. Robin Becker <> writes on Sun, 12 Jun 2005 09:22:52 +0000:
    > I'm using a polling loop in a thread that looks approximately like this
    >
    > while 1:
    > p = find_a_process()
    > rc = p.poll()
    > if rc is not None:
    > out, err = p.communicate()
    > #deal with output etc
    > sleep(1)
    >....
    > I notice that under both win32 and freebsd that things are fine
    > provided that the subprocess doesn't write too much to
    > stdout/stderr. However, the subprocess seems to lock often if too much
    > is written (under freebsd I see a process state of POLL). I assume
    > that the subprocess is filling up the pipe and then failing to wake up
    > again. I had expected that subprocess would take care of this for me,


    You just found out that this is not the case.

    The warning attached to "communicate"s docstring might have
    you averted: "Note: the data read is buffered in memory...
    do not use for large size".

    If subprocess would do what you expect, it would need to
    buffer the output (to make room in the pipes again).
    But, for large data, this could have dramatic consequences.

    Thus, you should use "select" on the pipes to find out
    when to read data.


    Dieter
    Dieter Maurer, Jun 13, 2005
    #2
    1. Advertising

  3. Robin Becker

    Robin Becker Guest

    Dieter Maurer wrote:
    ......
    >
    > You just found out that this is not the case.


    thanks I suppose I was being a moron.

    >
    > The warning attached to "communicate"s docstring might have
    > you averted: "Note: the data read is buffered in memory...
    > do not use for large size".
    >
    > If subprocess would do what you expect, it would need to
    > buffer the output (to make room in the pipes again).
    > But, for large data, this could have dramatic consequences.
    >
    > Thus, you should use "select" on the pipes to find out
    > when to read data.
    >


    Unfortunately select isn't an option for windows (I think that's only for
    sockets). I guess I need to read repeatedly from the stdout etc to get the
    output. Clearly the poll call can't return a status until we've finished
    reading.

    >
    > Dieter



    --
    Robin Becker
    Robin Becker, Jun 14, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    12,515
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    922
    Valentin Tihomirov
    Oct 12, 2003
  3. Marc Carter

    subprocess and non-blocking IO (again)

    Marc Carter, Oct 10, 2005, in forum: Python
    Replies:
    4
    Views:
    976
    Thomas Bellman
    Oct 11, 2005
  4. nukleus
    Replies:
    14
    Views:
    827
    Chris Uppal
    Jan 22, 2007
  5. Kevin Walzer
    Replies:
    2
    Views:
    130
    Terry Reedy
    Dec 22, 2012
Loading...

Share This Page