Need feedback on subprocess-using function

Discussion in 'Python' started by gb345, Oct 3, 2009.

  1. gb345

    gb345 Guest

    I'm relatively new to Python, and I'm trying to get the hang of
    using Python's subprocess module. As an exercise, I wrote the Tac
    class below, which can prints output to a file "in reverse order",
    by piping it through the Unix tac utility. (The idea is to delegate
    the problem of managing the memory for an arbitrarily large task
    to tac.)

    class Tac(object):
    def __init__(self, path):
    out = open(path, 'w')
    self.pipe = subprocess.Popen(['tac'], stdout=out,
    stdin=subprocess.PIPE,
    stderr=subprocess.PIPE)
    def prn(self, string):
    try:
    self.pipe.stdin.write('%s\n' % string)
    except:
    self.close()
    raise

    def close(self):
    p = self.pipe
    p.stdin.close()
    err = p.stderr.read()
    if err:
    raise OSError(err)

    This works OK, as far as I can tell, but I'm not sure that I've
    dotted all the i's and crossed all the t's... E.g., I had to add
    the line "p.stdin.close()" to the close method when I when I ran
    into sporadic deadlock at the p.stderr.read() statement. Are there
    other similar problems lurking in this code? Also, there's no
    robust mechanism for invoking this close method in case of an
    exception (unless the exception happens during the execution of
    prn).

    Any comments and suggestions would be greatly appreciated.

    G
     
    gb345, Oct 3, 2009
    #1
    1. Advertising

  2. gb345

    Nobody Guest

    On Sat, 03 Oct 2009 13:21:00 +0000, gb345 wrote:

    > I'm relatively new to Python, and I'm trying to get the hang of
    > using Python's subprocess module. As an exercise, I wrote the Tac
    > class below, which can prints output to a file "in reverse order",
    > by piping it through the Unix tac utility. (The idea is to delegate
    > the problem of managing the memory for an arbitrarily large task
    > to tac.)


    > self.pipe = subprocess.Popen(['tac'], stdout=out,
    > stdin=subprocess.PIPE,
    > stderr=subprocess.PIPE)


    > This works OK, as far as I can tell, but I'm not sure that I've
    > dotted all the i's and crossed all the t's... E.g., I had to add
    > the line "p.stdin.close()" to the close method when I when I ran
    > into sporadic deadlock at the p.stderr.read() statement. Are there
    > other similar problems lurking in this code?


    Yep. If the process writes more than a buffer-full of data to stderr, it
    will deadlock. tac will block trying to write to stderr, and won't be
    reading its stdin, so your program will block trying to write to tac.

    This is why the POSIX popen() call only lets you attach a pipe to stdin or
    stdout but not both.

    If you want a "double-ended" slave process, you need to use polling or
    non-blocking I/O or asynchronous I/O or multiple threads. I'm not aware of
    any single solution which works on all platforms.

    The easiest way around this problem is to redirect stderr to a temporary
    file and read in the file's contents in the close() method.
     
    Nobody, Oct 5, 2009
    #2
    1. Advertising

  3. gb345

    ryles Guest

    On Oct 4, 9:46 pm, Nobody <> wrote:
    > On Sat, 03 Oct 2009 13:21:00 +0000, gb345 wrote:
    > > I'm relatively new to Python, and I'm trying to get the hang of
    > > using Python's subprocess module.  As an exercise, I wrote the Tac
    > > class below, which can prints output to a file "in reverse order",
    > > by piping it through the Unix tac utility.  (The idea is to delegate
    > > the problem of managing the memory for an arbitrarily large task
    > > to tac.)
    > >         self.pipe = subprocess.Popen(['tac'], stdout=out,
    > >                                      stdin=subprocess.PIPE,
    > >                                      stderr=subprocess.PIPE)
    > > This works OK, as far as I can tell, but I'm not sure that I've
    > > dotted all the i's and crossed all the t's...  E.g., I had to add
    > > the line "p.stdin.close()" to the close method when I when I ran
    > > into sporadic deadlock at the p.stderr.read() statement.  Are there
    > > other similar problems lurking in this code?

    >
    > Yep. If the process writes more than a buffer-full of data to stderr, it
    > will deadlock. tac will block trying to write to stderr, and won't be
    > reading its stdin, so your program will block trying to write to tac.
    >
    > This is why the POSIX popen() call only lets you attach a pipe to stdin or
    > stdout but not both.
    >
    > If you want a "double-ended" slave process, you need to use polling or
    > non-blocking I/O or asynchronous I/O or multiple threads. I'm not aware of
    > any single solution which works on all platforms.
    >
    > The easiest way around this problem is to redirect stderr to a temporary
    > file and read in the file's contents in the close() method.


    There is also Popen.communicate():

    http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
     
    ryles, Oct 5, 2009
    #3
  4. gb345

    Nobody Guest

    On Mon, 05 Oct 2009 02:29:38 -0700, ryles wrote:

    >> If you want a "double-ended" slave process, you need to use polling or
    >> non-blocking I/O or asynchronous I/O or multiple threads. I'm not aware of
    >> any single solution which works on all platforms.
    >>
    >> The easiest way around this problem is to redirect stderr to a temporary
    >> file and read in the file's contents in the close() method.

    >
    > There is also Popen.communicate():


    That doesn't help if you want to write data incrementally.

    You could always lift the code from Popen._communicate(), which uses
    threads for Windows and select() for POSIX.
     
    Nobody, Oct 7, 2009
    #4
  5. gb345

    gb345 Guest

    In <> Nobody <> writes:

    >You could always lift the code from Popen._communicate(), which uses
    >threads for Windows and select() for POSIX.


    Thanks. A lot of useful advice in your replies.

    G.
     
    gb345, Oct 10, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hasani \(remove nospam\)

    Creating templated websites in asp.net -- i need feedback

    Hasani \(remove nospam\), Nov 25, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    1,249
    Jeffrey Palermo [MCP]
    Nov 26, 2004
  2. Rolfe
    Replies:
    2
    Views:
    379
    Rolfe
    Sep 29, 2003
  3. Mark C
    Replies:
    1
    Views:
    295
    Karl Seguin [MVP]
    Jan 23, 2007
  4. hiral
    Replies:
    2
    Views:
    605
    Jean-Michel Pichavant
    May 5, 2010
  5. Jamis Buck
    Replies:
    12
    Views:
    171
    Jamis Buck
    Sep 5, 2004
Loading...

Share This Page