Python Subprocess module

Discussion in 'Python' started by Dave Sampson, Jul 13, 2007.

  1. Dave Sampson

    Dave Sampson Guest

    hey folks,

    A simple question hopefully. despite all my searching I have not found a
    satisfactory response.

    The goal. Interact with a command line program. Simple enough, but the
    key is INTERACT.

    I tried the shell and comand approaches but that initiates, it does not
    allow interaction with the programs.

    So then I went with Popen and such... which then led to the subprocess
    module.

    I can create the object and read a few lines of output. but if I go too
    far then the program hangs. the number of lines will differ depandening
    on many function including the format of an input file so I can;t
    hardcode how many lines to read.

    I want to read all of STDOUT without failing because I went out of range.

    next I want to read the final line of the output because it tells me
    what is required for the next line of input.

    I am supposed to be able to entre 'y' and return for the program to
    continue if I agree with what I see in the stdout.

    A problem exists though that I have tried
    'y'
    'y\n'
    'y\r'

    and nothing seems to get the program going again for I still cant; read
    past the same point in the standard output. then I have to kill and
    start over.

    So the next approach included looking at Pexpect, which got realy
    confusing realy fast and despite running fedora core and python 2.4.4 I
    would like my application to be cross platform and there is no Pexpect
    for Windows That I can see.

    I have checked out many mailing lists and posts and tutorials but they
    all admit to not be samples of complex interactivity. I guess I am
    having that special case.

    ASPN Python cookbok provided some code for a new Popen Class that allows
    for interaction but I don't think I need to go that root....

    I am essentialy trying to build and Python wrapper for a coomand line
    program that i want to build a GUI around.

    Any links or suggestions would be great. Unfortunaetly I don't
    understand that abstract concepts in the manual reference. I am fresh in
    the python world and find examples better than technical abstracts.

    Cheers
    Dave Sampson, Jul 13, 2007
    #1
    1. Advertising

  2. You could start up a pty and do your own Pexpect-like thing, using
    read/readline/write.

    However, this may not be portable to windows. Maybe if you get the python
    that comes with cygwin, or build your own python under the cygwin
    environment - that might be more likely to get you pty's under windows.

    On Fri, 13 Jul 2007 17:15:49 -0400, Dave Sampson wrote:

    > hey folks,
    >
    > A simple question hopefully. despite all my searching I have not found a
    > satisfactory response.
    >
    > The goal. Interact with a command line program. Simple enough, but the
    > key is INTERACT.
    >
    > I tried the shell and comand approaches but that initiates, it does not
    > allow interaction with the programs.
    >
    > So then I went with Popen and such... which then led to the subprocess
    > module.
    >
    > I can create the object and read a few lines of output. but if I go too
    > far then the program hangs. the number of lines will differ depandening
    > on many function including the format of an input file so I can;t
    > hardcode how many lines to read.
    >
    > I want to read all of STDOUT without failing because I went out of range.
    >
    > next I want to read the final line of the output because it tells me
    > what is required for the next line of input.
    >
    > I am supposed to be able to entre 'y' and return for the program to
    > continue if I agree with what I see in the stdout.
    >
    > A problem exists though that I have tried
    > 'y'
    > 'y\n'
    > 'y\r'
    >
    > and nothing seems to get the program going again for I still cant; read
    > past the same point in the standard output. then I have to kill and
    > start over.
    >
    > So the next approach included looking at Pexpect, which got realy
    > confusing realy fast and despite running fedora core and python 2.4.4 I
    > would like my application to be cross platform and there is no Pexpect
    > for Windows That I can see.
    >
    > I have checked out many mailing lists and posts and tutorials but they
    > all admit to not be samples of complex interactivity. I guess I am
    > having that special case.
    >
    > ASPN Python cookbok provided some code for a new Popen Class that allows
    > for interaction but I don't think I need to go that root....
    >
    > I am essentialy trying to build and Python wrapper for a coomand line
    > program that i want to build a GUI around.
    >
    > Any links or suggestions would be great. Unfortunaetly I don't
    > understand that abstract concepts in the manual reference. I am fresh in
    > the python world and find examples better than technical abstracts.
    >
    > Cheers
    Dan Stromberg - Datallegro, Jul 14, 2007
    #2
    1. Advertising

  3. Dave Sampson

    Guest

    On Jul 13, 4:15 pm, Dave Sampson <> wrote:
    > hey folks,
    >
    > A simple question hopefully. despite all my searching I have not found a
    > satisfactory response.
    >
    > The goal. Interact with a command line program. Simple enough, but the
    > key is INTERACT.
    >
    > I tried the shell and comand approaches but that initiates, it does not
    > allow interaction with the programs.
    >
    > So then I went with Popen and such... which then led to the subprocess
    > module.
    >
    > I can create the object and read a few lines of output. but if I go too
    > far then the program hangs. the number of lines will differ depandening
    > on many function including the format of an input file so I can;t
    > hardcode how many lines to read.
    >
    > I want to read all of STDOUT without failing because I went out of range.
    >
    > next I want to read the final line of the output because it tells me
    > what is required for the next line of input.
    >
    > I am supposed to be able to entre 'y' and return for the program to
    > continue if I agree with what I see in the stdout.
    >
    > A problem exists though that I have tried
    > 'y'
    > 'y\n'
    > 'y\r'
    >
    > and nothing seems to get the program going again for I still cant; read
    > past the same point in the standard output. then I have to kill and
    > start over.
    >
    > So the next approach included looking at Pexpect, which got realy
    > confusing realy fast and despite running fedora core and python 2.4.4 I
    > would like my application to be cross platform and there is no Pexpect
    > for Windows That I can see.
    >
    > I have checked out many mailing lists and posts and tutorials but they
    > all admit to not be samples of complex interactivity. I guess I am
    > having that special case.
    >
    > ASPN Python cookbok provided some code for a new Popen Class that allows
    > for interaction but I don't think I need to go that root....
    >
    > I am essentialy trying to build and Python wrapper for a coomand line
    > program that i want to build a GUI around.
    >
    > Any links or suggestions would be great. Unfortunaetly I don't
    > understand that abstract concepts in the manual reference. I am fresh in
    > the python world and find examples better than technical abstracts.
    >
    > Cheers


    There is something called http://www.rutherfurd.net/python/sendkeys/
    send keys for the p.c. you can use a try: except: concept and on the
    windows platform there is also a way to use autoit in python using
    pywin. At one time (2001) there was a module that allowed you to use
    gvim but it has dissapeared unless it is listed somewhere else (gvim
    doesn't seem to work with the sendkey method and it is a shame it has
    alot of cool text functions including command line support)

    http://sourceforge.net/projects/dex-tracker/
    , Jul 14, 2007
    #3
  4. Dave Sampson

    Noah Guest

    On Jul 13, 2:15 pm, Dave Sampson <> wrote:
    > hey folks,
    >
    > A simple question hopefully. despite all my searching I have not found a
    > satisfactory response.
    >
    > The goal. Interact with a command line program. Simple enough, but the
    > key is INTERACT.


    You need Pexpect.

    > So then I went with Popen and such... which then led to the subprocess
    > module. I can create the object and read a few lines of output. but if I go too
    > far then the program hangs. the number of lines will differ depandening
    > on many function including the format of an input file so I can;t
    > hardcode how many lines to read.


    There is nothing you can do about this when using a pipe.
    This is because stdio will change the type of buffering when sending
    stdout and stderr to a pipe. From the client side there is no way you
    can change the buffer mode (you CAN change the mode, but it only
    changes
    it on YOUR side of the pipe -- not your child application's side).

    > and nothing seems to get the program going again for I still cant; read
    > past the same point in the standard output. then I have to kill and
    > start over.


    Your child application is waiting for you, but you can't see what it
    wrote
    becuase it has not filled or flushed the output buffer. When a program
    write to stdout and it is connected to a TTY it will automatically
    flush
    the output buffer when it writes a newline (or, rather, the clib will
    do this).
    But when writing to a pipe it will automatically use block buffering
    and
    will only flush the buffer when it is full.

    It's not very complicated, but it can be very confusing at first.
    It is also surprising that this behavior cannot be changed by the
    parent application.

    > So the next approach included looking atPexpect, which got realy
    > confusing realy fast and despite running fedora core and python 2.4.4 I
    > would like my application to be cross platform and there is noPexpect
    > for Windows That I can see.


    You are going to need something like Pexpect that uses pseudo-ttys
    (pty).
    Pexpect works under Cygwin on Windows, but not under the native
    Windows Python.

    Email me if you have questions about Pexpect and I'll try to help you.

    Yours,
    Noah
    Noah, Jul 15, 2007
    #4
  5. Dave Sampson wrote:
    > I am supposed to be able to entre 'y' and return for the program to
    > continue if I agree with what I see in the stdout.
    >
    > A problem exists though that I have tried
    > 'y'
    > 'y\n'
    > 'y\r'


    Depending on the platform, you may need to send '\r\n'.


    > ASPN Python cookbok provided some code for a new Popen Class that allows
    > for interaction but I don't think I need to go that root....


    You probably mean:
    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

    Why is that not sufficient? It will allow interaction with a called
    subprocess, in a cross-platform manner, etc.


    - Josiah
    Josiah Carlson, Jul 16, 2007
    #5
  6. Dave Sampson

    Dave Sampson Guest

    On Jul 13, 7:50 pm, Dan Stromberg - Datallegro
    <> wrote:
    > You could start up a pty and do your own Pexpect-like thing, using
    > read/readline/write.
    >
    > However, this may not be portable to windows. Maybe if you get the python
    > that comes with cygwin, or build your own python under the cygwin



    thanks for the suggestion. I will follow up on Noah's pexpect
    sugestion bellow.

    thanks for the input, cross platform is key with low overhead.
    Dave Sampson, Jul 21, 2007
    #6
  7. Dave Sampson

    Dave Sampson Guest

    send keys for the p.c. you can use a try: except: concept and on the
    > windows platform there is also a way to use autoit in python using
    > pywin.


    thanks for the uggestion about sendkeys. I might be able to build that
    into the wrapper.

    So you're sugesting:

    NIX = Pexpect

    WINDOZE = sendkeys

    Correct?
    Dave Sampson, Jul 21, 2007
    #7
  8. Dave Sampson

    Dave Sampson Guest

    Thanks for your response. It;s good to see you're promtoing Pexpect. I
    was actualy wokring on a project at work that could have used Pexpect
    to automate the creation of maps by tieing togethr a bunch of
    processes through SSH and Pexpect looked like the money shot.

    however it was enough of a battle to bring Python into the game since
    it is government that bringing CYGWIN was a battle not worth
    fighting. Which is unfortunate because I think Pexpect was exactly
    what we needed in that case
    >
    > You need Pexpect.
    >


    > There is nothing you can do about this when using a pipe.
    > This is because stdio will change the type of buffering when sending
    > stdout and stderr to a pipe. From the client side there is no way you
    > can change the buffer mode (you CAN change the mode, but it only
    > changes
    > it on YOUR side of the pipe -- not your child application's side).



    I obviously lack skills and understanding in many aspects of
    programming. After all I'm a geographer / recreologist not a comp-sci
    professional.

    I understand the CONCEPT of pipes but don't know how to use or
    implement.. the bigest things I ever do is something like

    dmesg | grep wlan0

    Where | is my pipe... but that is the extent of my knowledge and
    skill.

    Pexpect may very well be the option I need, and having a project that
    supports the module is great. I am curous what is stopping Pexpect
    from becoming cross platform.

    Above, a user suggested implementing send keys in windows. Why not
    include that as an try/except in Pexpect instead of my wrapper? That
    way it can benifit many more people?

    Cheers
    Dave Sampson, Jul 21, 2007
    #8
  9. Dave Sampson

    Dave Sampson Guest


    > You probably mean:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
    >
    > Why is that not sufficient? It will allow interaction with a called
    > subprocess, in a cross-platform manner, etc.
    >
    > - Josiah


    Sorry maybe 'not sufficient' was the wrong excuse. Maybe the right
    thing to say is that when I look at it I don't know if I'm supposed to
    copy paste in my own module, or paste into a seprate module, or use it
    in a pre-existing module. Does the liscence for the content through
    ASPN even allow reuse of code. or Am I supposed to modify it?

    If I use, how is it to be used?. Wher eare the exmpales and
    documentation.

    By sufficient I mean I don't have sufficient info from the site to
    implement it. And I don't have sufficient know how to implement it
    base don the comments in the source code.

    IN SHORT... IT'S SCARY....

    So if you can answer any of those questions I am open.
    Dave Sampson, Jul 21, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Harishankar
    Replies:
    18
    Views:
    602
    Harishankar
    Apr 23, 2008
  2. Harishankar
    Replies:
    2
    Views:
    310
    Mark Wooding
    Apr 23, 2008
  3. Michel Lespinasse

    Descriptor leak in python 2.4 subprocess module

    Michel Lespinasse, Aug 28, 2008, in forum: Python
    Replies:
    0
    Views:
    311
    Michel Lespinasse
    Aug 28, 2008
  4. Matt Nordhoff

    Re: using subprocess module in Python CGI

    Matt Nordhoff, Dec 24, 2008, in forum: Python
    Replies:
    0
    Views:
    953
    Matt Nordhoff
    Dec 24, 2008
  5. hiral
    Replies:
    2
    Views:
    583
    Jean-Michel Pichavant
    May 5, 2010
Loading...

Share This Page