Popen Question

Discussion in 'Python' started by moogyd, Nov 4, 2010.

  1. moogyd

    moogyd Guest

    Hi,
    I usually use csh for my simulation control scripts, but these scripts
    are becoming more complex, so I plan to use python for the next
    project.
    To this end, I am looking at subprocess.Popen() to actually call the
    simulations, and have a very basic question which is demonstrated
    below.

    [sde:staff@lbux03 ~]$ python
    Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
    [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os, subprocess
    >>> os.environ['MYVAR'] = "myval"
    >>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)
    >>>
    >>> p = subprocess.Popen(['echo', '$MYVAR'])
    >>> $MYVAR


    >>> p = subprocess.Popen('echo $MYVAR',shell=True)
    >>> myval


    >>> p = subprocess.Popen('echo $MYVAR')

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    errread, errwrite)
    File "/usr/lib64/python2.6/subprocess.py", line 1106, in
    _execute_child
    raise child_exception
    OSError: [Errno 2] No such file or directory

    I am not really sure I understand these results.
    1) No idea what is going on
    2) As (1). What isn't myval printed out (rather than $MYVAR)
    3) Works as I wanted it to
    4) Why do I need shell=True ?
    The documentation isn't very clear to me (it seems you need to
    understand the underlying system calls).

    Can anyone explain (or provide link) for this behaviour in simple
    English?
    Thanks,
    Steven
     
    moogyd, Nov 4, 2010
    #1
    1. Advertising

  2. moogyd

    Ravi Guest

    On Nov 4, 7:06 pm, moogyd <> wrote:
    > Hi,
    > I usually use csh for my simulation control scripts, but these scripts
    > are becoming more complex, so I plan to use python for the next
    > project.
    > To this end, I am looking at subprocess.Popen() to actually call the
    > simulations, and have a very basic question which is demonstrated
    > below.
    >
    > [sde:staff@lbux03 ~]$ python
    > Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
    > [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
    > Type "help", "copyright", "credits" or "license" for more information.
    >
    > >>> import os, subprocess
    > >>> os.environ['MYVAR'] = "myval"
    > >>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)

    >
    > >>> p = subprocess.Popen(['echo', '$MYVAR'])
    > >>> $MYVAR
    > >>> p = subprocess.Popen('echo $MYVAR',shell=True)
    > >>> myval
    > >>> p = subprocess.Popen('echo $MYVAR')

    >
    > Traceback (most recent call last):
    >   File "<stdin>", line 1, in <module>
    >   File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    >     errread, errwrite)
    >   File "/usr/lib64/python2.6/subprocess.py", line 1106, in
    > _execute_child
    >     raise child_exception
    > OSError: [Errno 2] No such file or directory
    >
    > I am not really sure I understand these results.
    > 1) No idea what is going on
    > 2) As (1). What isn't myval printed out (rather than $MYVAR)
    > 3) Works as I wanted it to
    > 4) Why do I need shell=True ?
    > The documentation isn't very clear to me (it seems you need to
    > understand the underlying system calls).
    >
    > Can anyone explain (or provide link) for this behaviour in simple
    > English?
    > Thanks,
    > Steven


    try giving /bin/echo
     
    Ravi, Nov 4, 2010
    #2
    1. Advertising

  3. moogyd <> writes:

    >>>> import os, subprocess
    >>>> os.environ['MYVAR'] = "myval"
    >>>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)
    >>>>
    >>>> p = subprocess.Popen(['echo', '$MYVAR'])
    >>>> $MYVAR

    >
    >>>> p = subprocess.Popen('echo $MYVAR',shell=True)
    >>>> myval

    >
    >>>> p = subprocess.Popen('echo $MYVAR')

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    > errread, errwrite)
    > File "/usr/lib64/python2.6/subprocess.py", line 1106, in
    > _execute_child
    > raise child_exception
    > OSError: [Errno 2] No such file or directory
    >
    > I am not really sure I understand these results.
    > 1) No idea what is going on
    > 2) As (1). What isn't myval printed out (rather than $MYVAR)
    > 3) Works as I wanted it to
    > 4) Why do I need shell=True ?


    Expanding $MYVAR into its value is a feature of the shell (afaik all
    shells use the same syntax). Popen without shell=True uses the execvp()
    system call directly, without going through the shell variable expansion
    process (cases 2 and 4 above). For example, case 4 above asks execvp to
    (find and) execute a program named "echo $MYVAR" (an 11-letter name,
    where the fifth letter is space and the sixth is $ -- a perfectly valid
    file/program name).

    Then, if you use shell=True with a list, only the first word is used as
    a command, and the others are kept in positional parameters. That's why
    your first try fails (try 'sh -c echo $HOME' in a shell, without the
    single quotes, and you'll get empty output).

    > The documentation isn't very clear to me (it seems you need to
    > understand the underlying system calls).


    You're probably right. The base fact here is: the use of variables is a
    feature of the shell. No shell, no variable.

    > Can anyone explain (or provide link) for this behaviour in simple
    > English?


    Shell variables are explained in detail in any shell man page. The
    execvp() system call has its own man page.

    -- Alain.
     
    Alain Ketterlin, Nov 4, 2010
    #3
  4. moogyd

    Chris Torek Guest

    In article <>
    moogyd <> wrote:
    >[sde:staff@lbux03 ~]$ python
    >Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
    >[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
    >Type "help", "copyright", "credits" or "license" for more information.
    >>>> import os, subprocess
    >>>> os.environ['MYVAR'] = "myval"
    >>>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)


    Alain Ketterlin has already explained these to some extent.
    Here is a bit more.

    This runs, underneath:

    ['/bin/sh', '-c', 'echo', '$MYVAR']

    (with arguments expressed as a Python list). /bin/sh takes the
    string after '-c' as a command, and the remaining argument(s) if
    any are assigned to positional parameters ($0, $1, etc).

    If you replace the command with something a little more explicit,
    you can see this:

    >>> p = subprocess.Popen(

    ... [r'echo \$0=$0 \$1=$1', 'arg0', '$MYVAR'], shell=True)
    >>> $0=arg0 $1=$MYVAR

    p.wait()
    0
    >>>


    (I like to call p.communicate() or p.wait(), although p.communicate()
    is pretty much a no-op if you have not done any redirecting. Note that
    p.communicate() does a p.wait() for you.)

    >>>> p = subprocess.Popen(['echo', '$MYVAR'])
    >>>> $MYVAR


    This time, as Alain noted, the shell does not get involved so no
    variable expansion occurs. However, you could do it yourself:

    >>> p = subprocess.Popen(['echo', os.environ['MYVAR']])
    >>> myval

    p.wait()
    0
    >>>


    >>>> p = subprocess.Popen('echo $MYVAR',shell=True)
    >>>> myval


    (here /bin/sh does the expansion, because you invoked it)

    >>>> p = subprocess.Popen('echo $MYVAR')

    >Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    > errread, errwrite)
    > File "/usr/lib64/python2.6/subprocess.py", line 1106, in
    >_execute_child
    > raise child_exception
    >OSError: [Errno 2] No such file or directory


    This attempted to run the executable named 'echo $MYVAR'. It did
    not exist so the underlying exec (after the fork) failed. The
    exception was passed back to the subprocess module, which raised
    it in the parent for you to see.

    If you were to create an executable named 'echo $MYVAR' (including
    the blank and dollar sign) somewhere in your path (or use an explicit
    path to it), it would run. I will also capture the actual output
    this time:

    $ cat '/tmp/echo $MYVAR'
    #! /usr/bin/awk NR>1{print}
    this is a self-printing file
    anything after the first line has NR > 1, so gets printed
    $ chmod +x '/tmp/echo $MYVAR'
    $ python
    Python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12)
    [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import subprocess
    >>> p = subprocess.Popen('/tmp/echo $MYVAR', stdout=subprocess.PIPE)
    >>> print p.communicate()[0]

    this is a self-printing file
    anything after the first line has NR > 1, so gets printed

    >>> p.returncode

    0
    >>>


    Incidentally, fun with #!: you can make self-renaming scripts:

    sh-3.2$ echo '#! /bin/mv' > /tmp/selfmove; chmod +x /tmp/selfmove
    sh-3.2$ ls /tmp/*move*
    /tmp/selfmove
    sh-3.2$ /tmp/selfmove /tmp/I_moved
    sh-3.2$ ls /tmp/*move*
    /tmp/I_moved
    sh-3.2$

    or even self-removing scripts:

    sh-3.2$ echo '#! /bin/rm' > /tmp/rmme; chmod +x /tmp/rmme
    sh-3.2$ /tmp/rmme
    sh-3.2$ /tmp/rmme
    sh: /tmp/rmme: No such file or directory

    (nothing to do with python, just the way #! interpreter lines work).
    --
    In-Real-Life: Chris Torek, Wind River Systems
    Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
    email: gmail (figure it out) http://web.torek.net/torek/index.html
     
    Chris Torek, Nov 5, 2010
    #4
  5. moogyd

    moogyd Guest

    Hi,

    Thanks everyone for the replies - it is now clearer.

    Steven
     
    moogyd, Nov 7, 2010
    #5
  6. In message <>, Chris Torek wrote:

    > ['/bin/sh', '-c', 'echo', '$MYVAR']
    >
    > (with arguments expressed as a Python list). /bin/sh takes the
    > string after '-c' as a command, and the remaining argument(s) if
    > any are assigned to positional parameters ($0, $1, etc).


    Doesn’t work. I don’t know what happens to the extra arguments, but they
    just seem to be ignored if -c is specified.

    sh -c 'echo hi'

    echoes “hiâ€, while

    sh -c echo hi

    just outputs a blank line.
     
    Lawrence D'Oliveiro, Nov 8, 2010
    #6
  7. moogyd

    Mark Wooding Guest

    Lawrence D'Oliveiro <_zealand> writes:

    > In message <>, Chris Torek wrote:
    >
    > > ['/bin/sh', '-c', 'echo', '$MYVAR']
    > >
    > > (with arguments expressed as a Python list). /bin/sh takes the
    > > string after '-c' as a command, and the remaining argument(s) if
    > > any are assigned to positional parameters ($0, $1, etc).

    >
    > Doesn’t work.


    What doesn't work? You were being given an explanation, not a solution.

    > I don’t know what happens to the extra arguments, but they just seem
    > to be ignored if -c is specified.


    The argument to -c is taken as a shell script; the remaining arguments
    are made available as positional parameters to the script as usual (only
    starting with $0 rather than $1, for some unknown reason).

    -- [mdw]
     
    Mark Wooding, Nov 8, 2010
    #7
  8. moogyd

    Ian Guest

    On Nov 8, 2:43 am, (Mark Wooding) wrote:
    > > I don’t know what happens to the extra arguments, but they just seem
    > > to be ignored if -c is specified.

    >
    > The argument to -c is taken as a shell script; the remaining arguments
    > are made available as positional parameters to the script as usual (only
    > starting with $0 rather than $1, for some unknown reason).


    Perhaps this example better demonstrates what is going on:

    >>> p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],

    .... shell=True)
    one two three four five

    Cheers,
    Ian
     
    Ian, Nov 8, 2010
    #8
  9. moogyd

    Hans Mulder Guest

    Ian wrote:
    > On Nov 8, 2:43 am, (Mark Wooding) wrote:
    >>> I don’t know what happens to the extra arguments, but they just seem
    >>> to be ignored if -c is specified.

    >> The argument to -c is taken as a shell script; the remaining arguments
    >> are made available as positional parameters to the script as usual (only
    >> starting with $0 rather than $1, for some unknown reason).

    >
    > Perhaps this example better demonstrates what is going on:
    >
    >>>> p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],

    > ... shell=True)
    > one two three four five


    Maybe I'm thick, but I still don't understand. If I run a shell script,
    then the name of the script is in $0 and the first positional arguments
    is in $1, similar to how Python sets up sys.argv.

    But in this case the first positional argument is in $0. Why is that?

    Puzzled,

    -- HansM
     
    Hans Mulder, Nov 8, 2010
    #9
  10. In message <4cd87b24$0$81481$4all.nl>, Hans Mulder wrote:

    > But in this case the first positional argument is in $0.


    That’s what confused me.
     
    Lawrence D'Oliveiro, Nov 8, 2010
    #10
  11. moogyd

    Ian Guest

    On Nov 8, 3:35 pm, Hans Mulder <> wrote:
    > > Perhaps this example better demonstrates what is going on:

    >
    > >>>> p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],

    > > ...                      shell=True)
    > > one two three four five

    >
    > Maybe I'm thick, but I still don't understand.  If I run a shell script,
    > then the name of the script is in $0 and the first positional arguments
    > is in $1, similar to how Python sets up sys.argv.
    >
    > But in this case the first positional argument is in $0.  Why is that?


    It's just a quirk in the way the shell handles the -c option. The
    syntax for the shell invocation boils down to something like this:

    sh [-c command_string] command_name arg1 arg2 arg3 ...

    Without the -c option, sh runs the file indicated by command_name,
    setting $0 to command_name, $1 to arg1, $2 to arg2, etc.

    With the -c option, it does the same thing; it just runs the
    command_string instead of a file pointed to by command_name. The
    latter still conceptually exists as an argument, however, which is why
    it still gets stored in $0 instead of $1.

    We could argue about whether this approach is correct or not, but it's
    what the shell does, and that's not likely to change.

    Cheers,
    Ian
     
    Ian, Nov 9, 2010
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jim Benson

    simple popen question

    Jim Benson, Apr 2, 2004, in forum: Python
    Replies:
    1
    Views:
    306
    Cameron Laird
    Apr 3, 2004
  2. Jim Benson

    Re: simple popen question

    Jim Benson, Apr 3, 2004, in forum: Python
    Replies:
    1
    Views:
    308
    Cameron Laird
    Apr 3, 2004
  3. Robin Siebler
    Replies:
    6
    Views:
    540
  4. Kevin Walzer

    Question about pipes/os.popen

    Kevin Walzer, Sep 15, 2006, in forum: Python
    Replies:
    4
    Views:
    1,458
    Donn Cave
    Sep 15, 2006
  5. File.popen/IO.popen

    , May 20, 2006, in forum: Ruby
    Replies:
    1
    Views:
    260
    Robert Klemme
    May 20, 2006
Loading...

Share This Page