Popen Question


M

moogyd

Hi,
I usually use csh for my simulation control scripts, but these scripts
are becoming more complex, so I plan to use python for the next
project.
To this end, I am looking at subprocess.Popen() to actually call the
simulations, and have a very basic question which is demonstrated
below.

[sde:[email protected] ~]$ python
Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import os, subprocess
os.environ['MYVAR'] = "myval"
p = subprocess.Popen(['echo', '$MYVAR'],shell=True)

p = subprocess.Popen(['echo', '$MYVAR'])
$MYVAR
p = subprocess.Popen('echo $MYVAR',shell=True)
myval
p = subprocess.Popen('echo $MYVAR')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1106, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I am not really sure I understand these results.
1) No idea what is going on
2) As (1). What isn't myval printed out (rather than $MYVAR)
3) Works as I wanted it to
4) Why do I need shell=True ?
The documentation isn't very clear to me (it seems you need to
understand the underlying system calls).

Can anyone explain (or provide link) for this behaviour in simple
English?
Thanks,
Steven
 
Ad

Advertisements

R

Ravi

Hi,
I usually use csh for my simulation control scripts, but these scripts
are becoming more complex, so I plan to use python for the next
project.
To this end, I am looking at subprocess.Popen() to actually call the
simulations, and have a very basic question which is demonstrated
below.

[sde:[email protected] ~]$ python
Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import os, subprocess
os.environ['MYVAR'] = "myval"
p = subprocess.Popen(['echo', '$MYVAR'],shell=True)
p = subprocess.Popen(['echo', '$MYVAR'])
$MYVAR
p = subprocess.Popen('echo $MYVAR',shell=True)
myval
p = subprocess.Popen('echo $MYVAR')

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.6/subprocess.py", line 1106, in
_execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I am not really sure I understand these results.
1) No idea what is going on
2) As (1). What isn't myval printed out (rather than $MYVAR)
3) Works as I wanted it to
4) Why do I need shell=True ?
The documentation isn't very clear to me (it seems you need to
understand the underlying system calls).

Can anyone explain (or provide link) for this behaviour in simple
English?
Thanks,
Steven

try giving /bin/echo
 
A

Alain Ketterlin

moogyd said:
import os, subprocess
os.environ['MYVAR'] = "myval"
p = subprocess.Popen(['echo', '$MYVAR'],shell=True)

p = subprocess.Popen(['echo', '$MYVAR'])
$MYVAR
p = subprocess.Popen('echo $MYVAR',shell=True)
myval
p = subprocess.Popen('echo $MYVAR')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1106, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I am not really sure I understand these results.
1) No idea what is going on
2) As (1). What isn't myval printed out (rather than $MYVAR)
3) Works as I wanted it to
4) Why do I need shell=True ?

Expanding $MYVAR into its value is a feature of the shell (afaik all
shells use the same syntax). Popen without shell=True uses the execvp()
system call directly, without going through the shell variable expansion
process (cases 2 and 4 above). For example, case 4 above asks execvp to
(find and) execute a program named "echo $MYVAR" (an 11-letter name,
where the fifth letter is space and the sixth is $ -- a perfectly valid
file/program name).

Then, if you use shell=True with a list, only the first word is used as
a command, and the others are kept in positional parameters. That's why
your first try fails (try 'sh -c echo $HOME' in a shell, without the
single quotes, and you'll get empty output).
The documentation isn't very clear to me (it seems you need to
understand the underlying system calls).

You're probably right. The base fact here is: the use of variables is a
feature of the shell. No shell, no variable.
Can anyone explain (or provide link) for this behaviour in simple
English?

Shell variables are explained in detail in any shell man page. The
execvp() system call has its own man page.

-- Alain.
 
C

Chris Torek

[sde:[email protected] ~]$ python
Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import os, subprocess
os.environ['MYVAR'] = "myval"
p = subprocess.Popen(['echo', '$MYVAR'],shell=True)

Alain Ketterlin has already explained these to some extent.
Here is a bit more.

This runs, underneath:

['/bin/sh', '-c', 'echo', '$MYVAR']

(with arguments expressed as a Python list). /bin/sh takes the
string after '-c' as a command, and the remaining argument(s) if
any are assigned to positional parameters ($0, $1, etc).

If you replace the command with something a little more explicit,
you can see this:
>>> p = subprocess.Popen( ... [r'echo \$0=$0 \$1=$1', 'arg0', '$MYVAR'], shell=True)
>>> $0=arg0 $1=$MYVAR
p.wait()
0
(I like to call p.communicate() or p.wait(), although p.communicate()
is pretty much a no-op if you have not done any redirecting. Note that
p.communicate() does a p.wait() for you.)
p = subprocess.Popen(['echo', '$MYVAR'])
$MYVAR

This time, as Alain noted, the shell does not get involved so no
variable expansion occurs. However, you could do it yourself:
>>> p = subprocess.Popen(['echo', os.environ['MYVAR']])
>>> myval
p.wait()
0
(here /bin/sh does the expansion, because you invoked it)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1106, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

This attempted to run the executable named 'echo $MYVAR'. It did
not exist so the underlying exec (after the fork) failed. The
exception was passed back to the subprocess module, which raised
it in the parent for you to see.

If you were to create an executable named 'echo $MYVAR' (including
the blank and dollar sign) somewhere in your path (or use an explicit
path to it), it would run. I will also capture the actual output
this time:

$ cat '/tmp/echo $MYVAR'
#! /usr/bin/awk NR>1{print}
this is a self-printing file
anything after the first line has NR > 1, so gets printed
$ chmod +x '/tmp/echo $MYVAR'
$ python
Python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> p = subprocess.Popen('/tmp/echo $MYVAR', stdout=subprocess.PIPE)
>>> print p.communicate()[0]
this is a self-printing file
anything after the first line has NR > 1, so gets printed

Incidentally, fun with #!: you can make self-renaming scripts:

sh-3.2$ echo '#! /bin/mv' > /tmp/selfmove; chmod +x /tmp/selfmove
sh-3.2$ ls /tmp/*move*
/tmp/selfmove
sh-3.2$ /tmp/selfmove /tmp/I_moved
sh-3.2$ ls /tmp/*move*
/tmp/I_moved
sh-3.2$

or even self-removing scripts:

sh-3.2$ echo '#! /bin/rm' > /tmp/rmme; chmod +x /tmp/rmme
sh-3.2$ /tmp/rmme
sh-3.2$ /tmp/rmme
sh: /tmp/rmme: No such file or directory

(nothing to do with python, just the way #! interpreter lines work).
 
L

Lawrence D'Oliveiro

['/bin/sh', '-c', 'echo', '$MYVAR']

(with arguments expressed as a Python list). /bin/sh takes the
string after '-c' as a command, and the remaining argument(s) if
any are assigned to positional parameters ($0, $1, etc).

Doesn’t work. I don’t know what happens to the extra arguments, but they
just seem to be ignored if -c is specified.

sh -c 'echo hi'

echoes “hiâ€, while

sh -c echo hi

just outputs a blank line.
 
Ad

Advertisements

M

Mark Wooding

Lawrence D'Oliveiro said:
['/bin/sh', '-c', 'echo', '$MYVAR']

(with arguments expressed as a Python list). /bin/sh takes the
string after '-c' as a command, and the remaining argument(s) if
any are assigned to positional parameters ($0, $1, etc).

Doesn’t work.

What doesn't work? You were being given an explanation, not a solution.
I don’t know what happens to the extra arguments, but they just seem
to be ignored if -c is specified.

The argument to -c is taken as a shell script; the remaining arguments
are made available as positional parameters to the script as usual (only
starting with $0 rather than $1, for some unknown reason).

-- [mdw]
 
I

Ian

The argument to -c is taken as a shell script; the remaining arguments
are made available as positional parameters to the script as usual (only
starting with $0 rather than $1, for some unknown reason).

Perhaps this example better demonstrates what is going on:
p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],
.... shell=True)
one two three four five

Cheers,
Ian
 
H

Hans Mulder

Ian said:
The argument to -c is taken as a shell script; the remaining arguments
are made available as positional parameters to the script as usual (only
starting with $0 rather than $1, for some unknown reason).

Perhaps this example better demonstrates what is going on:
p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],
... shell=True)
one two three four five

Maybe I'm thick, but I still don't understand. If I run a shell script,
then the name of the script is in $0 and the first positional arguments
is in $1, similar to how Python sets up sys.argv.

But in this case the first positional argument is in $0. Why is that?

Puzzled,

-- HansM
 
Ad

Advertisements

I

Ian

Perhaps this example better demonstrates what is going on:
p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],
...                      shell=True)
one two three four five

Maybe I'm thick, but I still don't understand.  If I run a shell script,
then the name of the script is in $0 and the first positional arguments
is in $1, similar to how Python sets up sys.argv.

But in this case the first positional argument is in $0.  Why is that?

It's just a quirk in the way the shell handles the -c option. The
syntax for the shell invocation boils down to something like this:

sh [-c command_string] command_name arg1 arg2 arg3 ...

Without the -c option, sh runs the file indicated by command_name,
setting $0 to command_name, $1 to arg1, $2 to arg2, etc.

With the -c option, it does the same thing; it just runs the
command_string instead of a file pointed to by command_name. The
latter still conceptually exists as an argument, however, which is why
it still gets stored in $0 instead of $1.

We could argue about whether this approach is correct or not, but it's
what the shell does, and that's not likely to change.

Cheers,
Ian
 
Ad

Advertisements


Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top