Output from subprocess.Popen()

[Clodoaldo Pinto Neto]

Output from the shell:

[cpn@s0 teste]$ set | grep IFS
IFS=$' \t\n'

Output from subprocess.Popen():

import subprocess as sub
p = sub.Popen('set | grep IFS', shell=True, stdout=sub.PIPE)
p.stdout.readlines()[1]

"IFS=' \t\n"

Both outputs for comparison:
IFS=$' \t\n'
"IFS=' \t\n"

The subprocess.Popen() output is missing the $ and the last '

How to get the raw shell output from subprocess.Popen()?

Regards, Clodoaldo Pinto Neto

 

[Fredrik Lundh]

Output from the shell:

[cpn@s0 teste]$ set | grep IFS
IFS=$' \t\n'

Output from subprocess.Popen():

import subprocess as sub
p = sub.Popen('set | grep IFS', shell=True, stdout=sub.PIPE)
p.stdout.readlines()[1]

"IFS=' \t\n"

Both outputs for comparison:stdout=subprocess.PIPE)
IFS=$' \t\n'
"IFS=' \t\n"

The subprocess.Popen() output is missing the $ and the last '

if you expect one line of output, why are you doing readlines()[1] ?

>>> f = subprocess.Popen("set | grep IFS", shell=True,
>>> f.stdout.readlines()[1]

Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range

this works for me:
["IFS=$' \\t\\n'\n"]

what does the above return on your machine?

</F>

 

[Clodoaldo Pinto Neto]

this works for me:
["IFS=$' \\t\\n'\n"]

what does the above return on your machine?

["BASH_EXECUTION_STRING='set | grep IFS'\n", "IFS=' \t\n"]

I'm on FC5

 

[Sion Arrowsmith]

this works for me:

f = subprocess.Popen("set | grep IFS", shell=True, stdout=subprocess.PIPE)
f.stdout.readlines()

["IFS=$' \\t\\n'\n"]

what does the above return on your machine?

f = subprocess.Popen("set | grep IFS", shell=True, stdout=subprocess.PIPE)
f.stdout.readlines()

["BASH_EXECUTION_STRING='set | grep IFS'\n", "IFS=' \t\n"]

I'm on FC5

OK, there's something going on here:
$ set | grep IFS
IFS=$' \t\n'
$ python
Python 2.4.1 (#2, May 5 2005, 11:32:06)
[GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import subprocess
f = subprocess.Popen("set | grep IFS", shell=True, stdout=subprocess.PIPE)
f.stdout.readlines() ["IFS=' \t\n"]

(Hmph, I don't have any other non-Windows boxes with >=2.4 to hand.)

 

[Clodoaldo Pinto Neto]

Now we have 3 different outputs from 3 people to the command:

From me on FC5:

["BASH_EXECUTION_STRING='set | grep IFS'\n", "IFS=' \t\n"]

From Fredrik Lundh on unknown OS:

["IFS=$' \\t\\n'\n"]

From Sion Arrowsmith on Debian 1:3.3.5-12:

["IFS=' \t\n"]

I found this problem while developing a CGI shell:
http://code.google.com/p/cgpy-shell/

Although the cgi-shell is working quite well, this issue is a show
stopper for that project. I need a stable/reliable output from Popen().
It looks like each bash or sh version outputs things differently, with
different escaping.

It is not a problem if there is only one or more than one line in the
returned list as i will always get the last one. The problem is the
escaping.

 

[sjdevnull]

Output from the shell:

[cpn@s0 teste]$ set | grep IFS
IFS=$' \t\n'

Output from subprocess.Popen():

import subprocess as sub
p = sub.Popen('set | grep IFS', shell=True, stdout=sub.PIPE)
p.stdout.readlines()[1]

"IFS=' \t\n"

Both outputs for comparison:
IFS=$' \t\n'
"IFS=' \t\n"

The subprocess.Popen() output is missing the $ and the last '

How to get the raw shell output from subprocess.Popen()?

You are getting the raw shell output, it's just that subprocess uses
the default shell (/bin/sh) and not your personal shell. Try running
/bin/sh and run the set|grep command.

I can't see any obvious way to ask subprocess to use a shell other than
the default.
If you need a particular shell, you could use a shell script, e.g.
myscript.sh:
#!/bin/bash
set|grep IFS

and call that with shell=False
p = sub.Popen('/path/to/myscript.sh', stdout=sub.PIPE)

 

[Fredrik Lundh]

I can't see any obvious way to ask subprocess to use a shell other than
the default.

-c ?

>>> f = Popen(["/bin/bash", "-c", "set|grep IFS"], stdout=PIPE)
>>> f.stdout.read() "IFS=$' \\t\\n'\n"
>>> f = Popen(["/bin/sh", "-c", "set|grep IFS"], stdout=PIPE)
>>> f.stdout.read()

"IFS=' \t\n"

</F>

 

[Clodoaldo Pinto Neto]

I can't see any obvious way to ask subprocess to use a shell other than
the default.

-c ?

f = Popen(["/bin/bash", "-c", "set|grep IFS"], stdout=PIPE)
f.stdout.read() "IFS=$' \\t\\n'\n"
f = Popen(["/bin/sh", "-c", "set|grep IFS"], stdout=PIPE)
f.stdout.read()

"IFS=' \t\n"

It solves my problem:

f = sub.Popen(['/bin/sh', '-c', 'set|grep IFS'], stdout=sub.PIPE)
f.stdout.read() "BASH_EXECUTION_STRING='set|grep IFS'\nIFS=' \t\n"
f = sub.Popen(['/bin/bash', '-c', 'set|grep IFS'], stdout=sub.PIPE)
f.stdout.read()

"BASH_EXECUTION_STRING='set|grep IFS'\nIFS=$' \\t\\n'\n"

But I still don't understand what is happening. The manual says that
when shell=True the executable argument specifies which shell to use:
"BASH_EXECUTION_STRING='set|grep IFS'\nIFS=' \t\n"

To make my confusion bigger, in Fedora sh is just a link to bash:
$ ll /bin/*sh
-rwxr-xr-x 1 root root 720888 Feb 11 2006 /bin/bash
lrwxrwxrwx 1 root root 4 Aug 28 22:53 /bin/csh -> tcsh
-rwxr-xr-x 1 root root 1175496 Aug 17 13:19 /bin/ksh
lrwxrwxrwx 1 root root 4 Feb 24 2006 /bin/sh -> bash
-rwxr-xr-x 1 root root 349312 Aug 17 15:20 /bin/tcsh
-rwxr-xr-x 1 root root 514668 Feb 12 2006 /bin/zsh

 

[Fredrik Lundh]

But I still don't understand what is happening. The manual says that
when shell=True the executable argument specifies which shell to use:

no, it says that when shell=True, it runs the command "through" the
default shell. that is, it hands it over to the shell for execution,
pretty much as if you'd typed it in yourself.

To make my confusion bigger, in Fedora sh is just a link to bash:

that only means that the programs share the same binary, not that they
necessarily have exactly the same behaviour:

$ man bash

...

INVOCATION

...

If bash is invoked with the name sh, it tries to mimic the
startup behavior of historical versions of sh as closely as
possible, while conforming to the POSIX standard as well.

...

</F>

 

[Clodoaldo Pinto Neto]

no, it says that when shell=True, it runs the command "through" the
default shell. that is, it hands it over to the shell for execution,
pretty much as if you'd typed it in yourself.

"If shell=True, the executable argument specifies which shell to use."

Taken from 6.8.1 v2.4:

"The executable argument specifies the program to execute. It is very
seldom needed: Usually, the program to execute is defined by the args
argument. If shell=True, the executable argument specifies which shell
to use. On Unix, the default shell is /bin/sh. On Windows, the default
shell is specified by the COMSPEC environment variable."