os.system vs subprocess

[Nate]

I get different behavior with os.system and subprocess (no surprise
there I guess), but I was hoping for some clarification, namely why.

If I type this directly into the command window:

java -Xms128M -Xmx512M -jar gmapcreator.jar -dfile=censettings.xml >
mapoutput.txt

mapoutput.txt stores the output:
Command line mode: input file=censettings.xml
1358 files will be created in C:\Documents and Settings\Nate\Desktop
\freqanalysis\tilefiles\CENSUS1-tiles
1358 tiles created out of 1358 in 16 seconds

If I execute said command with subprocess, the output is not written
to mapoutput.txt - the output just appears in the command window.

If I execute said command with os.system, the output is written to
mapoutput.txt like I expected.

In reality all I want to do is access the first two lines of the above
output before the process finishes, something which I haven't been
able to manage with subprocess so far. I saw that somehow I might be
able to use os.read(), but this is my first attempt at working with
pipes/processes, so I'm a little overwhelmed.

Thanks!

 

[Chris Rebert]

I get different behavior with os.system and subprocess (no surprise
there I guess), but I was hoping for some clarification, namely why.

If I type this directly into the command window:

java -Xms128M -Xmx512M -jar gmapcreator.jar -dfile=censettings.xml >
mapoutput.txt

mapoutput.txt stores the output:
Command line mode: input file=censettings.xml
1358 files will be created in C:\Documents and Settings\Nate\Desktop
\freqanalysis\tilefiles\CENSUS1-tiles
1358 tiles created out of 1358 in 16 seconds

If I execute said command with subprocess, the output is not written
to mapoutput.txt - the output just appears in the command window.

Show us the subprocess version of you code. People tend to not get the
parameters quite right if they're not familiar with the library.

Cheers,
Chris

 

[Nate]

Show us the subprocess version of you code. People tend to not get the
parameters quite right if they're not familiar with the library.

Cheers,
Chris
--http://blog.rebertia.com- Hide quoted text -

- Show quoted text -

Here it is:

gmapcreator = subprocess.Popen("java -Xms128M -Xmx512M -jar
gmapcreator.jar -dfile=censettings.xml", stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)

 

[Nate]

gmapcreator = subprocess.Popen("java -Xms128M -Xmx512M -jar
gmapcreator.jar -dfile=censettings.xml", stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Try this:

gmapcreator = subprocess.Popen(
    ["java", "-Xms128M", "-Xmx512M", "-jar", "gmapcreator.jar",
    "-dfile=censettings.xml"], stdin=subprocess.PIPE,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)

out, err = gmapcreator.communicate("stdin input")

The subprocess doesn't use the shell so you have to apply the command as
a list of strings.

Do you really need stdin, stdout and stderr as pipes? If you aren't
carefully with the API your program can block.

Christian

Christian,

Thanks for your response. Related to this talk about shells, maybe you
could point me towards a resource where I could read about how windows
commands are processed w/w/o shells? I guess I assumed all subprocess
commands were intepreted by the same thing, cmd.exe., or perhaps the
shell.

I guess this also ties in with me wondering what the whole subprocess
Popen instantiation encompassed (the __init__ of the subprocess.Popen
class?).

I don't think I need stdin, stdout, stderr as pipes, I just was faced
with several options for how to capture these things and I chose
subprocess.PIPE. I could also os.pipe my own pipes, or create a file
descriptor in one of a couple of ways, right? I'm just unclear on
which method is preferable for me. All I want to do is pull the first
2 lines from the output before gmapcreator.jar finishes. The first 2
lines of output are always the same except a few numbers.

 

[Dennis Lee Bieber]

commands are processed w/w/o shells? I guess I assumed all subprocess
commands were intepreted by the same thing, cmd.exe., or perhaps the
shell.

cmd.exe IS the "shell" on Windows (unless running the older
command.com)

which method is preferable for me. All I want to do is pull the first
2 lines from the output before gmapcreator.jar finishes. The first 2
lines of output are always the same except a few numbers.

Which may not be possible -- many programs, when they discover
stdout is not a terminal console, go into a buffering mode such that the
output may not appear until either the buffer is filled, or the program
completes. The assumption being that the output is going to a disk file
or equivalent, and they don't want to interrupt processing by performing
disk I/O on each new-line.

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Lawrence D'Oliveiro]

The subprocess doesn't use the shell ...

It can if you tell it to.

 

[Tim Golden]

Thanks for your response. Related to this talk about shells, maybe you
could point me towards a resource where I could read about how windows
commands are processed w/w/o shells? I guess I assumed all subprocess
commands were intepreted by the same thing, cmd.exe., or perhaps the
shell.

Just a comment here: on Windows, you almost *never* need to specify
shell=True. Certainly, far less than most people seem to think you
need to. The only things which certainly need shell to be set True
are those which don't really exist as programs in their own right:
dir, copy etc. You don't need to set it for batch files (altho' this
does seem to vary slightly between versions) and you certainly don't
need to set it for console programs in general, such as Python.


<code>
import subprocess

subprocess.call (["dir"], shell=False)
subprocess.call (["dir"], shell=True)

with open ("test.bat", "w") as f:
f.write ("echo hello")

subprocess.call (["test.bat"], shell=False)

subprocess.call (["python", "-c", "import sys; print sys.executable"], shell=False)

</code>

This all works as expected using Python 2.6.1 on WinXP SP3

TJG

 

[Nate]

I ended up going with this:

http://code.activestate.com/recipes/440554/

seems to feed me new lines of output atleast before the subprocess
finishes, with some adjustment of the time delays. I'll guess I'll
just be packing winpy into the installer. Or something.