Chaining programs with pipe

[avishay]

Hello
I'm trying to chain two programs with a pipe (the output of one
feeding the input of the other). I managed to capture the output and
feeding the input of each program independently with popen, but how do
I tie them together? Is there a solution that works equally on all
platforms?

Thanks,
Avishay

 

[Karthik Gurusamy]

Hello
I'm trying to chain two programs with a pipe (the output of one
feeding the input of the other). I managed to capture the output and
feeding the input of each program independently with popen, but how do
I tie them together? Is there a solution that works equally on all
platforms?

Not sure on non-unix platforms, but in unix like platforms it's best
to reuse shell's power.

 

[Gabriel Genellina]

Not sure on non-unix platforms, but in unix like platforms it's best
to reuse shell's power.

Using the shell is OK but I think the subprocess module is more
portable. This example is on Windows:

import subprocess
subprocess.call("dir /b | sort", shell=True)

~DFDADD.tmp
~DFF71E.tmp
1.bak
1.cmd
1.csv
1.py
1.txt
2.py
2A0B17.dmp [...]

 

[Grant Edwards]

I'm trying to chain two programs with a pipe (the output of
one feeding the input of the other). I managed to capture the
output and feeding the input of each program independently
with popen, but how do I tie them together?

On Unix, you do the same thing you would in C. Create a pipe
using os.pipe(), then run one program with stdout connected to
the "write" end of the pipe and the other program with stdin
connected to the "read" end of the pipe.

Or you can take a bit of a shortcut by letting the subprocess
module create the pipe for you:

http://docs.python.org/lib/node536.html

Is there a solution that works equally on all platforms?

The doc page for subprocess doesn't say what platforms support
it. I've had a lot of problems trying to use the subprocess
module on windows. As is typical for Windows, there are all
sorts of special cases that either don't work at all or don't
work the way they should. You pays your money and you takes
your chances.

 

[Grant Edwards]

Not sure on non-unix platforms, but in unix like platforms it's best
to reuse shell's power.

Executing a shell just because you want a pipe seems like a bit
of overkill. Doing it the "right" way with subprocess is
pretty trivial.

 

[Karthik Gurusamy]

Executing a shell just because you want a pipe seems like a bit
of overkill. Doing it the "right" way with subprocess is
pretty trivial.

Probably I should've put extra stress on the word "reuse".
The example quoted was trivial; if you replace the pipeline to have
say 5 processes, the advantage of not-reinventing the wheel becomes
more obvious.

I would call hand-crafting the pipe-setup an "overkill" when a very
good solution already exists to solve the problem. Yes, it may be
trivial to do; but not simpler than delegating to a shell.

Karthik

 

[Steve Holden]

On Unix, you do the same thing you would in C. Create a pipe
using os.pipe(), then run one program with stdout connected to
the "write" end of the pipe and the other program with stdin
connected to the "read" end of the pipe.

Or you can take a bit of a shortcut by letting the subprocess
module create the pipe for you:

http://docs.python.org/lib/node536.html


The doc page for subprocess doesn't say what platforms support
it. I've had a lot of problems trying to use the subprocess
module on windows. As is typical for Windows, there are all
sorts of special cases that either don't work at all or don't
work the way they should. You pays your money and you takes
your chances.

Grant:

I will shortly have to write some training material on using subprocess
under Windows, so if you have any pointers to where your accumulated
knowledge can be gleaned I would be grateful for the time saving.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[Grant Edwards]

Grant:

I will shortly have to write some training material on using
subprocess under Windows, so if you have any pointers to where
your accumulated knowledge can be gleaned I would be grateful
for the time saving.

There were two main problems I ran into:

1) When you try to execute a program with with a pathname like
\\host\path\to\prog.exe instead of R:\path\to\prog.exe,
cmd.exe chokes and says it can't execute the file. If you
use a UNC path (the "\\" version) you have to use
subprocess's shell=False option. [I did find references to
a patched version of cmd.exe on a few web pages, but didn't
persue that option.] os.system() has the same issue with
UNC path names.

2) In a wxPython app (possibly in non-console apps in
general?), the child process's stdout and/or stderr didn't
default to usable values. When the child tried to write to
them you'd get crashes with rather cryptic error messages.
Explicitly setting the child's stderr and stdout fixes
that. I don't remember os.system() having this problem,
since it only cropped up after I switched to subprocess to
try to solve 1).

In hindsight, these don't seem like big problems, but I tripped
over them at the same time and it took me _days_ to figure out
what was wrong. I'm sure most Win32 programmers already knew
about those problems, but I'm a Unix guy who occasionally tries
to ship a Windows version of a Python app: the concept of a
process defaulting to having a stderr or stdout that wasn't
writable was utterly foreign to me.

 

[Gabriel Genellina]

but I'm a Unix guy who occasionally tries
to ship a Windows version of a Python app: the concept of a

Ah, that explains your previous post telling that things on Windows
don't work as they "should". They work, but not necesarily as a
"foreigner" would expect.

 

[Lawrence D'Oliveiro]

Ah, that explains your previous post telling that things on Windows
don't work as they "should". They work, but not necesarily as a
"foreigner" would expect.

So what's the good reason for Windows having unusable defaults for stderr
and stdout, then?

 

[Gabriel Genellina]

En Mon, 27 Aug 2007 05:35:51 -0300, Lawrence D'Oliveiro

In message <[email protected]>,


So what's the good reason for Windows having unusable defaults for stderr
and stdout, then?

You should ask the wxPython/wxWidgets guys why they choose to do things
that way. Perhaps it's just an option that should be turned on. I'm not a
wx guru. Tk programs don't have that problem, by example: you have a GUI
*and* a console, if you want. A simple print statement with no console
just goes into void space - no error, no crash, no GPF...
You get what you ask for: if you pass /SUBSYSTEM:WINDOWS as an option to
the linker (or put equivalent flags in the executable) you don't get a
console by default. That's fine for most GUI programs that don't use
stdout/stderr. If you want a console, create one using AllocConsole. Or do
that on the parent process, and let the children inherit it (as when
running the application from the command line, as opposed to double
clicking an icon). Or link with /SUBSYSTEM:CONSOLE. Or use
GetStdHandle/SetStdHandle inside the app to redirect STD_OUTPUT_HANDLE etc.

 

[Lawrence D'Oliveiro]

En Mon, 27 Aug 2007 05:35:51 -0300, Lawrence D'Oliveiro


You should ask the wxPython/wxWidgets guys why they choose to do things
that way.

But I assumed you knew, since you were the one who used to term "foreigner"
to describe Grant Edwards' mystification at why things worked this way. So
in fact you are equally a "foreigner" to the way Windows works?

Tk programs don't have that problem, by example: you have a GUI
*and* a console, if you want. A simple print statement with no console
just goes into void space - no error, no crash, no GPF...
You get what you ask for: if you pass /SUBSYSTEM:WINDOWS as an option to
the linker (or put equivalent flags in the executable) you don't get a
console by default. That's fine for most GUI programs that don't use
stdout/stderr. If you want a console, create one using AllocConsole.

But why should I need to do that? On Unix/Linux systems, there is no
distinction between "GUI" and "non-GUI" programs--_all_ processes can (and
usually do) have stdin, stdout and stderr. stderr is particularly important
for GUI programs, so the desktop environment typically captures this to a
file, commonly called ".xsession-errors". This is very useful when things
go wrong (programs crash etc), to see what error messages were reported.

Copying text messages from a file is somewhat easier than trying to capture
them from a screenshot.

 

[Gabriel Genellina]

In message <[email protected]>, Gabriel







But I assumed you knew, since you were the one who used to term "foreigner"
to describe Grant Edwards' mystification at why things worked this way. So
in fact you are equally a "foreigner" to the way Windows works?

I do know how to create console applications, but that is not
relevant. If a wxPython application crashes when something is written
to stdout/stderr, this is either the programmer fault or the framework
fault, *not* Windows fault. As I said, a Tk application doesn't crash
in those circunstances, by example. Many other applications don't
crash either. The default behavior for a GUI application is to simply
discard the output.
(And my comment was not referring to that specific problem - this
thread is too broad now).

But why should I need to do that?

Because the distinction IS relevant on Windows?

On Unix/Linux systems, there is no
distinction between "GUI" and "non-GUI" programs--_all_ processes can (and
usually do) have stdin, stdout and stderr.

On Windows, windowed applications (as opposed to console applications)
start with no stdin/stdout/stderr by default. The application
developer can modify that if desired, of course - I've menctioned some
ways to do that.
This fact simply means that those OS's *are* different - most of the
time one can ignore the differences, but not always. Neither of them
is doing the absolute "Right Thing".

stderr is particularly important
for GUI programs, so the desktop environment typically captures this to a
file, commonly called ".xsession-errors". This is very useful when things
go wrong (programs crash etc), to see what error messages were reported.

There are plenty of logging systems to choose. Redirecting stderr to a
file can be done either by the app itself or by its caller (for a well
behaved application, of course; the crashing wxPython app menctioned
earlier appears not to be one of those).

 

[Grant Edwards]

On Windows, windowed applications (as opposed to console
applications) start with no stdin/stdout/stderr by default.
The application developer can modify that if desired, of
course - I've menctioned some ways to do that. This fact
simply means that those OS's *are* different - most of the
time one can ignore the differences, but not always. Neither
of them is doing the absolute "Right Thing".

We're never going have any good flames with that sort of
attitude.

 

[Gabriel Genellina]

We're never going have any good flames with that sort of
attitude.

Does this help with the right attitude?

"How serious can be a system where you can't say 'Open this file using
my preferred application for it'"?

"A 'stream of bytes' was not a good file metaphor even in the 70's -
why still insist on that? How old-fashioned...! Ever read about ADS?"

"Gay-Lussac's Law is not true in chemistry anymore - why insist on
using small integers for all things? I got a PID from a process. I can
see a running process with that same PID. Are they the same? Go
figure!"

- I got this file name from here. Could you please tell me which
character encoding is it using?
- Character encoding? What's that?
- This name has some characters that aren't ASCII obviously, and I
want to know how to interpret them.
- Sorry, I can't help you. The only thing I can see here are byte
sequences.
- But how should I display them? The user is concerned with
characters, not bytes!
- I'm really sorry, there's nothing I can do.
- Could you please forward my request to the file system then?!?!
- FS has no answer either.
- Fu"$&=)==)("$@2@!