Direct interaction with subprocess - the curse of blocking I/O

[Pascal Chambon]

Hello everyone

I've had real issues with subprocesses recently : from a python script,
on windows, I wanted to "give control" to a command line utility, i.e
forward user in put to it and display its output on console. It seems
simple, but I ran into walls :
- subprocess.communicate() only deals with a forecast input, not
step-by-step user interaction
- pexpect module is unix-only, and for automation, not interactive input
- when wanting to do all the job manually (transfering data between the
standard streams of the python program and the binary subprocess, I met
the issue : select() works only on windows, and python's I/O are
blocking, so I can't just, for example, get data from the subprocess'
stdout and expect the function to return if no input is present - the
requesting thread might instead block forever.

Browsing the web, I found some hints :
- use the advanced win32 api to create non-blocking I/O : rather
complicated, non portable and far from the python normal files
- use threads that block on the different streams and eat/feed them
without ever stopping : rather portable, but gives problems on shutdown
(How to terminate these threads without danger ? On some OSes, a process
never dies as long as any thread - even "daemonic" - lives, I've seen
people complaining about it).

So well, I'd like to know, do you people know any solution to this
simple problem - making a user interact directly with a subprocess ? Or
would this really require a library handling each case separately (win32
api, select().....) ?

Thanks a lot for your interest and advice,
regards,
Pascal

 

[cgoldberg]

So well, I'd like to know, do you people know any solution to this

simple problem - making a user interact directly with a subprocess?

you might want something like Expect.

check out the pexpect module:
http://pexpect.sourceforge.net/pexpect.html

-Corey

 

[Nobody]

I've had real issues with subprocesses recently : from a python script,
on windows, I wanted to "give control" to a command line utility, i.e
forward user in put to it and display its output on console.

Are you talking about a popen(..., 'w') situation? I.e. where Python
feeds data to the child's stdin but the child's stdout doesn't go through
Python?

Or a slave process, where both stdin and stdout/stderr are piped to/from
Python?

The latter is inherently tricky (which is why C's popen() lets you connect
to stdin or stdout but not both). You have to use either multiple threads,
select/poll, or non-blocking I/O.

If the child's output is to the console, it should presumably be the
former, i.e. piping stdin but allowing the child to inherit stdout, in
which case, where's the problem? Or are you piping its stdout via Python
for the hell of it?

 

[Lawrence D'Oliveiro]

I met the issue : select() works only on windows ...

No it doesn't. It works only on sockets on Windows, on Unix/Linux it works
with all file descriptors <http://docs.python.org/library/select.html>.

 

[ryles]

 and I personally wouldn't have it any other way. Simulating a shell
with hooks on its I/O should be so complicated that a "script kiddie"
has trouble writing a Trojan.

+1 QOTW

 

[Lawrence D'Oliveiro]

In message <b1d86eff-

+1 QOTW

Trouble is, script kiddies, by definition, don't need to write anything.
They just need to download something somebody else has already written.

 

[spillz]

Hello everyone

I've had real issues with subprocesses recently : from a python script,
on windows, I wanted to "give control" to a command line utility, i.e
forward user in put to it and display its output on console. It seems
simple, but I ran into walls :

If you are willing to have a wxPython dependency, wx.Execute handles
non-blocking i/o with processes on all supported platforms

more discussion of python's blocking i/o issues here:
http://groups.google.com/group/comp...18f99630/60886b8beb55cfbc?q=#60886b8beb55cfbc

 

[yam850]

If you are willing to have a wxPython dependency, wx.Execute handles
non-blockingi/o with processes on all supported platforms

I made a python method/function for non blocking read from a file
object.
I use it in one of my python programs.
When looking at the code bear in mind that I am not an expert and I am
happy to see comments.

#------------------------------------------------------------------
def non_blocking_readline(f_read=sys.stdin, timeout_select=0.0):
"""to readline non blocking from the file object 'f_read'
for 'timeout_select' see module 'select'"""
import select
text_lines = '' # empty string
while True: # as long as there are bytes
to read
try: # try select
rlist, wlist, xlist = select.select([f_read], [], [],
timeout_select)
except: # select ERROR
print >>sys.stderr, ("non_blocking_read select ERROR")
break
if DEBUG: print("rlist=%s, wlist=%s, xlist=%s" % (repr(rlist),
repr(wlist), repr(xlist)))
if len(rlist) > 0:
text_read = f_read.readline() # get a line
if DEBUG: print("after read/readline text_read:'%s', len=
%s" % (text_read, repr(len(text_read))))
if len(text_read) > 0: # there were some bytes
text_lines = "".join([text_lines, text_read])
if DEBUG: print("text_lines:'%s'" % (text_lines))
else:
break # there was no byte in a
line
else:
break # there was no byte in the
f_read
if text_lines == '':
return None
else:
return text_lines

 

[yam850]

I made a python method/function for nonblockingread from a file
object.... I am happy to see comments.

OK, here's a fairly careful set of comments with a few caveats: [snip] valuable comments
--Scott David Daniels
(e-mail address removed)

Wow, thats a GREAT answer!!!
Thanks!!!
I will learn a lot while "digesting" this mail.


G