[Python 2.4/2.5] subprocess module is sorely deficient?

H

Harishankar

Hi,

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

1. Create non-blocking pipes which can be read in a separate thread (I am
currently writing a mencoder GUI in Tkinter and need a full fledged process
handler to control the command line and to display the progress in a
text-box)

2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

Is there any way to use non-blocking Popen objects using subprocess? and 2 -
is there a way to kill the subprocess in a platform independent manner in a
purely Pythonic way? I thought initially that this problem is simple enough,
but over the last couple of days I've been really struggling to find any
answer. I've been through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.

My only solution seems to be to offer the end user the mencoder command line
and make them execute it manually and be done with it but that seems a rather
weak solution.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 
P

Paul Boddie

Is there any way to use non-blocking Popen objects using subprocess? and 2 -
is there a way to kill the subprocess in a platform independent manner in a
purely Pythonic way? I thought initially that this problem is simple enough,
but over the last couple of days I've been really struggling to find any
answer. I've been through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.

If you want some hints about using subprocesses with non-blocking I/O,
you might find some in my jailtools and pprocess projects:

http://www.python.org/pypi/jailtools
http://www.python.org/pypi/pprocess

Although these projects involve things which are not exactly cross-
platform, the communications mechanisms should be portable, perhaps
with a bit of effort (since I don't recall whether the poll library
function is available on Windows, so you might have to use the select
function instead). It can be awkward sustaining non-blocking
communications with processes if they use buffered I/O, and the only
way I could make Python-based subprocesses work in jailtools was to
invoke them with the unbuffered option (-u).
My only solution seems to be to offer the end user the mencoder command line
and make them execute it manually and be done with it but that seems a rather
weak solution.

The subprocess module may be an improvement over the popen2 module and
various os module functions, but it's still rather arcane.

Paul
 
H

Harishankar

If you want some hints about using subprocesses with non-blocking I/O,
you might find some in my jailtools and pprocess projects:

http://www.python.org/pypi/jailtools
http://www.python.org/pypi/pprocess
Thank you. I will take a look at those. Actually I feel a mechanism like this
should be built-in to Python in the future.
Although these projects involve things which are not exactly cross-
platform, the communications mechanisms should be portable, perhaps
with a bit of effort (since I don't recall whether the poll library
function is available on Windows, so you might have to use the select
function instead). It can be awkward sustaining non-blocking
communications with processes if they use buffered I/O, and the only
way I could make Python-based subprocesses work in jailtools was to
invoke them with the unbuffered option (-u).


The subprocess module may be an improvement over the popen2 module and
various os module functions, but it's still rather arcane.

Yes. I am quite sure there must be an elegant solution to the subprocess
handling/management. Problem is I've been at this for three days and I'm
getting quite bleary eyed trying to pore through a lot of documentation ;-)

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 
N

Nicola Musatti

Hi,

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

1. Create non-blocking pipes which can be read in a separate thread (I am
currently writing a mencoder GUI in Tkinter and need a full fledged process
handler to control the command line and to display the progress in a
text-box)

I suggest you check out this: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Cheers,
Nicola Musatti
 
H

Harishankar

There is a recipe in the cookbook

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Which I've used and it works.
Thanks. I found that recipe too. I was hoping I could cook up something
similar without having to use the module win32api, but looks like that's not
the case.
I'm on Linux. Debian. Pexpect would do the job fine too. The only thing is
it's a third party module and so would reduce the portability of my
application. But never mind. I guess I have to make a compromise one way or
the other.
import os
from subprocess import *
from subprocess import mswindows
from time import sleep

if mswindows:
import win32api
else:
import signal

class PopenNB(Popen):
# - see cookbook recipe for rest of stuff
# ...
def kill(self, killpg=False):
"""
Kill the running process
"""
pid = self.pid
if mswindows:
# Kill the process using win32api and pid - ignore errors
try:
PROCESS_TERMINATE = 1
handle = win32api.OpenProcess(PROCESS_TERMINATE, False,
pid) win32api.TerminateProcess(handle, -1)
win32api.CloseHandle(handle)
except pywintypes.error:
pass
else:
# Kill the process by sending the pid / process group a
signal
if killpg:
try:
pgid = os.getpgid(pid)
except OSError:
killpg = False
try:
if killpg:
os.killpg(pgid, signal.SIGTERM)
else:
os.kill(pid, signal.SIGTERM)
except OSError:
return
sleep(1.0)
try:
if killpg:
os.killpg(pgid, signal.SIGKILL)
else:
os.kill(pid, signal.SIGKILL)
except OSError:
return

Thanks for this bit of code. It should probably be adaptible to my needs.

By the way, the win32api seems to be a nonstandard module (i.e. not present in
the main distribution).

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 
T

Tim Golden

Harishankar said:
Thanks. I found that recipe too. I was hoping I could cook up something
similar without having to use the module win32api...

Well if you want to, you can reproduce the same effect by using ctypes
which *is* in the standard library. But why reinvent the wheel?
By the way, the win32api seems to be a nonstandard module (i.e. not present in
the main distribution).

Correct. It's part of the pywin32 extensions, one of many useful packages available
to the discerning Python programmer who doesn't feel in some way bound to
whatever comes bundled with the standard library.

TJG
 
H

Harishankar

Well if you want to, you can reproduce the same effect by using ctypes
which *is* in the standard library. But why reinvent the wheel?

The reason is once again, rightly or wrongly I feel that using non-standard
extensions could make it:

1. Difficult to distribute the application as I am not able to package the
third-party extension with distutils.
2. Difficult to predict its behaviour with future versions of Python.
Correct. It's part of the pywin32 extensions, one of many useful packages
available to the discerning Python programmer who doesn't feel in some way
bound to whatever comes bundled with the standard library.

TJG
I wouldn't feel "bound" if I restricted the program to myself. But if I want
to distribute it (as I intend to) I have to think of others as well.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 
M

Mark Wooding

Nick Craig-Wood said:
Harishankar said:
1. Create non-blocking pipes which can be read in a separate thread
[...]

You are correct on both of those points.

I must be missing something. What's wrong with spawning the subprocess
with subprocess.Popen as usual, passing subprocess.PIPE as stdin/stdout/
whatever, making your end nonblocking with fcntl.fcntl and then using
os.read/os.write in the obvious ways?

-- [mdw]
 
S

sturlamolden

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to

Have you looked at the processing module in cheese shop?
 
C

Christian Heimes

Nick said:
Nothing apart from the fact it doesn't work on windows. The buffering
will cause you grief too. If you want to do this properly under unix
use pexpect not subprocess.

http://www.noah.org/wiki/Pexpect

Proper non blocking IO is an absolute nightmare under Windows in my
experience! It really isn't the Windows way so you are fighting the
system the whole time.

Nick is correct. The subproces tries to work around the issues with
threads. But it's no more than an ugly workaround fir Windows' short
comings on async file IO. It's a shame Windows implements the select()
syscall in wsock32 and limits its usage to sockets.

By the way I'm willing to dedicate some time to help enhancing the
subprocess. Everybody is invited to submit patches and I'll review and
check them into the trunk and py3k ASAP. Any help is appreciated:
enhancements for async IO, doc updates, more examples ...

Christian
Python core developer
 
M

Martin v. Löwis

2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

What's wrong with the .terminate method of the Popen object?

Regards,
Martin
 
M

Mike Hansen

I think the best solution would be to port Pexpect to windows which
wouldn't be that difficult according to my reading of the code. If
only I had more free time!

Sage ( http://www.sagemath.org ) uses pexpect fairly extensively to
interface with all sorts of other systems. We recently received
funding from Microsoft to do a native port of Sage (and all of its
components to Windows. Part of this will most likely be a port of
pexpect to Windows.

--Mike
 
H

Harishankar

Nick is correct. The subproces tries to work around the issues with
threads. But it's no more than an ugly workaround fir Windows' short
comings on async file IO. It's a shame Windows implements the select()
syscall in wsock32 and limits its usage to sockets.

By the way I'm willing to dedicate some time to help enhancing the
subprocess. Everybody is invited to submit patches and I'll review and
check them into the trunk and py3k ASAP. Any help is appreciated:
enhancements for async IO, doc updates, more examples ...

Christian
Python core developer

Thanks a lot to everybody who's been following this discussion. Very
interesting indeed.

I'm currently thinking of working around this problem by actually opening a
new terminal window and running the command from there, thus allowing the
user full control over the process.

Is there any platform independent way to launch a terminal window from a
desktop (Windows, Linux, etc.)?

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 
H

Harishankar

At OSAF we used a slightly modified killableprocess module with a
wrapper to deal with complexities of various redirections in
cross-platform way. I actually blogged about this a week ago so rather
than rehash the issues I'll point you to the article which contains
links to all the pieces we used:

http://www.heikkitoivonen.net/blog/2008/04/16/pythons-ossystem-considered-h
armful/
killableprocess.py looks like a good solution indeed. I actually came across
your website in my searches. I just wanted to be absolutely sure that it
worked because you had mentioned that it has some drawbacks.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 
C

Christian Heimes

Harishankar said:
Is there any platform independent way to launch a terminal window from a
desktop (Windows, Linux, etc.)?

No, there isn't. It usually not possible to create a graphical terminal
window on a remote server.

Christian
 
H

Harishankar

No, there isn't. It usually not possible to create a graphical terminal
window on a remote server.

Christian

Ah, well, since my application is a desktop tool and it requires a GUI I'm
doing something like this:

However, I have to then force the user to use xterm (which is a popular/common
X Terminal)

if (sys.platform.startswith ('win'):
# launch the windows cmd.exe with the command
...
else:
# warn the user that xterm is required and then launch xterm
...

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top