Background subprocess help?

[danshumaker]

Hi,
I'm trying to do something as simple as this:

"sleep 10; mail -s "test" dans < communicate_with_process &"

which executes immediately because it is backgrounded with "&".

or more generically in english:

"do some long process in the background; send me mail when it's done; allow
me to quit the tool that started launched the long process."

I've found several examples that are close but none of the permeatations I
try work. The closest examples I've found start sending signals to
processes and/or callbacks and 50+ lines of python code.

I found the cliUtils module which came close.. but no cigar.

I find it hard to believe that a one liner in bash/tcsh has to end up being
many lines in python. That makes me think I'm missing something.

Surely thousands of people have done this before -- start a background
process and want notification when it's done.

Anybody have any suggestions on modules to try or quick subprocess commands
that work? This came close
http://docs.python.org/library/subprocess.html#replacing-shell-pipeline but
no cigar too.

Thank you so much for any help or suggestions.

-d

 

[Piet van Oostrum]

d> Hi,
d> I'm trying to do something as simple as this:

d> "sleep 10; mail -s "test" dans < communicate_with_process &"

d> which executes immediately because it is backgrounded with "&".

d> or more generically in english:

d> "do some long process in the background; send me mail when it's done; allow
d> me to quit the tool that started launched the long process."

This description does not correspond to the command above. First the
command is confusing because it uses quotes in a strange way. If the
quotes are meant as Python string quotes there is a problem because the
inner quotes are not escaped. If they are meant as shell quotes they
are completely wrong. So let us assume that the outer quotes are
meant at the linguistic level and the inner quotes are shell quotes. Of
course using quotes in such an ambiguous way is confusing. Moreover the
quotes around "test" are superfluous. They are only necessary if there
are spaces or other strange characters (shell metachars) in the subject
argument.

So let us assume that the shell command you want to execute is:
sleep 10; mail -s "test" dans < communicate_with_process &

This will not do the whole thing in the background. It will do the first
part (sleep 10) and wait for it to complete; after that it will do the
mail command in the background. From your description it seems that you
want to do the combo in the background. In the shell this would be:

(sleep 10; mail -s "test" dans < communicate_with_process) &

And communicate_with_process is apparently a file; I suppose it is
written by the preceding command. If it would be the standard output of
the process it would be something like:

(sleep 10 | mail -s "test" dans) &
of course with sleep 10 replaced by something that generates output.

This is also suggested by your mentioning pipeline below.

d> I've found several examples that are close but none of the permeatations I
d> try work. The closest examples I've found start sending signals to
d> processes and/or callbacks and 50+ lines of python code.

d> I found the cliUtils module which came close.. but no cigar.

d> I find it hard to believe that a one liner in bash/tcsh has to end up being
d> many lines in python. That makes me think I'm missing something.

d> Surely thousands of people have done this before -- start a background
d> process and want notification when it's done.

Sequencing programs is the strength of a shell. It is not Pythons main
task, therefore it takes a little more effort for Python.

Running a combo in the background means there must be a process that
waits for the first program to complete and then starts the second one.
In the shell there will be another shell started, because of the () and
that will execute the two commands sequentially. So if you want to do it
in Python the same should be done. Python just doesn't have a convenient
syntax for this like the shell.

d> Anybody have any suggestions on modules to try or quick subprocess commands
d> that work? This came close
d> http://docs.python.org/library/subprocess.html#replacing-shell-pipeline but
d> no cigar too.

The easiest would be to also use the shell to do the sequencing. Why
reinvent the wheel? Like

p = Popen("sleep 10 | mail -s \"test\" dans", shell=True)

As you don't want to wait for the thing to complete, the p is almost
useless.

If you insist on doing it in Python I would recommend a fork to start
the new process. I suppose you are on a Posix system as you use & after
the shell command.

import os
from subprocess import Popen, PIPE

pid = os.fork()
if pid == 0: # child code
p1 = Popen(["sleep", "10"], stdout=PIPE)
p2 = Popen(["mail", "-s", "test", "dans"], stdin=p1.stdout)
os._exit(0)

You could use: os._exit(os.waitpid(p2.pid, 0)[1]) but as you don't wait
for the child the actual exit status code is useless.

On Windows there is no fork(), therefore you would have to spawn a
separate Python program that only does the body of the if.