Problem with running external process

[ToMasz]

There is a script running continuously which executes an external
script in a loop using os.spawnlp(os.P_WAIT, 'extscript.py') function.
After several hundred executions (cycles of the loop), the main script
gets an exception while trying to start the process. This situation is
repeated until the main script is killed. Before it is killed, no other
command can be executed - I'm getting the message:

-bash: fork: Resource temporarily unavailable.

How to properly execute another process from my script?

 

[Chmouel Boudjnah]

see the help for ulimit (builtils) :

-$ help limit

 

[Donn Cave]

Quoth "ToMasz" <[email protected]>:
| There is a script running continuously which executes an external
| script in a loop using os.spawnlp(os.P_WAIT, 'extscript.py') function.
| After several hundred executions (cycles of the loop), the main script
| gets an exception while trying to start the process. This situation is
| repeated until the main script is killed. Before it is killed, no other
| command can be executed - I'm getting the message:
|
| -bash: fork: Resource temporarily unavailable.
|
| How to properly execute another process from my script?

Not sure what's causing this. I suggest you invoke "ps" after a
hundred iterations or so - maybe "ps -ef", or "ps wwaux" on BSD
platforms like MacOS. I think the output may show a large number
of processes, and you will know where they came from.

Donn Cave, (e-mail address removed)

 

[ToMasz]

This is the result of ulimit command on my machine:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1014
virtual memory (kbytes, -v) unlimited

It looks like either pipe size or stack size may be too low. But my
main script must be running all the time to collect data so how much
should I increase these values to allow infinite executions of external
process?

 

[Dan Sommers]

... how much should I increase these values to allow infinite
executions of external process?

That sounds like a bad idea. Long before you've run an infinite number
of processes, a real resource will be exhausted.

You are probably creating zombie processes (you can check with /bin/ps).
To get rid of them, execute os.waitpid(-1, os.WNOHANG) (or something
similar to that) periodically from within your python script. Possibly
execute that statement repeatedly until it indicates that there are no
more zombies. A good time/place to do this is right before you start
another process.

Regards,
Dan

 

[ToMasz]

Yes, each time the process is created, it remains.

os.waitpid(-1, os.WNOHANG) doesn't work before starting the process
for the first time.

I tried this:

pid = os.fork()
if pid == 0:
os.execl('ext_script.py','ext_script.py')
else:
(pid,status) = os.waitpid(pid, 0)

It seems to work without leaving zombies.

 

[Dan Sommers]

Yes, each time the process is created, it remains. os.waitpid(-1,
os.WNOHANG) doesn't work before starting the process for the first
time.

Argh. That's what I get for looking at my example too quickly. ;-)

Of course: the best place for os.waitpid is immediately *after*
starting a new process. :-/

I tried this:

pid = os.fork()
if pid == 0:
os.execl('ext_script.py','ext_script.py')
else:
(pid,status) = os.waitpid(pid, 0)

It seems to work without leaving zombies.

If your parent process just sits around and waits for each child process
to finish before moving on, why not use

os.spawnl(os.P_WAIT, 'ext_script.py')

instead of forking and execing "manually"?

Regards,
Dan

 

[ToMasz]

No, no, that wasn't my intention (I'm just not conscious enough what's
going on with these fork, exec, spawn.. functions).
My parent process should start the child process and go back to it's
tasks. Before executing it for the next time the parent should check if
the previous child process is done and start it again.

Is it what these lines do?

os.spawnlp(os.P_NOWAIT,'ext_script.py','')
os.waitpid(-1, os.WNOHANG)

 

[Donn Cave]

No, no, that wasn't my intention (I'm just not conscious enough what's
going on with these fork, exec, spawn.. functions).
My parent process should start the child process and go back to it's
tasks. Before executing it for the next time the parent should check if
the previous child process is done and start it again.

Is it what these lines do?

os.spawnlp(os.P_NOWAIT,'ext_script.py','')
os.waitpid(-1, os.WNOHANG)

No, do you see them doing it? "Check if the previous child
process is done?"

In your original post you said you were using os.P_WAIT,
but I suppose you really were using os.P_NOWAIT all along,
right?

This case may call for a working sample program that makes a
genuine attempt to do what you want to do.

Donn Cave, (e-mail address removed)

 

[Dennis Lee Bieber]

No, no, that wasn't my intention (I'm just not conscious enough what's
going on with these fork, exec, spawn.. functions).
My parent process should start the child process and go back to it's
tasks. Before executing it for the next time the parent should check if
the previous child process is done and start it again.

Is it what these lines do?

os.spawnlp(os.P_NOWAIT,'ext_script.py','')
os.waitpid(-1, os.WNOHANG)

Did you look at the internal help documentation? (I'm on windows and
spawnlp doesn't seem listed)

-=-=-=-=-=-=-=-=-=-Help on built-in function waitpid:

waitpid(...)
waitpid(pid, options) -> (pid, status << 8)

Wait for completion of a given process. options is ignored on
Windows.
Help on function spawnl in module os:

spawnl(mode, file, *args)
spawnl(mode, file, *args) -> integer

Execute file with arguments from args in a subprocess.
If mode == P_NOWAIT return the pid of the process.
If mode == P_WAIT return the process's exit code if it exits
normally;
otherwise return -SIG, where SIG is the signal that killed it.
-=-=-=-=-=-=-=-=-

I don't recall any directive that is the equivalent of "Is
process-PID still running?"; typically one either knows they need to
/wait/ (at some point -- maybe not immediately, hence the P_WAIT or
P_NOWAIT and waitpid() calls), or it is a fire&forget (P_NOWAIT and no
subsequenct waitpid() is used).
--