Wait for background processes to complete

[pgodfrin]

Greetings,
Well - I've spent a bunch of time trying to figure this out - to no
avail.

Here's what I want to do - run several commands in the background and
have the perl program wait for the commands to complete. Fork doesn't
do it, nor does wait nor waitpid.

Any thoughts?

Here's a sample program which starts the processes:

while (<*.txt>)
{
print "Copying $_ \n";
system("cp $_ $_.old &") ;
}
print "End of excercise\n";
exit;

I mean if this were a shell program - this would work:

for x in `ls *.txt`
do
print "Copying $_ \n"
cp $_ $_.old &
done
wait

thanks,
pg

 

[xhoster]

Greetings,
Well - I've spent a bunch of time trying to figure this out - to no
avail.

Here's what I want to do - run several commands in the background and
have the perl program wait for the commands to complete. Fork doesn't
do it, nor does wait nor waitpid.

None of them individually do it, no. You have to use them together.

Any thoughts?

Here's a sample program which starts the processes:

while (<*.txt>)
{
print "Copying $_ \n";
system("cp $_ $_.old &") ;

This starts a shell, which then starts cp in the background. As soon as
the cp is *started*, the shell exits. So Perl has nothing to wait for, as
the shell is already done (and waited for) before system returns. You need
to use fork and system or fork and exec. Or you could use
Parallel::ForkManager, which will wrap this stuff up nicely for you and
also prevent you from fork-bombing your computer if there are thousands of
*.txt

}
print "End of excercise\n";
exit;

1 until -1==wait(); # on my system, yours may differ


Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Ben Morrow]

Here's what I want to do - run several commands in the background and
have the perl program wait for the commands to complete. Fork doesn't
do it, nor does wait nor waitpid.

Any thoughts?

Here's a sample program which starts the processes:

while (<*.txt>)
{
print "Copying $_ \n";
system("cp $_ $_.old &") ;

This string contains a shell metachar (&), so system will fork a shell
and wait for it. The shell will run cp in the background, and then exit,
at which point system will return. Unfortunately, the only process which
knew cp's pid was the shell, which has just exitted, so you can't wait
for that process at all (cp now has init as its parent, like any other
orphaned process).

You need to either implement the behaviour you want with fork, exec and
waitpid (it's a little complicated, but entirely possible) or use
IPC::Run, something like

use IPC::Run qw/run/;

my @cmds;

while (<*.txt>) {
print "Copying $_\n";
push @cmds, [cp => $_, "$_.old"];
}

run map { ($_, '&') } @cmds;

This is also safer than system STRING in the case where your filenames
have funny characters in them.

}
print "End of excercise\n";
exit;

Falling off the end is a perfectly valid way to end a Perl program. exit
is usually reserved for exceptional circumstances.

I mean if this were a shell program - this would work:

for x in `ls *.txt`
do
print "Copying $_ \n"
cp $_ $_.old &
done
wait

This works because the shell implements '&' directly, rather than using
a different shell, so it can remember the pids to wait for itself.

Ben

 

[pgodfrin]

Quoth pgodfrin <[email protected]>:

Here's what I want to do - run several commands in the background and
have the perl program wait for the commands to complete. Fork doesn't
do it, nor does wait nor waitpid.

Any thoughts?

Here's a sample program which starts the processes:

while (<*.txt>)
{
print "Copying $_ \n";
system("cp $_ $_.old &") ;

This string contains a shell metachar (&), so system will fork a shell
and wait for it. The shell will run cp in the background, and then exit,
at which point system will return. Unfortunately, the only process which
knew cp's pid was the shell, which has just exitted, so you can't wait
for that process at all (cp now has init as its parent, like any other
orphaned process).

You need to either implement the behaviour you want with fork, exec and
waitpid (it's a little complicated, but entirely possible) or use
IPC::Run, something like

use IPC::Run qw/run/;

my @cmds;

while (<*.txt>) {
print "Copying $_\n";
push @cmds, [cp => $_, "$_.old"];
}

run map { ($_, '&') } @cmds;

This is also safer than system STRING in the case where your filenames
have funny characters in them.

}
print "End of excercise\n";
exit;

Falling off the end is a perfectly valid way to end a Perl program. exit
is usually reserved for exceptional circumstances.

I mean if this were a shell program - this would work:

for x in `ls *.txt`
do
print "Copying $_ \n"
cp $_ $_.old &
done
wait

This works because the shell implements '&' directly, rather than using
a different shell, so it can remember the pids to wait for itself.

Ben

OK - would you have a good example of the fork-system-waitpid method -
not the same one that's in all the other posts or the camel book?

smiles,
pg

 

[pgodfrin]

None of them individually do it, no. You have to use them together.






This starts a shell, which then starts cp in the background. As soon as
the cp is *started*, the shell exits. So Perl has nothing to wait for, as
the shell is already done (and waited for) before system returns. You need
to use fork and system or fork and exec. Or you could use
Parallel::ForkManager, which will wrap this stuff up nicely for you and
also prevent you from fork-bombing your computer if there are thousands of
*.txt


1 until -1==wait(); # on my system, yours may differ

Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Well - that's beginning to make a little sense - the shell completes
and perl has nothing to wait for. No wonder I'm pulling out what
little of my hair is left! I guess the fork process returns the pid
of the process, but - if it's the pid of the shell process, then we're
back to square one.

 

[Ben Morrow]

[please trim your quotations]

OK - would you have a good example of the fork-system-waitpid method -
not the same one that's in all the other posts or the camel book?

It's a more efficient use of everybody's time for you to use IPC::Run,
which has been written and tested by someone who understands these
issues and is prepared to solve them properly and portably, than it is
for a random Usenaut to provide you with a code snippet.

However, off the top of my head, completely untested, probably not
portable to Win32 or other non-POSIX systems, etc. etc.,

use POSIX qw/WNOHANG/;

{
my %kids;

$SIG{CHLD} = sub {
my ($pid, @died);
push @died, $pid while $pid = waitpid -1, WNOHANG;
delete @kids{@died};
};

sub background {
my (@cmd) = @_;

defined (my $pid = fork)
or die "can't fork for '$cmd[0]': $!";

if ($pid) {
$kids{$pid} = 1;
return;
}
else {
local $" = "' '";
exec @cmd or die "can't exec '@cmd': $!";
}
}

sub finish {
waitpid $_, 0 for keys %kids;
%kids = ();
}
}

while (<*.txt>) {
print "Copying $_\n";
background cp => $_, "$_.old";
}

finish;

This will break if used in conjunction with system or anything else that
relies on SIGCHLD, and an OO person would probably say it should be
implemented as an object. Use IPC::Run.

Ben

 

[pgodfrin]

None of them individually do it, no. You have to use them together.






This starts a shell, which then starts cp in the background. As soon as
the cp is *started*, the shell exits. So Perl has nothing to wait for, as
the shell is already done (and waited for) before system returns. You need
to use fork and system or fork and exec. Or you could use
Parallel::ForkManager, which will wrap this stuff up nicely for you and
also prevent you from fork-bombing your computer if there are thousands of
*.txt


1 until -1==wait(); # on my system, yours may differ

Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Hi Xho,
Well - It seems you're on to something. Thanks. I have tested this
using system and fork (also exec) and it appears to be exactly what
you're saying - the pid returned by fork is not the pid of the cp
command.

Frankly I'm amazed that something as basic as this, handled easily by
a plain old shell, is not an easy thing in Perl. (no I don't want to
use a shell script for this - I'm a perl bigot ).

I'll take a look at Parallel::ForkManager and any other modules that
might be useful...

cheers,
pg

 

[xhoster]

Well - that's beginning to make a little sense - the shell completes
and perl has nothing to wait for. No wonder I'm pulling out what
little of my hair is left! I guess the fork process returns the pid
of the process, but - if it's the pid of the shell process, then we're
back to square one.

The fork returns (to the parent) the pid of the process forked off.
(but you don't actually need to know the pid if you merely want to wait,
rather than waitpid.) If that forked-off process then itself starts the cp
in the background, of course you are no better off. But if the forked-off
process either becomes cp (using exec) or it starts up cp in the foreground
(using system without a "&"), then you now have something to wait for. In
the first case, you wait for cp itself. In the second case, you wait for
the forked-off perl process which is itself waiting for the cp.

$ perl -wle 'use strict; fork or exec "sleep " . $_*3 foreach 1..3 ; \
my $x; do {$x=wait; print $x} until $x==-1'
438
439
440
-1

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[pgodfrin]

The fork returns (to the parent) the pid of the process forked off.
(but you don't actually need to know the pid if you merely want to wait,
rather than waitpid.) If that forked-off process then itself starts the cp
in the background, of course you are no better off. But if the forked-off
process either becomes cp (using exec) or it starts up cp in the foreground
(using system without a "&"), then you now have something to wait for. In
the first case, you wait for cp itself. In the second case, you wait for
the forked-off perl process which is itself waiting for the cp.

$ perl -wle 'use strict; fork or exec "sleep " . $_*3 foreach 1..3 ; \
my $x; do {$x=wait; print $x} until $x==-1'
438
439
440
-1

Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Hi Xho,
Well - your code and concepts work fine when you want to wait
sequentially. My goal here is to fire off x number of process and then
wait for ALL of them to complete (this is basically rudimentary job
control, trying to use the shell concepts and maximize parallelism).
So, that requires the use of the & to send a process to the
background. It looks like exec doesn't recognize the &, which leaves
the system() command - which leaves us back at square one. The shell
completes and the program has nothing to wait for.

What's interesting is by using the system(" ... &") construct, the
background tasks all get the same PGID, which makes me wonder if
waitpid(0, <flags> ) should wait for wait for process in the process
group of the main perl program...

I wonder...

smiles,
pg

 

[xhoster]

Hi Xho,
Well - your code and concepts work fine when you want to wait
sequentially.

Is there an alternative to waiting sequentially? Waiting sequentially
is what the shell does, too, behind the scenes.

My goal here is to fire off x number of process and then
wait for ALL of them to complete

That is what my code does.

(this is basically rudimentary job
control, trying to use the shell concepts and maximize parallelism).
So, that requires the use of the & to send a process to the
background.

No, that is not required. Since Perl is not a shell, there really
isn't such a thing as the "background" in a Perl context. However,
fork will launch another process which runs without blocking the original
process (well, at least not until the original process asks to be blocked)
or blocking sibling processes. That is what you want, no?

It looks like exec doesn't recognize the &, which leaves
the system() command - which leaves us back at square one. The shell
completes and the program has nothing to wait for.

Yes, so don't do that. "&" basically means "Fork and then don't wait".
Since that isn't what you want to do, then don't do that. Do your own
fork, and then do your own wait.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[comp.llang.perl.moderated]

Greetings,
Well - I've spent a bunch of time trying to figure this out - to no
avail.

Here's what I want to do - run several commands in the background and
have the perl program wait for the commands to complete. Fork doesn't
do it, nor does wait nor waitpid.

Any thoughts?

Here's a sample program which starts the processes:

while (<*.txt>)
{
print "Copying $_ \n";
system("cp $_ $_.old &") ;
}
print "End of excercise\n";
exit;

I mean if this were a shell program - this would work:

for x in `ls *.txt`
do
print "Copying $_ \n"
cp $_ $_.old &
done
wait

A 'quick 'n dirty' take

while (<*.txt>) {
...
system("cp $_ $_.old &") ;
}
my $jobs;
do { sleep 1; $jobs =`ps |grep cp`; }
while $jobs;

 

[pgodfrin]

Is there an alternative to waiting sequentially? Waiting sequentially
is what the shell does, too, behind the scenes.


That is what my code does.


No, that is not required. Since Perl is not a shell, there really
isn't such a thing as the "background" in a Perl context. However,
fork will launch another process which runs without blocking the original
process (well, at least not until the original process asks to be blocked)
or blocking sibling processes. That is what you want, no?


Yes, so don't do that. "&" basically means "Fork and then don't wait".
Since that isn't what you want to do, then don't do that. Do your own
fork, and then do your own wait.

Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Hi Xho,
It seems to me that firing off say 5 processes with the '&' character
to send them to the background is 5 parallel processes, while
executing them off one at a time is sequential. Your code (fork or
exec "sleep" ... ) waits for each sleep process to complete - so that
is what I meant by "waiting sequentially". Technically speaking you're
right - but the idea is to have tasks run in parallel versus
sequentially, which is ostensibly faster.

Insofar as using fork() - your original observation still stands -
system() runs a shell command and then terminates after sending the
command within to run (I watched two different PIDs - the long running
command within the system() statement continued while the shell became
<defunct>)

To make a long story short, this is how I solved the problem. It seems
that the PIDs of the commands run via system() and the '&' background
thing end up belonging to the same Process Group - seen in the ps
command, plus a little extra:

ps -C cp -o pid,pgid,command
PID PGID COMMAND
29068 29063 cp example01.txt example01.txt.old
29070 29063 cp example02.txt example02.txt.old
29072 29063 cp example03.txt example03.txt.old
29074 29063 cp example04.txt example04.txt.old
29076 29063 cp example05.txt example05.txt.old
29078 29063 cp example06.txt example06.txt.old

So I wrap the ps command and do some looping:

for (;
{
open PGRP, "ps -C cp h |" ;
@pidlist=<PGRP> ;
if ($#pidlist<0) {die "\nNo more processes\n" ;}
}

It's not pretty but it works...

But, I believe this is an architectural FLAW with Perl.

regards,
pg

 

[J. Gleixner]

pgodfrin wrote:
[...]

So I wrap the ps command and do some looping:

for (;
{
open PGRP, "ps -C cp h |" ;
@pidlist=<PGRP> ;
if ($#pidlist<0) {die "\nNo more processes\n" ;}
}

It's not pretty but it works...

Why not use Parallel::ForkManager? It IS pretty and will
use much less system resources, compared to running many
'ps' commands.

But, I believe this is an architectural FLAW with Perl.

Everyone else believes it's due to your level of knowledge.

 

[Ben Morrow]

It seems to me that firing off say 5 processes with the '&' character
to send them to the background is 5 parallel processes, while
executing them off one at a time is sequential. Your code (fork or
exec "sleep" ... ) waits for each sleep process to complete - so that
is what I meant by "waiting sequentially".

No, I think you don't understand how fork works. It *is* rather
confusing until you're used to it. Read the documentation again (both
Perl's and your system's), or find a good Unix programming book.

Technically speaking you're
right - but the idea is to have tasks run in parallel versus
sequentially, which is ostensibly faster.

The tasks do run in parallel with Xho's version.

To make a long story short, this is how I solved the problem. It seems
that the PIDs of the commands run via system() and the '&' background
thing end up belonging to the same Process Group - seen in the ps
command, plus a little extra:

So I wrap the ps command and do some looping:

for (;
{
open PGRP, "ps -C cp h |" ;

Use lexical filehandles and three-or-more-arg open.
Check the return value.

open my $PGRP, '-|', 'ps', '-C', 'cp', 'h'
or die "couldn't fork ps: $!";

@pidlist=<PGRP> ;
if ($#pidlist<0) {die "\nNo more processes\n" ;}

IMHO any use of $#ary is an error; certainly in this case you should be
using @pidlist instead.

@pidlist or die "No more processes\n";

This will run around in a tight loop running probably hundreds of ps
processes per second. This is not a effective use of your system's
resources, to say the least. If you must poll like this you need a sleep
in there somewhere to limit the damage.

}

It's not pretty but it works...

Yuck. The whole point of the wait syscall is to avoid nastiness like
that. I suggest you learn how it works, or use a module written by
someone who does (you have been given at least two suggestions so far),
or stick to shell.

But, I believe this is an architectural FLAW with Perl.

No. Firstly, the only flaw here is in your understanding; secondly, if
there was a flaw it would be in Unix, not Perl, since Perl just exposes
the underlying system interfaces.

Ben

 

[nolo contendere]

Hi Xho,
Well - your code and concepts work fine when you want to wait
sequentially. My goal here is to fire off x number of process and then
wait for ALL of them to complete (this is basically rudimentary job
control, trying to use the shell concepts and maximize parallelism).

Reading Xho's code, it looks like 3 processes are kicked off. 1 sleeps
3 seconds, 2 sleeps 6 seconds, and 3 sleeps 9 seconds.
If the processes were sequential, you wouldn't see the last pid until
after 18 seconds. instead you see it after 9 seconds.

HTH

 

[pgodfrin]

Use lexical filehandles and three-or-more-arg open.
Check the return value.

open my $PGRP, '-|', 'ps', '-C', 'cp', 'h'
or die "couldn't fork ps: $!";


IMHO any use of $#ary is an error; certainly in this case you should be
using @pidlist instead.

@pidlist or die "No more processes\n";

This will run around in a tight loop running probably hundreds of ps
processes per second. This is not a effective use of your system's
resources, to say the least. If you must poll like this you need a sleep
in there somewhere to limit the damage.



Yuck. The whole point of the wait syscall is to avoid nastiness like
that. I suggest you learn how it works, or use a module written by
someone who does (you have been given at least two suggestions so far),
or stick to shell.


No. Firstly, the only flaw here is in your understanding; secondly, if
there was a flaw it would be in Unix, not Perl, since Perl just exposes
the underlying system interfaces.

Ben

Whew! I knew I might get some feather's ruffled with the 'flaw'
comment. Sorry my knowledge is not as extensive as yours
(collectively) - but it appears it is true.

I had an exit statement in the if..elsif construct. By removing that
exit and changing the system() to exec() I am at least getting the
fork() process to kick off multiple tasks.

I'm still having problems making it wait though. I think I'm getting
there. I'll report back later - when my knowledge has increased
pg
(sorry Xho...)

 

[pgodfrin]

No, I think you don't understand how fork works. It *is* rather
confusing until you're used to it. Read the documentation again (both
Perl's and your system's), or find a good Unix programming book.


The tasks do run in parallel with Xho's version.







Use lexical filehandles and three-or-more-arg open.
Check the return value.

open my $PGRP, '-|', 'ps', '-C', 'cp', 'h'
or die "couldn't fork ps: $!";


IMHO any use of $#ary is an error; certainly in this case you should be
using @pidlist instead.

@pidlist or die "No more processes\n";

This will run around in a tight loop running probably hundreds of ps
processes per second. This is not a effective use of your system's
resources, to say the least. If you must poll like this you need a sleep
in there somewhere to limit the damage.



Yuck. The whole point of the wait syscall is to avoid nastiness like
that. I suggest you learn how it works, or use a module written by
someone who does (you have been given at least two suggestions so far),
or stick to shell.


No. Firstly, the only flaw here is in your understanding; secondly, if
there was a flaw it would be in Unix, not Perl, since Perl just exposes
the underlying system interfaces.

Ben

darn... still can't make it wait!
pg

 

[pgodfrin]

Hi Ben,
Well - it seems I have made some progress.But - I still need some
advice...

Here's my code:

#!/usr/bin/perl
use POSIX ":sys_wait_h";
$SIG{CHLD} = sub { wait }; # an 'installed' signal handler

$f=0 ;
while (</fausb/sample/*.txt>)
{
$f += 1;
if ($pid = fork)
{
print "Fork $f pid: $pid\n" ;
print "Copying $_ ($pid)\n";
# exec() NEVER returns...
exec("cp $_ $_.old") ;
} elsif (defined $pid)
{
print "Found child...($pid)\n" ;
} elsif ($! =~ /No more process/)
{
print "Fork returned no more processes\n";
} else
{
die "Fork error.\n";
} # end fork
} # end while
print "\n<<<<< End of exercise >>>>>\n";
exit;

So - this code ends up starting 6 forks, and then falls out of the
while(<*.txt>) loop - printing the 'End of exercise' message. A 'ps -
ef |grep fk' (fktest5 is the name of this program) shows it's no
longer running, but the cp command indeed are still running.

What I would like to do is wait for the cp processes to finish before
executing the exit statement.

Any ideas?

thanks,
pg

 

[xhoster]

Hi Ben,
Well - it seems I have made some progress.But - I still need some
advice...

Here's my code:

#!/usr/bin/perl
use POSIX ":sys_wait_h";
$SIG{CHLD} = sub { wait }; # an 'installed' signal handler

You probably don't need a signal handler. And if you want it,
you should aware that that signal can be fired even when there is
no child ready to be waited for, leading to blocking in the wait.
So you would probably want to do a nonblocking waitpid instead.

$f=0 ;
while (</fausb/sample/*.txt>)
{
$f += 1;
if ($pid = fork)
{
print "Fork $f pid: $pid\n" ;
print "Copying $_ ($pid)\n";
# exec() NEVER returns...
exec("cp $_ $_.old") ;
} elsif (defined $pid)
{
print "Found child...($pid)\n" ;
} elsif ($! =~ /No more process/)
{
print "Fork returned no more processes\n";
} else
{
die "Fork error.\n";
} # end fork
} # end while
print "\n<<<<< End of exercise >>>>>\n";

## On Linux, wait returns -1 when there are no living children to wait for.
1 until -1==wait();

exit;

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[pgodfrin]

You probably don't need a signal handler. And if you want it,
you should aware that that signal can be fired even when there is
no child ready to be waited for, leading to blocking in the wait.
So you would probably want to do a nonblocking waitpid instead.






## On Linux, wait returns -1 when there are no living children to wait for.
1 until -1==wait();


Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Thanks Xho - I've removed the signal handler, but it seems wait always
returns -1 so - the loop is a nop? Where in the code should it go?

thanks,
pg