fork and blocking...

[Tom]

Greetings,

I need a little help with forking, I got the following code off
the web and modified for doing "df" on file systems. I hope to change
this to "rsync" at some point. The problem is I don't want to run any
more that 5 rsyncs at a time. The forking is working fine, however my
blocking isn't... It gets the first 5 off fine but never decrements
the counter I'm using for blocking. Is there a better way than using a
counter?

Thanks in Advanced!
Tom



my @childs;
my @FILESYSTEMS=("Alienbrain_Proxy", "programs", "cae", "codesign",
"eng", "Gold_Build_Patches", "Gold_Builds", "Gold_Patches", "hwdev",
"Media_Shares");

$blockcount = 0;
foreach $item ( @FILESYSTEMS ) {
$blockcount++;
print "$item\n";
while ( $blockcount > 4 ) {
print "$blockcount\n";
sleep 1;
}
my $pid = fork();
if ($pid) {
# parent
# print "pid is $pid, parent $$\n";
push(@childs, $pid);
} elsif ($pid == 0) {
# child
fsize ("$item");
$blockcount--;
exit 0;
} else {
die "Couldn't Fork!: $!\n";
}
}

foreach (@childs) {
my $tmp = waitpid($_, 0);
print "Done with pid $tmp\n";
}

print "End of main program\n";


sub fsize {
my $input = shift;
system ("df /net/server1/$input/$input > /tmp/$input.log");
}

 

[Ted Zlatanov]

T> I need a little help with forking, I got the following code off
T> the web and modified for doing "df" on file systems. I hope to change
T> this to "rsync" at some point. The problem is I don't want to run any
T> more that 5 rsyncs at a time. The forking is working fine, however my
T> blocking isn't... It gets the first 5 off fine but never decrements
T> the counter I'm using for blocking. Is there a better way than using a
T> counter?

Have you looked at Parallel::ForkManager
(http://search.cpan.org/~dlux/Parallel-ForkManager-0.7.5/ForkManager.pm)?

Ted

 

[Jim Gibson]

Greetings,

I need a little help with forking, I got the following code off
the web and modified for doing "df" on file systems. I hope to change
this to "rsync" at some point. The problem is I don't want to run any
more that 5 rsyncs at a time. The forking is working fine, however my
blocking isn't... It gets the first 5 off fine but never decrements
the counter I'm using for blocking. Is there a better way than using a
counter?

Thanks in Advanced!
Tom



my @childs;
my @FILESYSTEMS=("Alienbrain_Proxy", "programs", "cae", "codesign",
"eng", "Gold_Build_Patches", "Gold_Builds", "Gold_Patches", "hwdev",
"Media_Shares");

$blockcount = 0;
foreach $item ( @FILESYSTEMS ) {
$blockcount++;
print "$item\n";
while ( $blockcount > 4 ) {
print "$blockcount\n";
sleep 1;
}

Once you get into this loop, when $blockcount > 4, you never get out.
You are in an infinite loop. Decrementing $blockcount is done by lines
below, but they never get executed. You will need to add a wait
statement here, waiting until one of your child processes has
completed, then decrementing $blockcount and removing the child process
from the array of PIDs.

 

[Jens Thoms Toerring]

... The forking is working fine, however my
blocking isn't... It gets the first 5 off fine but never decrements
the counter I'm using for blocking. Is there a better way than using a
counter?

my @childs;
my @FILESYSTEMS=("Alienbrain_Proxy", "programs", "cae", "codesign",
"eng", "Gold_Build_Patches", "Gold_Builds", "Gold_Patches", "hwdev",
"Media_Shares");

$blockcount = 0;
foreach $item ( @FILESYSTEMS ) {
$blockcount++;
print "$item\n";
while ( $blockcount > 4 ) {
print "$blockcount\n";
sleep 1;
}
my $pid = fork();
if ($pid) {
# parent
# print "pid is $pid, parent $$\n";
push(@childs, $pid);
} elsif ($pid == 0) {
# child
fsize ("$item");
$blockcount--;

Decrementing '$blockcount' is done in the child process,
which is a completely independend process with its own
memory. So the '$blockcount' variabe within this pro-
cess has no connection to that in the parent process.
Thus '$blockcount' gets only incremented in the parent
and never decremented.

One way this could be handled would be to install a signal
handler in the parent process for SIGCHLD signals and de-
crement '$blockcount' in there. I.e. add a line like

$SIG{ CHLD } = sub { $blockcount-- };

before you start the loop. That should do the trick (at least
on UNIX, I have no idea how Windows works in this respect).

And unless you want to avoid mixing the output for childs
created and exited, you could do the waitpid() call directly
in the signal handler

$SIG{ CHLD } = sub { while ( ( my $tmp = waitpid( -1, WNOHANG ) ) > 0 ) {
print "Done with pid $tmp\n";
exit 0 if --$blockcount == 0 ;
}
};

If you do that you need to 'use POSIX;' for the definition
of 'WNOHANG' and you need to replace the loop where you now
wait for all childs to finally exit with something like

pause while 1;

Don't do

pause while $blockcount,

but exit from the signal handler or you risk a so-called
race condition. That's because between the check here if
'$blockcount' is still larger than 0 and the start of the
pause() function a SIGCHLD signal could arrive that results
in '$blockcount' finally becoming 0. In that (not verly
likely but still possible) case the program would hang
forever.
Regards, Jens