Parallel Parser IPC and IO - Need Direction

[dwknapp]

Here is an outline of what I'm trying to do:
1. Gather a list of data text files.
2. Create say 4 child process' and divi up the file list to the
children.
3. Parent will read the parsed output.
4. Each child will open and parse its files and return the output to
the parent. Each file parsed by the child will output multiple lines
of data.
5. Parent sends the data to an output file.

Here is a dumbed down version of my code:

use IO::Select;
use Symbol;
use IO::File;
use IO::Handle;

my $sel = new IO::Select;
my $parent = 1;
my $num_children = 0;
my @filelist;
my @holdfiles;
my $files_per_proc = sprintf("%d", @filelist/4);

while (@holdfiles = splice(@filelist, 0, $files_per_proc)) {
my ($reader, $writer) = (gensym, gensym);
pipe($reader, $writer) or die "pipe: $!";

my $kid = fork();
if ($kid) {
close $writer;
$sel->add($reader);
$num_children += 1;
}
else {
close $reader;
gen_sub($writer, \@holdfiles, \@opts);
$parent = 0;
last;
}
}

while (1) {
foreach my $client ($sel->can_read()) {
my $line = <$client>;
if (!$line) {
$sel->remove($client);
next;
}
print $line; # STDOUT HERE BUT MAYBE EVENTUALLY A FILEHANDLE
}
if ($sel->count == 0) {
last;
}
}

if ($parent) {
while ($num_children) {
my $dead_child = wait;
$num_children--;
}
}

exit(0);

sub gen_sub {
my ($writer, $ra_holdfiles, $ra_options) = @_;

foreach my $file (@{$ra_holdfiles}) {
die "Error" unless (open(IN, $file));
while (my $line = <IN>) {
LOTS OF PARSING HERE

print $writer "SOME DATA THAT I'VE PARSED\n"; # MANY
TIMES PER FILE
}
close IN;
}
}

This may seem to work ok but the output of the parent is all garbaged
up. I think my problem is entirely to do with buffered data but I'm
really stuck. I'm seeing incomplete output from one child, multiple
child outputs on one line, and then also good lines. I know I can
have the children output their data to files but this is NOT what I
want to do. Any direction, help, or suggestions anyone can offer
would be greatly appreciated.

Thanks, -Dave.

 

[dwknapp]

For anyone else who is trying to do the same... I figured out what was wrong.