Parallel Parser IPC and IO - Need Direction

Discussion in 'Perl Misc' started by dwknapp, Apr 9, 2004.

  1. dwknapp

    dwknapp Guest

    Here is an outline of what I'm trying to do:
    1. Gather a list of data text files.
    2. Create say 4 child process' and divi up the file list to the
    children.
    3. Parent will read the parsed output.
    4. Each child will open and parse its files and return the output to
    the parent. Each file parsed by the child will output multiple lines
    of data.
    5. Parent sends the data to an output file.

    Here is a dumbed down version of my code:

    use IO::Select;
    use Symbol;
    use IO::File;
    use IO::Handle;

    my $sel = new IO::Select;
    my $parent = 1;
    my $num_children = 0;
    my @filelist;
    my @holdfiles;
    my $files_per_proc = sprintf("%d", @filelist/4);

    while (@holdfiles = splice(@filelist, 0, $files_per_proc)) {
    my ($reader, $writer) = (gensym, gensym);
    pipe($reader, $writer) or die "pipe: $!";

    my $kid = fork();
    if ($kid) {
    close $writer;
    $sel->add($reader);
    $num_children += 1;
    }
    else {
    close $reader;
    gen_sub($writer, \@holdfiles, \@opts);
    $parent = 0;
    last;
    }
    }

    while (1) {
    foreach my $client ($sel->can_read()) {
    my $line = <$client>;
    if (!$line) {
    $sel->remove($client);
    next;
    }
    print $line; # STDOUT HERE BUT MAYBE EVENTUALLY A FILEHANDLE
    }
    if ($sel->count == 0) {
    last;
    }
    }

    if ($parent) {
    while ($num_children) {
    my $dead_child = wait;
    $num_children--;
    }
    }

    exit(0);

    sub gen_sub {
    my ($writer, $ra_holdfiles, $ra_options) = @_;

    foreach my $file (@{$ra_holdfiles}) {
    die "Error" unless (open(IN, $file));
    while (my $line = <IN>) {
    LOTS OF PARSING HERE

    print $writer "SOME DATA THAT I'VE PARSED\n"; # MANY
    TIMES PER FILE
    }
    close IN;
    }
    }

    This may seem to work ok but the output of the parent is all garbaged
    up. I think my problem is entirely to do with buffered data but I'm
    really stuck. I'm seeing incomplete output from one child, multiple
    child outputs on one line, and then also good lines. I know I can
    have the children output their data to files but this is NOT what I
    want to do. Any direction, help, or suggestions anyone can offer
    would be greatly appreciated.

    Thanks, -Dave.
     
    dwknapp, Apr 9, 2004
    #1
    1. Advertising

  2. dwknapp

    dwknapp Guest

    For anyone else who is trying to do the same... I figured out what was wrong.

    > use IO::Select;
    > use Symbol;
    > use IO::File;
    > use IO::Handle;
    >
    > my $sel = new IO::Select;
    > my $parent = 1;
    > my $num_children = 0;
    > my @filelist;
    > my @holdfiles;
    > my $files_per_proc = sprintf("%d", @filelist/4);
    >
    > while (@holdfiles = splice(@filelist, 0, $files_per_proc)) {
    > my ($reader, $writer) = (gensym, gensym);
    > pipe($reader, $writer) or die "pipe: $!";
    >
    > my $kid = fork();
    > if ($kid) {
    > close $writer;
    > $sel->add($reader);
    > $num_children += 1;
    > }
    > else {
    > close $reader;
    > gen_sub($writer, \@holdfiles, \@opts);
    > $parent = 0;
    > last;
    > }
    > }


    > if ($parent) { # MOVED SO ONLY THE PARENT READS
    >
    > # HERE'S WHERE THE PROBLEM WAS. ONLY THE PARENT SHOULD BE READING. NOW THIS > # SCRIPT WORKS LIKE A CHAMP.
    > while (1) {
    > foreach my $client ($sel->can_read()) {
    > my $line = <$client>;
    > if (!$line) {
    > $sel->remove($client);
    > next;
    > }
    > print $line; # STDOUT HERE BUT MAYBE EVENTUALLY A FILEHANDLE
    > }
    > if ($sel->count == 0) {
    > last;
    > }
    > }
    >
    > while ($num_children) {
    > my $dead_child = wait;
    > $num_children--;
    > }
    > }
    >
    > exit(0);
    >
    > sub gen_sub {
    > my ($writer, $ra_holdfiles, $ra_options) = @_;
    >
    > foreach my $file (@{$ra_holdfiles}) {
    > die "Error" unless (open(IN, $file));
    > while (my $line = <IN>) {
    > LOTS OF PARSING HERE
    >
    > print $writer "SOME DATA THAT I'VE PARSED\n"; # MANY
    > TIMES PER FILE
    > }
    > close IN;
    > }
    > }
    >
     
    dwknapp, Apr 20, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Celia Oblinger

    Need Direction on WHAT to Implement....

    Celia Oblinger, Jan 16, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    306
    Steve C. Orr [MVP, MCSD]
    Jan 16, 2004
  2. Soren
    Replies:
    4
    Views:
    1,270
    c d saunter
    Feb 14, 2008
  3. Vivek Menon
    Replies:
    5
    Views:
    3,361
    Paul Uiterlinden
    Jun 8, 2011
  4. Vivek Menon
    Replies:
    0
    Views:
    1,772
    Vivek Menon
    Jun 10, 2011
  5. Nick da G
    Replies:
    2
    Views:
    148
    Nick da G
    Oct 8, 2008
Loading...

Share This Page