Re: Correct design for parallel opening of sockets to remote hosts?

Discussion in 'Perl Misc' started by Ben Bacarisse, May 19, 2014.

  1. Henry Law <> writes:

    > I have a Perl program design problem that I'm sure has been overcome
    > before; it's just that with my level of ability with the language I
    > can't see how to solve it.
    >
    > My program will check the communications with (at least ten) hosts
    > elsewhere on the network. I can try to open a socket (using
    > IO::Socket::INET) to each host in turn, using a port which I know is
    > open (they're my hosts), and wait to see if it times out. But each
    > timeout will take at least one second, so all twelve machines will
    > take at least twelve seconds: that's too long for the user to wait.
    >
    > So I need to fork copies of the program, one for each host, and let
    > them time out in parallel; my design problem is that I can't work out
    > how to do that. Here's my latest unsuccessful attempt, based on the
    > example under heading "Safe Pipe Opens" in perlipc:
    >
    > #! /usr/bin/perl
    > use strict;
    > use warnings;
    >
    > my $hosts = [
    > { name=>'beta' }, { name=>'saturn', port=>80 }
    > ]; # Sample host data, in hashes since I need
    > # to add other data here later
    >
    > for my $host ( @$hosts ) {
    > my $child_pid;
    > my $retries = 0;
    > do {
    > $child_pid = open( $host->{fh}, "-|" );
    > unless ( defined $child_pid ) {
    > die "Giving up on $host->{name}" if ++$retries>4;
    > sleep 1;
    > }
    > } until defined $child_pid;
    >
    > if ( $child_pid ) { # Parent
    > my $fh = $host->{fh};
    > while ( my $data = <$fh> ) { # <$host->{fh}> is syntax error
    > print "[Parent] Read '$data'\n";
    > }
    > close $host->{fh} ||
    > warn "Child process for $host->{name} has exited: $?\n";
    > }
    > else { # Child
    > my $delay = int(rand(10));
    > warn "[child] Simulating $delay seconds delay in contacting host
    > $host->{name}.\n";
    > sleep $delay;
    > print STDOUT "data for $host->{name}"; # Simulating the data returned
    > warn "Child for $host->{name} finishing\n";
    > exit;
    > }
    > }
    >
    > But the parent process still waits for each host in turn:
    >
    > ./checkhosts
    > [child] Simulating 3 seconds delay in contacting host beta.
    > Child for beta finishing
    > [Parent] Read 'data for beta'
    > [child] Simulating 1 seconds delay in contacting host saturn.
    > Child for saturn finishing
    > [Parent] Read 'data for saturn'
    >
    > My head is spinning. I wake up thinking about fork statements and
    > zombie processes ... There's an easy answer to this; can someone help
    > me find it?


    You have one loop. For every host, you fork and then wait -- blocking on
    a read -- for that child to write. It's inherently sequential.

    I am not exactly sure of the best of the best solution. As usual it
    will depend, but for starters, finish this loop before you try to read
    anything. Then, in a second loop (once all the children are running),
    loop though the hosts structure reading from the file handles.

    If instead of looping to read each child's output in turn (which will be
    sub-optimal since you can't be sure to read the data in the order it
    becomes available), you could use select to read data as an when it
    becomes available for any of the child descriptors. See IO::select.

    I am sure I could give you some code if it were not so late!

    --
    Ben.
    Ben Bacarisse, May 19, 2014
    #1
    1. Advertising

  2. Ben Bacarisse <> writes:
    <snip>
    > [...] See IO::select.


    I meant IO::Select of course.

    > I am sure I could give you some code if it were not so late!


    Not too late it seems. Here is a cut-down example based on you loop:

    for my $host (@$hosts) {
    my $fh;
    if (open($fh, "-|")) {
    $h_set->add($fh);
    }
    else {
    my $delay = int(rand(10));
    warn "[child] Simulating $delay seconds delay in contacting host $host->{name}.\n";
    sleep $delay;
    print STDOUT "data for $host->{name}"; # Simulating the data returned
    warn "Child for $host->{name} finishing\n";
    exit;
    }
    }

    while (my @rd = $h_set->can_read) {
    for my $fh (@rd) {
    while (my $data = <$fh>) {
    print "[Parent] Read '$data'\n";
    }
    $h_set->remove($fh);
    close $fh ||
    warn "Child process has exited: $?\n";
    }
    }

    --
    Ben.
    Ben Bacarisse, May 19, 2014
    #2
    1. Advertising

  3. Henry Law <> writes:

    > On 19/05/14 02:53, Ben Bacarisse wrote:
    >> Not too late it seems.

    >
    > You stayed up till 3AM writing code to help me! That's an extra mile
    > and a half.


    Well, I happened to be up *and* a wrote a little code. Not exactly the
    same! Anyway, I'm glad it was helpful.

    <snip>
    --
    Ben.
    Ben Bacarisse, May 19, 2014
    #3
  4. Ben Bacarisse <> writes:
    > Ben Bacarisse <> writes:
    > <snip>
    >> [...] See IO::select.

    >
    > I meant IO::Select of course.
    >
    >> I am sure I could give you some code if it were not so late!

    >
    > Not too late it seems. Here is a cut-down example based on you loop:
    >
    > for my $host (@$hosts) {
    > my $fh;
    > if (open($fh, "-|")) {
    > $h_set->add($fh);
    > }
    > else {
    > my $delay = int(rand(10));
    > warn "[child] Simulating $delay seconds delay in contacting host $host->{name}.\n";
    > sleep $delay;
    > print STDOUT "data for $host->{name}"; # Simulating the data returned
    > warn "Child for $host->{name} finishing\n";
    > exit;
    > }
    > }


    Instead of doing this, it should usually also be possible to switch all
    sockets to non-blocking mode prior to calling connect and then select
    for 'writable'. This means a so-called 'non-blocking connect' will be
    done by the kernel and each socket with some kind of 'final result'
    available (connection established or an error occured while trying) will
    become writable.

    Another, possibly simpler idea would be to connect all 'connecting
    processes' to the same pipe and let them write the name/ address of the
    host they're connecting to to that upon success. The supervisor process
    can then just do a blocking read on the pipe, dealing with the results
    as they arrive. Once the last 'connecting process' has exited, trying to
    read from the pipe will return 'an EOF' (aka 'read of size 0').

    NB: While that's a nice and simple idea, it is probably somewhat beyond
    the abilities of the available 'canned system call wrappers'.
    Rainer Weikusat, May 19, 2014
    #4
  5. Martijn Lievaart <> writes:
    > On Mon, 19 May 2014 01:55:31 +0100, Ben Bacarisse wrote:


    [...]

    >> If instead of looping to read each child's output in turn (which will be
    >> sub-optimal since you can't be sure to read the data in the order it
    >> becomes available),

    >
    > If you need all the results, it is not sub optimal, but almost optimal
    > and much simpler to program.
    >
    > And using Parallel::ForkManager makes it almost trivial.


    'Almost optimal' is nothing but 'sub optimal'. Even leaving other issues
    aside (eg - unsurprisingly - ForkManager also doesn't exit correctly
    from child processes), this module is intended to be used in a situation
    where a large number of parallellizable jobs are supposed to be executed
    by a limited number of 'worker processes'. That's not applicable here.
    Rainer Weikusat, May 19, 2014
    #5
  6. Martijn Lievaart <> writes:

    > On Mon, 19 May 2014 01:55:31 +0100, Ben Bacarisse wrote:
    >
    >>>
    >>> My head is spinning. I wake up thinking about fork statements and
    >>> zombie processes ... There's an easy answer to this; can someone help
    >>> me find it?

    >
    > ...
    >
    >> If instead of looping to read each child's output in turn (which will be
    >> sub-optimal since you can't be sure to read the data in the order it
    >> becomes available),

    >
    > If you need all the results, it is not sub optimal, but almost optimal
    > and much simpler to program.


    Yes, there won't be much in it. I wonder what the maths says? It might
    make a neat theoretical problem...

    <snip>
    --
    Ben.
    Ben Bacarisse, May 19, 2014
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Bla
    Replies:
    0
    Views:
    2,848
  2. KDawg44
    Replies:
    1
    Views:
    281
  3. Soren
    Replies:
    4
    Views:
    1,241
    c d saunter
    Feb 14, 2008
  4. dan baker

    backup utility for remote hosts

    dan baker, Oct 20, 2004, in forum: Perl Misc
    Replies:
    6
    Views:
    125
    botfood
    Jan 6, 2005
  5. Bla
    Replies:
    1
    Views:
    199
    Tad McClellan
    Apr 10, 2005
Loading...

Share This Page