Proposing a new module: Parallel::Loops

[Peter Valdemar Mørch]

perldoc perlmodlib suggests posting here before posting on CPAN, so
here goes:

I have a new module that I'd like to upload: Parallel::Loops, and
following is the bulk of the synopsis. Is the Parallel::Loops name
appropriate and does anybody have any comments on it before I post it
on CPAN?
Its repository can be found here (code, complete perldoc, etc.)
http://github.com/pmorch/perl-Parallel-Loops

Synopsis:

use Parallel::Loops;

my $maxProcs = 5;
my $pl = new Parallel::Loops($maxProcs);

my @input = ( 0 .. 9 );

my %output;
$pl->tieOutput( \%output );

$pl->foreach(
\@input,
sub {
# This sub "magically" executed in parallel forked child
# processes

# Lets just create a simple example, but this could be a
# massive calculation that will be parallelized, so that
# $maxProcs different processes are calculating sqrt
# simultaneously for different values of $_ on different CPUs

$output{$_} = sqrt($_);
}
);

 

[Chris Nehren]

perldoc perlmodlib suggests posting here before posting on CPAN, so
here goes:

I find it quaint that some people still follow that guideline. Most
folks just upload.

I have a new module that I'd like to upload: Parallel::Loops, and
following is the bulk of the synopsis. Is the Parallel::Loops name
appropriate and does anybody have any comments on it before I post it
on CPAN?

How does this differ from e.g. Coro or other similar modules?

my $pl = new Parallel::Loops($maxProcs);

Indirect object syntax considered harmful:
http://www.shadowcat.co.uk/blog/matt-s-trout/indirect-but-still-fatal/

my @input = ( 0 .. 9 );

my %output;
$pl->tieOutput( \%output );

Why are you using tie here?

 

[Peter Valdemar Mørch]

How does this differ from e.g. Coro or other similar modules?

It differs from Coro especially because there are several processes
involved in Parallel::Loops. Each of the iterations in the loop run in
each their own process - in parallel. Whereas Coro::Intro has:

only one thread ever has the CPU, and if another thread wants
the CPU, the running thread has to give it up

, the idea behind Parallel::Loops is exactly to make it easy to use
several CPUs in what resembles code for one CPU.

Why are you using tie here?

Hmm... I thought the idea would be more obvious than it apparently
is...

Outside the $pl->foreach() loop, we're running in the parent process.
Inside the $pl->foreach() loop, we're running in a child process. $pl-

tieOutput is actually the raison d'etre of Parallel::Loops. When the

child process has a result, it stores it in %output (which is tied
with Tie::Hash behind the scenes in the child process).

Behind the scenes, when the child process exits, it sends the results
(the keys written to %output) back to the parent process's version/
copy of %output, so that the user of Parallel::Loops doesn't have to
do any inter-process communication.

Perhaps the Synopsis needs to be a bit more clear on these points.

Indirect object syntax considered harmful:http://www.shadowcat.co.uk/blog/matt-s-trout/indirect-but-still-fatal/

OK, thanks, I'll fix that

 

[Peter Valdemar Mørch]

OK; how is this different from forks and forks::shared?

It is _much_ more similar to forks and forks::shared than to Coro.

While the forks and forks::shared API emulate the API of threads and
threads::shared (perfectly?), Parallel::Loops tries to emulate the
standard foreach and while loops as close as possible as in:

$pl->foreach(\@input, sub {
$output{$_} = do_some_hefty_calculation($_);
});

All the forking, waiting for subprocesses to finish etc. is done
behind the scenes. I find that so often, I have large calculations
that need to operate on all the elements of an array or hash, that
really could be parallelized, and with this close-to-foreach syntax,
it is so easy to write and understand/read later on.

I guess Parallel::Loops could have been written with forks and
forks::shared, and only provided syntactic sugar. (In fact it uses
Parallel::ForkManager and Tie::Hash/Tie::Array instead.)

Perhaps $pl->share(\%output) is a better name than $pl->tieOutput(\
%output), tough. I guess now is the time to change it! ;-)

I'm impressed that you guys take the time to read and comment. Thanks!

 

[Ted Zlatanov]

PVM> It is _much_ more similar to forks and forks::shared than to Coro.

PVM> While the forks and forks::shared API emulate the API of threads and
PVM> threads::shared (perfectly?), Parallel::Loops tries to emulate the
PVM> standard foreach and while loops as close as possible as in:

PVM> $pl->foreach(\@input, sub {
PVM> $output{$_} = do_some_hefty_calculation($_);
PVM> });

I like that syntax better personally than join() and detach().

PVM> I guess Parallel::Loops could have been written with forks and
PVM> forks::shared, and only provided syntactic sugar. (In fact it uses
PVM> Parallel::ForkManager and Tie::Hash/Tie::Array instead.)

`forks' brings in socket IPC which can be an issue. Your approach seems
a little cleaner IIUC.

Ted

 

[Peter Valdemar Mørch]

Personally I find

my %output :shared;

for my $i (@input) {
async {
$output{$i} = do_some_hefty_calculation($i);
}
}

somewhat clearer, but that's just a matter of taste. (With 5.10
presumably a 'my $_' would make $_ work too.)

In fact, I think that looks better too. I do have a few concerns:

* Having "my %output : shared" and just async without a
Parallel::Loops reference parameter inevitably leads to global
variables. I don't like them. One could have two different
calculations in different sections of the code, that don't need the
same variables "shared", so I'd prefer to have info about shared
variables associated with a specific Parallel::Loops instance. What do
you think?

* About the async {} instead of $pl->foreach: The implementation needs
to wait for the last loop to finish, and only continue after the '}'
after all the processes have finished. I don't know how to do that
unless something like:

for my $i (@input) {
# This fires up the parallel processes
$pl->async {
$output{$i} = do_some_hefty_calculation($i);
}
}
# This waits for them all to finish before continuing.
$pl->joinAll();

This syntax could easily co-exist with the $pl->foreach and $pl->while
syntax. I'm worried though that people will forget to call $pl-

joinAll()! I guess one could also have async return some reference to

the actual forked process (pid comes to mind) and then $pl->join($pid)
to wait for it to finish.

Regardless, I now think $pl->share(\%output) is a better name than $pl-

tieOutput(\%output)

The rest of this post is about "my %difficulties : with shared;" -
this syntax is how (threads|forks)::shared does it too. I like it, but
don't yet understand how to implement it. Have looked at "perldoc
attributes" and experimented a little. In fact I could get attributes
like "Shared" (==ucfirst("shared")) to work. "xxshared" works, but
issues a warning, but "shared" simply doesn't work (perl 5.10). Also,
I guess it isn't possible for several packages to be "listening" for
attributes at the same time, as they'd step on each other's exports of
e.g. sub MODIFY_SCALAR_ATTRIBUTES, wouldn't they?

Here is a little snippet I wrote to experiment:

me@it:~> cat attributes.pl
#!perl -w
use strict;
use attributes;
use Data:umper;

sub MODIFY_SCALAR_ATTRIBUTES {
my ($pkg, $ref, $attributes) = @_;
print Dumper(\@_, attributes::get($ref));
return ();
}

my $shared : shared;
my $xshared : xshared;
my $Shared : Shared;

me@it:~> perl attributes.pl
$VAR1 = [
'main',
\undef
];
$VAR1 = [
'main',
\undef,
'xshared'
];
SCALAR package attribute may clash with future reserved word: xshared
at attributes.pl line 13
$VAR1 = [
'main',
\undef,
'Shared'
];

 

[Peter Valdemar Mørch]

I like that syntax better personally than join() and detach().

Thanks for the support!

`forks' brings in socket IPC which can be an issue.  Your approach seems
a little cleaner IIUC.

As Ben says, it has to be done somehow. I use a pipe behind the
scenes.

Peter

 

[Peter Valdemar Mørch]

Commenting on Ben's post out of order:

Not like that it can't, since methods don't have prototypes. ....
If you want a method call it would have to look like

$pl->async(sub { ... });

Yes you're right, of course.

Stick it in DESTROY.

I don't see how that would help. I'm thinking of a user writing
something like:

$pl->share(\%results);
foreach (0..4) {
$pl->async(sub { $results{$_} = foobar($_) } );
}
$pl->joinAll();
useResults(\%results);

In this case, at the time of the call to useResults, %results will
contain the finished results from all forked processes because $pl-

joinAll() waits for them all to finish. If $pl->joinAll() doesn't get

called, the user will most likely see an empty %results. I don't see
how DESTROY comes in to play here or could help.

They're not global. %output can be scoped as tightly as you like around
the async call: async takes a closure, so it will make available (either
shared or as copies) any lexicals in scope at the time. (This is why $_
won't work: it isn't a lexical.)

I think I haven't made my concern clear. Is it possible to do:

my %resultsForCalc1 : Shared($pl1);

and have the sharing associated with a particular Parallel::Loops
instance (so my attribute handler gets a reference to $pl1, not the
string '$pl1')?

If so, cool. Don't read any further, I'm satisified (BTW, How?). If
not, lets say one does this:

my %resultsForCalc1 : Shared;
my $pl1 = Parallel::Loops->new(4);
$pl1->foreach([0..9], sub {
$resultsForCalc11{$_} = doSomething($_);
}
useResults(\%resultsForCalc1);

# Block above duplicated, just s/1/2/g
my %resultsForCalc2 : Shared;
my $pl2 = Parallel::Loops->new(4);
$pl1->foreach([0..9], sub {
$resultsForCalc12{$_} = doSomething($_);
}
useResults(\%resultsForCalc1);

Wouldn't the list ( \%resultsForCalc1, \%resultsForCalc2 ) have to be
global? How would I/perl keep track of that the user only wants to
share %resultsForCalc1 in the first calculation and only
%resultsForCalc2 in the second?

By the way, how would one avoid that %foo gets handled as shared in
the following case, since it has gone out of scope?

{
my %foo : Shared;
}
my %resultsForCalc1 : Shared;
my $pl1 = Parallel::Loops->new(4);
$pl1->foreach([0..9], sub {
$resultsForCalc11{$_} = doSomething($_);
}
useResults(\%resultsForCalc1);

I don't (yet?) see how I can detect which of the hashes with the
"Shared" attribute that are in scope at the time of the $pl1-

foreach() call.

But even if I could detect which of all the shared hashes that were in
scope "now", that may not be what the user wants. There could be other
reasons that the user wants %resultsForCalc1 (from way above) in an
outer scope and not have it shared in some of the calculations where
it happens to be in scope.

Perhaps we're getting a little off-topic here, but now I'm curious
about the attributes business! ;-)

Peter

 

[Peter Valdemar Mørch]

I was still looking at the question 'why aren't you simply using
forks?'. forks handles all this for you.

Well, because I don't want the forks API. I want the foreach
syntax. The main reason is that it is so much easier to write and
read later on.

I could've implemented it using forks, but I didn't. Forks _is_
mentioned in the "SEE ALSO" section so users have a chance to explore
alternatives.

When you say 'global' you mean 'shared in all P::L instances', right?
Yes.

Is this a problem?

A little bit. To me, that speaks in favor of

my %output;
$pl->share(\%output)

over

my %output : Shared;

(apart from the fact that $pl->share() seems much simpler to
understand and implement)

(You don't even need to do that if you just weaken the refs in your
master list. Perl will replace any that go out of scope with undef.)

Ah, good point.

I don't know how P::L deals with copying the results back. Presumably
you have no idea whether a variable has been modified in the sub-process
or not? What do you do if two sub-processes change the same shared var
in different ways?

I've mentioned in the pod that only setting of hash keys and pushing
to arrays is supported in the child. I'll append to that that setting
the same key from different iterations preserves a random one of them.

FWIW, I would cast this API rather differently.

Yeah, I'm beginning to gather that! Fine, you won't be one of
P::L's users I take it...

You don't seem to be
trying to emulate the forks API of 'you can do anything you like', but
instead restricting yourself to iterating over a list.
Exactly.

In that case, why not have the API like

my $PL = Parallel::Loops->new(sub { dosomething($_) });
my %results = $PL->foreach(0..9);

I guess if I change that to:

my $PL = Parallel::Loops->new( 4 );
my %results = $PL->foreach( [0..9], sub {
( $_ => dosomething($_) )
});

We could be in business. I'm presuming I can use wantarray() in the
foreach method to test if the caller is going to use the return value
and only transfer the return value from the child if it is going to be
used. It kind of breaks the analogy with foreach but doesn't hurt
otherwise, so why not.

Well, if the user wrote

my %results;
{
my $pl = Parallel::Loops->new;
$pl->share(\%results);
$pl->async(sub { $results{$_} = foobar($_) })
for 0..4;
}
useResults \%results;

then a call to ->joinAll in DESTROY would ensure it was called. Since
variables (particularly those containing potentially-expensive object,
like $pl) should be minimally-scoped, this would be the correct way to
write that code.

I don't understand how that can be guaranteed. perldoc perltoot says:

Perl's notion of the right time to call a destructor is not well-defined
currently, which is why your destructors should not rely on when they
are called.

Given that, how can i be sure that DESTROY has been called at the time
of the useResults call?

Peter

 

[Willem]

Peter Valdemar M?rch wrote:
)> > my %output;
)> > $pl->tieOutput( \%output );
)>
)> Why are you using tie here?
)
) Hmm... I thought the idea would be more obvious than it apparently
) is...
)
) Outside the $pl->foreach() loop, we're running in the parent process.
) Inside the $pl->foreach() loop, we're running in a child process. $pl-
)>tieOutput is actually the raison d'etre of Parallel::Loops. When the
) child process has a result, it stores it in %output (which is tied
) with Tie::Hash behind the scenes in the child process).
)
) Behind the scenes, when the child process exits, it sends the results
) (the keys written to %output) back to the parent process's version/
) copy of %output, so that the user of Parallel::Loops doesn't have to
) do any inter-process communication.

Isn't there some easier method, where you don't have to screw around with
output maps at all ?

If the following API would work, that would be the easiest, IMO:

my @result = async_map { do_something($_) } @array;

Where async_map takes care of all the details of creating the threads,
gathering all the output, et cetera. Or does that already exist ?

(The simple implementation is only a few lines of code, but it could
then be easily extended to use a limited number of threads, or keep
a thread pool handy, or something like that.)


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

 

[Peter Valdemar Mørch]

Isn't there some easier method, where you don't have to screw around with
output maps at all ?

If the following API would work, that would be the easiest, IMO:

  my @result = async_map { do_something($_) } @array;

Where async_map takes care of all the details of creating the threads,
gathering all the output, et cetera.  Or does that already exist ?

It doesn't pre-exist to my knowledge. Not with such a simple syntax.

Ben wrote (in another branch of this tread,
http://groups.google.com/group/comp.lang.perl.misc/msg/0dbec9f2d0e37750
):

It's now more analogous to map than foreach, but I don't see that as a
problem.

Given these two inputs (thank you!), I propose an addition to $pl-

foreach and $pl->while:

my @result = $pl->map(sub { do_something($_) }) @array;

And that will be $pl->map(sub {}) instead of async_map {} so the
object holds the number of processes to use. Alternatively, async_map
would have to be passed the number of processes to use. Which is also
a possibility. Or the number of processes is shared among all
Parallel::Loop async_map calls (which I like less).

(The simple implementation is only a few lines of code, but it could
then be easily extended to use a limited number of threads, or keep
a thread pool handy, or something like that.)

The problem with a thread pool is that then we need to keep all
variables synchronized between them. And I'm focusing on forking - not
threads - here.

But yeah, it isn't that difficult to write. Already, there is more pod
than code! There have just been so many instances already where I
find myself thinking: "This loop could and should be parallelized. But
(I'm too lazy|the schedule is too tight|who cares) right now."

Peter

 

[Ted Zlatanov]

BM> Personally I find

BM> my %output :shared;

BM> for my $i (@input) {
BM> async {
BM> $output{$i} = do_some_hefty_calculation($i);
BM> }
BM> }

BM> somewhat clearer, but that's just a matter of taste. (With 5.10
BM> presumably a 'my $_' would make $_ work too.)

I personally don't like "inline tagged" code blocks as much as passing
them off to a library subroutine. Inline tagging IMO creates spaghetti
code and is harder to refactor. But I can see the appeal


PVM> The problem with a thread pool is that then we need to keep all
PVM> variables synchronized between them. And I'm focusing on forking -
PVM> not threads - here.

Please don't try to make your module do everything for everyone. It's
OK to say "it won't support XYZ." Do a few things well rather than many
things badly.

Ted