coarse-grained parallel processing of a for-loop

[guba]

Hello,

I am searching for an easy way for a coarse-grained
parallel processing of a for-loop (image processing
with PerlMagick/Win on a QuadCore). Because no
communication is nedded the for-loop

for ($i = $count_min; $i =< $count_max; $i++) {
read selected source images
image composing
write result image on HD
};

could theoretically be splitted in 4 loops

$count_min..int($count_max/4)
int($count_max/4)+1..int($count_max/2)
int($count_max/2)+1..int(3*$count_max/4)
int(3*$count_max/4)+1..$count_max

that are processed in parallel (each on a Core).

cpan offers 300 results for "parallel" but I don't
know where to start. Thank you very much for hints!

Guenter

 

[Peter Makholm]

I am searching for an easy way for a coarse-grained
parallel processing of a for-loop (image processing
with PerlMagick/Win on a QuadCore). Because no
communication is nedded the for-loop

I don't know if Win32 is a problem, but I have used
Parallel::ForkManager for something alike

use Parallel::ForkManager;

my $pm = new Parallel::ForkManager($MAX_PROCESSES);
for $i ($count_min .. $count_max) {
my $pid = $pma->start and next;

read selected source images
image composing
write result image on HD

$pm->finish;
}
$pm->wait_all_children;

But taht gives you no way of communication from within the loop to the
main process.

could theoretically be splitted in 4 loops

$count_min..int($count_max/4)
int($count_max/4)+1..int($count_max/2)
int($count_max/2)+1..int(3*$count_max/4)
int(3*$count_max/4)+1..$count_max

If all jobs are of equal length this would be ok but otherwise you
would be better of with some sort of work queue.

//Makholm

 

[Dr.Ruud]

(e-mail address removed) schreef:

I am searching for an easy way for a coarse-grained
parallel processing of a for-loop (image processing
with PerlMagick/Win on a QuadCore). Because no
communication is nedded the for-loop

for ($i = $count_min; $i =< $count_max; $i++) {
read selected source images
image composing
write result image on HD
};

could theoretically be splitted in 4 loops

$count_min..int($count_max/4)
int($count_max/4)+1..int($count_max/2)
int($count_max/2)+1..int(3*$count_max/4)
int(3*$count_max/4)+1..$count_max

See also the % operator (read `perldoc perlop`).

 

[xhoster]

Hello,

I am searching for an easy way for a coarse-grained
parallel processing of a for-loop (image processing
with PerlMagick/Win on a QuadCore). Because no
communication is nedded the for-loop

for ($i = $count_min; $i =< $count_max; $i++) {
read selected source images
image composing
write result image on HD
};

Parallel::ForkManager might work well for this. It will do one fork
for each item in count_max (but not all at once, of course), so if the work
load for each individual image is very light, then the overhead of the
forking might come to dominate the workload. But I've never tried it on
Windows, so I don't know how it works there.

could theoretically be splitted in 4 loops

$count_min..int($count_max/4)
int($count_max/4)+1..int($count_max/2)
int($count_max/2)+1..int(3*$count_max/4)
int(3*$count_max/4)+1..$count_max

that are processed in parallel (each on a Core).

I frequently do something this like this, too, when each individual item
is light weight. But I don't know of any CPAN modules to facilitate it.

Xho