Parallel for loop

[Fredrik]

There doesn't seem to be any EASY way of doing a parallel computation
in Ruby.
I would like to do something like this :

array.map do |i|
fork do
i + 1
end
end
Process.waitall

wich would give back the array with one added to each element in an
array, and it would perform this "calculation" in parallel. However,
this doesn't work since fork runs a subprocess which is another Ruby
interpreter and I can't get anything back from that black hole, except
some exit status.

Actually, it would be really nice if there was a 'forkmap' method that
could do this:

array.forkmap do |i|
i + 1
end

But there isn't, right?

 

[Kyle Hunter]

http://peach.rubyforge.org/?peach

 

[Phillip Gawlowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Fredrik wrote:
| There doesn't seem to be any EASY way of doing a parallel computation
| in Ruby.
| I would like to do something like this :
|
| array.map do |i|
| fork do
| i + 1
| end
| end
| Process.waitall
|
| wich would give back the array with one added to each element in an
| array, and it would perform this "calculation" in parallel. However,
| this doesn't work since fork runs a subprocess which is another Ruby
| interpreter and I can't get anything back from that black hole, except
| some exit status.
|
| Actually, it would be really nice if there was a 'forkmap' method that
| could do this:
|
| array.forkmap do |i|
| i + 1
| end
|
| But there isn't, right?
|
|

Ruby uses green threads. All your threads would run within the Ruby
process, and aren't running parallel in the sense you seem to imply. If
you want to use threads, maybe JRuby and Java are what you seek.

JRuby uses Java threads, which are OS threads.

If I'm on the wrong tangent, just ignore this reply.

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

You thought I was taking your woman away from you. You're jealous.
You tried to kill me with your bare hands. Would a Kelvan do that?
Would he have to? You're reacting with the emotions of a human.
You are human.
~ -- Kirk, "By Any Other Name," stardate 4657.5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgFdsUACgkQbtAgaoJTgL/lnwCfad0hWxWIiJPFyTSdCAkpVIT5
UaQAmwSDyRWArC1bKgnlnYkydZWWHfAN
=6rBM
-----END PGP SIGNATURE-----

 

[ara.t.howard]

But there isn't, right?

cfp:~ > cat a.rb
module Enumerable
def forkify &b
map do |*a|
r, w = IO.pipe
fork do
r.close
w.write( Marshal.dump( b.call(*a) ) )
end
[ w.close, r ].last
end.map{|r| Marshal.load [ r.read, r.close ].first}
end
end


result =
[0, 1, 2, 3].forkify do |i|
p [ Process.ppid, Process.pid ]
i ** 2
end

p result





cfp:~ > ruby a.rb
[80870, 80871]
[80870, 80872]
[80870, 80873]
[80870, 80874]
[0, 1, 4, 9]


a @ http://codeforpeople.com/

 

[Michael Guterl]

There doesn't seem to be any EASY way of doing a parallel computation
in Ruby.
I would like to do something like this :

array.map do |i|
fork do
i + 1
end
end
Process.waitall

wich would give back the array with one added to each element in an
array, and it would perform this "calculation" in parallel. However,
this doesn't work since fork runs a subprocess which is another Ruby
interpreter and I can't get anything back from that black hole, except
some exit status.

Actually, it would be really nice if there was a 'forkmap' method that
could do this:

array.forkmap do |i|
i + 1
end

But there isn't, right?

http://skynet.rubyforge.org/

HTH,
Michael Guterl

 

[Fredrik]

But there isn't, right?

cfp:~ > cat a.rb
module Enumerable
def forkify &b
map do |*a|
r, w = IO.pipe
fork do
r.close
w.write( Marshal.dump( b.call(*a) ) )
end
[ w.close, r ].last
end.map{|r| Marshal.load [ r.read, r.close ].first}
end
end

result =
[0, 1, 2, 3].forkify do |i|
p [ Process.ppid, Process.pid ]
i ** 2
end

p result

cfp:~ > ruby a.rb
[80870, 80871]
[80870, 80872]
[80870, 80873]
[80870, 80874]
[0, 1, 4, 9]

a @http://codeforpeople.com/

Thanks! This code is just what I am looking for!
Peach for JRuby seems nice too, but I don't have JRuby

 

[Phillip Gawlowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Fredrik wrote:

|
| Thanks! This code is just what I am looking for!
| Peach for JRuby seems nice too, but I don't have JRuby

It's just a download away.
jruby.codehouse.org

However, you'll need a JVM that is compatible (IIRC, JRE 1.4.2 and newer).

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

~ "But the important thing is persistence." -Calvin trying to juggle eggs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgFgfsACgkQbtAgaoJTgL+V6wCfUXlPOkq/DXhXQgz/MPLOLWYi
T44An0NLjW0yBzJFEMaGPD//SKcOtGdq
=ODYR
-----END PGP SIGNATURE-----

 

[ara.t.howard]

Thanks! This code is just what I am looking for!

probably some errors there to catch - but the concept is solid
enough. you might also be interested in slave.rb

http://codeforpeople.com/lib/ruby/slave/slave-1.2.1/README

regards.

a @ http://codeforpeople.com/

 

[Fredrik]

Actually, I'll change it a bit. I added Process.waitall since there
are otherwise some dead(?) processes left.


module Enumerable
def fmap &b
result = map do |*a|
r, w = IO.pipe
fork do
r.close
w.write( Marshal.dump( b.call(*a) ) )
end
[ w.close, r ].last
end
Process.waitall
result.map{|r| Marshal.load [ r.read, r.close ].first}
end
end

 

[Charles Oliver Nutter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Fredrik wrote:

|
| Thanks! This code is just what I am looking for!
| Peach for JRuby seems nice too, but I don't have JRuby

It's just a download away.
jruby.codehouse.org

However, you'll need a JVM that is compatible (IIRC, JRE 1.4.2 and newer).

www.jruby.org will get you there, and JRuby 1.1 requires Java 1.5 or higher.

- Charlie

 

[Phillip Gawlowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Charles Oliver Nutter wrote:
| Phillip Gawlowski wrote:
|> -----BEGIN PGP SIGNED MESSAGE-----
|> Hash: SHA1
|>
|> Fredrik wrote:
|>
|> |
|> | Thanks! This code is just what I am looking for!
|> | Peach for JRuby seems nice too, but I don't have JRuby
|>
|> It's just a download away.
|> jruby.codehouse.org
|>
|> However, you'll need a JVM that is compatible (IIRC, JRE 1.4.2 and
|> newer).
|
| www.jruby.org will get you there,

Right, codehaus, not house. *facepalm*

| and JRuby 1.1 requires Java 1.5 or
| higher.

Oops, my mistake (didn't the 1.0 series require only JRE 1.4.2, or so?).

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

~ - You know you've been hacking too long when...
...you see a flock of birds and try to figure out the algorithms that
determine their movement.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgFimEACgkQbtAgaoJTgL9AzwCgpV+uKtcKsXrDUHnydsTzpUzP
X0cAn1cJhepq29B51Fp5f7oHytMyk21m
=KUqL
-----END PGP SIGNATURE-----

 

[ara.t.howard]

Actually, I'll change it a bit. I added Process.waitall since there
are otherwise some dead(?) processes left.

indeed, and you can blow up in the child and not know. read this code
to see how to handle that:

http://codeforpeople.com/lib/ruby/open4/open4-0.9.6/lib/open4.rb

the bit about EOFError

it's damn tricky - it *excepts* get get an exception marshaled up a
dedicated pipe, if this does *not* occur we know the child process
started successfully. you can adapt.

cheers.

a @ http://codeforpeople.com/

 

[Charles Oliver Nutter]

Oops, my mistake (didn't the 1.0 series require only JRE 1.4.2, or so?).

Yes, JRuby 1.0 worked on Java 1.4.2, but there were too many benefits
moving to Java 5 to keep it that way, especially availability of
annotations and the concurrency APIs.

- Charlie

 

[Iñaki Baz Castillo]

MjAwOC80LzE2LCBGcmVkcmlrIDxmcmVkam9oYUBnbWFpbC5jb20+Ogo+IEFjdHVhbGx5LCBJJ2xs
IGNoYW5nZSBpdCBhIGJpdC4gSSBhZGRlZCBQcm9jZXNzLndhaXRhbGwgc2luY2UgdGhlcmUKPiAg
YXJlIG90aGVyd2lzZSBzb21lIGRlYWQoPykgcHJvY2Vzc2VzIGxlZnQuCj4KPgo+ICBtb2R1bGUg
RW51bWVyYWJsZQo+ICAgZGVmIGZtYXAgJmIKPiAgICAgcmVzdWx0ID0gbWFwIGRvIHwqYXwKPgo+
ICAgICAgIHIsIHcgPSBJTy5waXBlCj4gICAgICAgZm9yayBkbwo+ICAgICAgICAgci5jbG9zZQo+
ICAgICAgICAgdy53cml0ZSggTWFyc2hhbC5kdW1wKCBiLmNhbGwoKmEpICkgKQo+ICAgICAgIGVu
ZAo+ICAgICAgIFsgdy5jbG9zZSwgciBdLmxhc3QKPiAgICAgZW5kCj4KPiAgICAgUHJvY2Vzcy53
YWl0YWxsCj4gICAgIHJlc3VsdC5tYXB7fHJ8IE1hcnNoYWwubG9hZCBbIHIucmVhZCwgci5jbG9z
ZSBdLmZpcnN0fQo+ICAgZW5kCj4gIGVuZAoKSXQncyBncmVhdDoKCmlyYj4gQmVuY2htYXJrLnJl
YWx0aW1lIHsgICBbMSwyLDMsNCw1LDYsNyw4LDldLm1hcCB7IHxpfCBzbGVlcCAxOyBpICsxIH0g
ICB9Cj0+IDguOTk2MzY5MTIzNDU4ODYKCmlyYj4gQmVuY2htYXJrLnJlYWx0aW1lIHsgICBbMSwy
LDMsNCw1LDYsNyw4LDldLmZvcmttYXAgeyB8aXwgc2xlZXAgMTsKaSArMSB9ICAgfQo9PiAxLjAy
MzcxMDAxMjQzNTkxCgoKWEQKCgotLSAKScOxYWtpIEJheiBDYXN0aWxsbwo8aWJjQGFsaWF4Lm5l
dD4K

 

[Fredrik]

I added an argument to limit the number of concurrent processes (my
workstation practically died when I ran all the processes I wanted to
run):

module Enumerable
def forkmap n, &b
result = map do |*a|
nproc = 0
r, w = IO.pipe
fork do
r.close
w.write( Marshal.dump( b.call(*a) ) )
end
if (nproc+=1) >= n
Process.wait ; nproc -= 1
end
[ w.close, r ].last
end
Process.waitall
result.map{|r| Marshal.load [ r.read, r.close ].first}
end
end

It seems to be doing its job correctly :

irb> Benchmark.realtime { [1,2,3].forkmap(3){|i| sleep(1) ; i * 2} }
=> 1.01134896278381
irb> Benchmark.realtime { [1,2,3].forkmap(1){|i| sleep(1) ; i * 2} }
=> 3.01262402534485

/Fredrik

 

[Fredrik]

indeed, and you can blow up in the child and not know. read this code
to see how to handle that:

http://codeforpeople.com/lib/ruby/open4/open4-0.9.6/lib/open4.rb

the bit about EOFError

it's damn tricky - it *excepts* get get an exception marshaled up a
dedicated pipe, if this does *not* occur we know the child process
started successfully. you can adapt.

I'm not sure I understand what you mean. But are you saying that open4
can solve all my problems?

 

[Fredrik]

Sorry...posting wrong code. These lines should be flipped:

result = map do |*a|
nproc = 0

should be

nproc = 0
result = map do |*a|

Sorry 'bout that...

/Fredrik

 

[Iñaki Baz Castillo]

El Jueves, 17 de Abril de 2008, Fredrik escribi=F3:

I added an argument to limit the number of concurrent processes (my
workstation practically died when I ran all the processes I wanted to
run):

module Enumerable
def forkmap n, &b
result =3D map do |*a|
nproc =3D 0
r, w =3D IO.pipe
fork do
r.close
w.write( Marshal.dump( b.call(*a) ) )
end
if (nproc+=3D1) >=3D n
Process.wait ; nproc -=3D 1
end
[ w.close, r ].last
end
Process.waitall
result.map{|r| Marshal.load [ r.read, r.close ].first}
end
end

It seems to be doing its job correctly :

irb> Benchmark.realtime { [1,2,3].forkmap(3){|i| sleep(1) ; i * 2} }
=3D> 1.01134896278381
irb> Benchmark.realtime { [1,2,3].forkmap(1){|i| sleep(1) ; i * 2} }
=3D> 3.01262402534485

It's really great. I just see one thing to improve:
The new "n" parameter is mandatory since it's the first parameter. It would=
be=20
nice if it could be not defined (so =3D infinite):

forkmap(4) { code } --> max 4 process
forkmap { code } --> max infinite


Do you think your code can be feasible for production enviroments? maybe it=
=20
envolves some danger or risk? If not I suggest you to publish it in any way=
=20
since it's really cool and a missing feature of Ruby.




=2D-=20
I=F1aki Baz Castillo

 

[ara.t.howard]

Do you think your code can be feasible for production enviroments? =20
maybe it
envolves some danger or risk? If not I suggest you to publish it in =20=

any way
since it's really cool and a missing feature of Ruby.

i've got something close to gem'ing... there is nothing wrong with the =20=

concept - this is precisly how objects are returned from drb: =20
marshaled data over a socket/pipe.

a @ http://codeforpeople.com/

 

[Fredrik]

The new "n" parameter is mandatory since it's the first parameter. It would be

nice if it could be not defined (so = infinite):

forkmap(4) { code } --> max 4 process
forkmap { code } --> max infinite

I was thinking about that too, but as far as I understand it Ruby only
allows optional arguments to be the last arguments - i.e. the "n"
parameter would have to appear after the code block. And that would
look strange : forkmap{ code }(4).