Ruby Threads 101

[Ben]

I am leading a peer-learning group that is using "Programming Ruby" to
learn Ruby. We are reading chapters 11-13 for our next session, and
those chapters include Ruby Threads.

I have promised to find some exercise that uses multiple threads. The
group is very experienced with multi-threaded programming (we're
mostly embedded C++ developers) so the thread coordination part
doesn't need to be overly simple. The Rubyish-ness shouldn't be to
complicated - if that makes any sense.

I looked through rubyquiz.com (where I've been pulling a few of our
exercises), but nothing jumped out at me. Did I miss something?

Any suggestions?

Thanks
-Ben

 

[James Edward Gray II]

I looked through rubyquiz.com (where I've been pulling a few of our
exercises), but nothing jumped out at me. Did I miss something?

Glad to hear it's proving useful.

We haven't had a good threading problem yet, but I'm very open to the
idea if someone can think one up...

James Edward Gray II

 

[Joe Van Dyk]

Glad to hear it's proving useful.

We haven't had a good threading problem yet, but I'm very open to the
idea if someone can think one up...

I have one about a bunch of hungry commie philosophers...

 

[Robert Klemme]

I have one about a bunch of hungry commie philosophers...

LOL

Ben, you could start with a simple problem (like farmer worker e.g. a small
app that crawls web pages in parallel with controllable degree of
parallelism) to get the basic feeling and then work your way up to more
complicated things.

I once cooked something up on the Wiki (including a superfluous queue
implementation - there is one that comes with Ruby) although I guess it
won't be much use for you as you seem to be rather senior when it comes to
threading.
http://www.rubygarden.org/ruby?MultiThreading

One thing to keep in mind is that Ruby threads are not (yet) native threads.
Despite of that they work quite nicely IMHO.

Kind regards

robert

 

[Ben]

I once cooked something up on the Wiki (including a superfluous queue
implementation - there is one that comes with Ruby) although I guess it
won't be much use for you as you seem to be rather senior when it comes t= o
threading.
http://www.rubygarden.org/ruby?MultiThreading

One thing to keep in mind is that Ruby threads are not (yet) native threa= ds.
Despite of that they work quite nicely IMHO.

Thanks for the wiki pointer. It's always nice to see how something
familiar is done in another language. Gives me a feeling for what the
differences are.

Why is it that Ruby threads aren't native? Lack of time to make
them so, or is it the long term plan to never use native threads?
Also, the book was a little vague about when all threads block
waiting for one thread. It said a "call to the operating system"
would cause all threads to block. What did that mean exactly? Shell
commands? Opening files? Sockets? A lot of common actions results
in system calls.

As for my project, I am considering having the group write an app
that downloads fantasy football stats for several different positions
from a web page. (Each position would be grabbed and parsed by a
different thread). Those stats would be scored and each positions
players would be sorted by score. A shared table sorted by overall
score would be updated on a player by player basis.

DId that make sense? Thoughts?

-Ben

 

[Zed A. Shaw]

Ben,

I had a lot of fun implementing the following patterns as a way of
avoiding almost all Thread primitives in a distributed video
processing system and boosting performance quite a bit:

http://alumnus.caltech.edu/~croft/research/agent/tuplespaces/

In fact, Ruby comes with a nice tuple space implementation, so you
could take the above patterns, implement single instance versions, and
then try to distribute them.

Those patterns are also good examples of why the logical proof of
single machine thread primitives does not instantly extend to
distributed processing. They are much simpler to use and do
exactly the same thing as distributed locking with nothing more than
simple tuple-space semantics rather than the complicated mutex and
semaphore semantics.

I would also take a look at the work done on lock-free data structures
and algorithms. Lots of really great stuff that basically shows you
don't need locking on many data structures. Even more proof
that 1970's locking technology needs to be rethunk.

My favorite fun project is doing a Replicated-Worker pattern with just
message queues. Sure the Queue needs locking, but why do you need to
know that?

Enjoy!

Zed A. Shaw
http://www.zedshaw.com/

 

[Zed A. Shaw]

Haha, I thought this would come up. I smell another blog entry coming
up.

 

[Lyndon Samson]

------=_Part_8650_31579009.1127783733242
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

How about multiple stomp http://activemq.codehaus.org/Stomp subscribers ( 1
thread each ) writing to a queue flushed to a db from another thread.

------=_Part_8650_31579009.1127783733242--

 

[Logan Capaldo]

Ben,

I had a lot of fun implementing the following patterns as a way of
avoiding almost all Thread primitives in a distributed video
processing system and boosting performance quite a bit:

http://alumnus.caltech.edu/~croft/research/agent/tuplespaces/

In fact, Ruby comes with a nice tuple space implementation, so you
could take the above patterns, implement single instance versions, and
then try to distribute them.

Those patterns are also good examples of why the logical proof of
single machine thread primitives does not instantly extend to
distributed processing. They are much simpler to use and do
exactly the same thing as distributed locking with nothing more than
simple tuple-space semantics rather than the complicated mutex and
semaphore semantics.

I would also take a look at the work done on lock-free data structures
and algorithms. Lots of really great stuff that basically shows you
don't need locking on many data structures. Even more proof
that 1970's locking technology needs to be rethunk.

My favorite fun project is doing a Replicated-Worker pattern with just
message queues. Sure the Queue needs locking, but why do you need to
know that?

Enjoy!

Zed A. Shaw
http://www.zedshaw.com/

Speaking of lock-free data sructures (sort of) has anyone ever
implemented Erlang or E -style promises in Ruby?

 

[Pit Capitain]

Speaking of lock-free data sructures (sort of) has anyone ever
implemented Erlang or E -style promises in Ruby?

What features are you looking for?

Regards,
Pit

 

[Robert Klemme]

Ben,

I had a lot of fun implementing the following patterns as a way of
avoiding almost all Thread primitives in a distributed video
processing system and boosting performance quite a bit:

http://alumnus.caltech.edu/~croft/research/agent/tuplespaces/

In fact, Ruby comes with a nice tuple space implementation, so you
could take the above patterns, implement single instance versions, and
then try to distribute them.

As far as I can see the idea is basically to have a synchronized
repository (typically a queue) where several workers can fetch their tasks
from and during working on the task there is no synchronization needed (in
Doug Lea terms "lightweight processing framework" with "thread
confinement"). Not really fancy or did I miss something?

Those patterns are also good examples of why the logical proof of
single machine thread primitives does not instantly extend to
distributed processing. They are much simpler to use and do
exactly the same thing as distributed locking with nothing more than
simple tuple-space semantics rather than the complicated mutex and
semaphore semantics.

Honestly, I don't think mutexes and semaphores are out of business simply
because you don't see them directly. Concurrency - however done - adds
significantly to the complexity of an application and people creating such
applications shoule be well aware of the basic problems they have to solve
to make them reliably do what they want.

I would also take a look at the work done on lock-free data structures
and algorithms. Lots of really great stuff that basically shows you
don't need locking on many data structures. Even more proof
that 1970's locking technology needs to be rethunk.

Hmm... Still you need *some* form of synchronization / locking. The
point is to make it as efficient as possible, i.e., try to reduce
synchronization points. I don't know what exactly you mean by "1970's
locking technology" but I think the basics didn't change that much from
the 70's on.

My favorite fun project is doing a Replicated-Worker pattern with just
message queues. Sure the Queue needs locking, but why do you need to
know that?

In oder to understand what you are doing. People that apply this pattern
(see above) need to know why it's working IMHO.

Kind regards

robert

 

[Robert Klemme]

Thanks for the wiki pointer. It's always nice to see how something
familiar is done in another language. Gives me a feeling for what the
differences are.

Why is it that Ruby threads aren't native? Lack of time to make
them so, or is it the long term plan to never use native threads?

No, AFAIK it's planned to use native threads in Ruby 2. I believe the
reason behind the current design is portability (at the time Ruby was
created). Matz?

Also, the book was a little vague about when all threads block
waiting for one thread. It said a "call to the operating system"
would cause all threads to block. What did that mean exactly? Shell
commands? Opening files? Sockets? A lot of common actions results
in system calls.

Yes. IO is done nonblocking under the hood so you don't have to fear that
while reading from a file no other thread can be active. Other than that
since Ruby threads are completely user space the interpreter has no way to
make a context switch between system calls. But usually this is not a
problem apart from DNS lookups that sometimes seem to make scripts slow.

As for my project, I am considering having the group write an app
that downloads fantasy football stats for several different positions
from a web page. (Each position would be grabbed and parsed by a
different thread). Those stats would be scored and each positions
players would be sorted by score. A shared table sorted by overall
score would be updated on a player by player basis.

DId that make sense? Thoughts?

Sounds ok for a first bit. You have at least one shared resource (the
table) and can extend that with different approaches to thread creation
(create a thread per position, create n threads and let them do the work
farmer-worker like etc.).

Kind regards

robert