Inter-Process Messaging

D

Daniel DeLorme

What are the possibilities in ruby for passing information from one
process to another? Specifically, I'm thinking of messaging between a
parent process and its forked child.

Right now I can think of some messaging primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket (unstable?)
- IO.pipe (doesn't need port#)
- Process.kill (impossible to send data)

And some messaging libraries:
- DRb (standard)
- eventmachine (efficient?)

...what else? And, if anyone has the experience, what are the
advantages/disadvantages of each in terms of speed, reliability, system
resources?

Thanks,
Daniel
 
M

Markus Schirp

Am Wed, 10 Oct 2007 16:11:15 +0900
schrieb Daniel DeLorme said:
What are the possibilities in ruby for passing information from one
process to another? Specifically, I'm thinking of messaging between a
parent process and its forked child.

Right now I can think of some messaging primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket (unstable?)
- IO.pipe (doesn't need port#)
- Process.kill (impossible to send data)

And some messaging libraries:
- DRb (standard)
- eventmachine (efficient?)

...what else? And, if anyone has the experience, what are the
advantages/disadvantages of each in terms of speed, reliability,
system resources?

Thanks,
Daniel

shared memory?

I don't expect UNIXSocket to be unstable. I'm running over 2 years
with using ruby's socket library without problems. Dont forget to
remove the created socket if your process ends :)
 
M

M. Edward (Ed) Borasky

Francis said:
Intramachine messaging with any of the technologies you mentioned will be
pretty lightweight. The usual thing one tries to do with related processes
is to open a pipe or a socketpair in the parent. Then after the fork, just
send data over the sockets.

There's also a Ruby MPI binding, but I'm not sure how well maintained it
is. I had trouble tracking it down on the web. So I'd go with
EventMachine, because that *is* well maintained.
 
D

Daniel DeLorme

Markus said:
shared memory?

Is there a ruby lib to handle shared memory? Also, I can see how it can
be used for "passing information", but for messaging between processes
wouldn't it require constant polling to check if a message is waiting in
the shared memory?
I don't expect UNIXSocket to be unstable. I'm running over 2 years
with using ruby's socket library without problems. Dont forget to
remove the created socket if your process ends :)

I just vaguely remember reading in some mailing list or blog something
along the lines that "unix domain sockets are flaky". I might have
misunderstood.

Daniel
 
J

James Edward Gray II

So I'd go with EventMachine, because that *is* well maintained.

I would go with a pipe, because it's easy and sounds like it's all
that's needed here. I'm an EventMachine fan, but don't underestimate
the trivial approach.

James Edward Gray II
 
M

M. Edward (Ed) Borasky

James said:
I would go with a pipe, because it's easy and sounds like it's all
that's needed here. I'm an EventMachine fan, but don't underestimate
the trivial approach.

James Edward Gray II

Well, if the architecture is *always* going to be "parent-forked child
message passing on a single machine", sure. But what if the task grows
beyond the capabilities of that architecture? Remember, I'm one of those
people who's in *favor* of "premature" optimization. :)

But seriously, Ruby has so many user-friendly ways to do concurrency
both inside a single machine and across machines that I wouldn't limit
myself to parent-forked child message passing on a single machine. I'd
use EventMachine because it's more flexible. And to the person who
suggested "shared memory", I'd say simply, "Bah!" :)
 
A

ara.t.howard

What are the possibilities in ruby for passing information from one
process to another? Specifically, I'm thinking of messaging between
a parent process and its forked child.

Right now I can think of some messaging primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket (unstable?)
- IO.pipe (doesn't need port#)
- Process.kill (impossible to send data)

And some messaging libraries:
- DRb (standard)
- eventmachine (efficient?)

...what else? And, if anyone has the experience, what are the
advantages/disadvantages of each in terms of speed, reliability,
system resources?

Thanks,
Daniel

this is, by far, the easiest approach - it addresses exactly your
problem by sticking a drb server in a forked child. the child has
arrangements made such that it can *never* outlive it's parent:

cfp:~ > cat a.rb
require 'slave' ### gem install slave

class Child
def pid
Process.pid
end
def foobar
42
end
end

slave = Slave.new{ Child.new }
child = slave.object

p Process.pid
p child.pid
p child.foobar


cfp:~ > ruby a.rb
20309
20310
42


also, if your arch changes it short work to rework such that the drb
objects are not in a child process but, rather, are distributed
across machines.

this code is used *heavily* in many of our production systems.

kind regards.

* http://codeforpeople.com/lib/ruby/slave/slave-1.2.1/README


a @ http://codeforpeople.com/
 
E

Eric Hodel

What are the possibilities in ruby for passing information from one
process to another? Specifically, I'm thinking of messaging between
a parent process and its forked child.

Right now I can think of some messaging primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket (unstable?)
- IO.pipe (doesn't need port#)
- Process.kill (impossible to send data)

And some messaging libraries:
- DRb (standard)
- eventmachine (efficient?)

...what else? And, if anyone has the experience, what are the
advantages/disadvantages of each in terms of speed, reliability,
system resources?

DRb will cause you to do the least amount of work. You only need to
speak ruby, you don't need to do any serializing of data because it
all gets handled for you.
 
J

Joel VanderWerf

Markus Schirp wrote:
...
I don't expect UNIXSocket to be unstable. I'm running over 2 years
with using ruby's socket library without problems. Dont forget to
remove the created socket if your process ends :)

Ditto. I've been using DRb over UNIX sockets extensively for about 2
years, and no problems here. Data volume and rate tend to be small in my
case, though.
 
J

Joel VanderWerf

Daniel said:
Right now I can think of some messaging primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket (unstable?)
- IO.pipe (doesn't need port#)
- Process.kill (impossible to send data)

For completeness:

- named pipes (man fifo)
 
G

Gary Wright

Daniel said:
Right now I can think of some messaging primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket (unstable?)
- IO.pipe (doesn't need port#)
- Process.kill (impossible to send data)

Not that it would be useful but I guess that if you used
two different signals (say SIGUSR1 and SIGUSR2) then you've
got a method of communicating a sequence of binary digits
to another process. If you've got a reasonably accurate
way of timing things you could just manage this with a single
signal because the absence of the signal could be detected.
 
J

Joel VanderWerf

Gary said:
Not that it would be useful but I guess that if you used
two different signals (say SIGUSR1 and SIGUSR2) then you've
got a method of communicating a sequence of binary digits
to another process. If you've got a reasonably accurate
way of timing things you could just manage this with a single
signal because the absence of the signal could be detected.

I smell a ruby quiz: implement drb messaging on top of sigusr[12].
 
G

Gary Wright

Gary said:
Not that it would be useful but I guess that if you used
two different signals (say SIGUSR1 and SIGUSR2) then you've
got a method of communicating a sequence of binary digits
to another process. If you've got a reasonably accurate
way of timing things you could just manage this with a single
signal because the absence of the signal could be detected.

I smell a ruby quiz: implement drb messaging on top of sigusr[12].

Hmm. It would be difficult. Since signals are not reliable
in general, you would probably first have to construct the
equivalent of TCP/IP across signals. Drb messaging should be
easy after that...

Gary Wright
 
G

Gary Wright

There's some cleverness to this idea but I would avoid it. Signals
interact
very badly with threads and other system facilities, they're not
deterministic, they have serious platform dependencies among different
Unixes, they're heavyweight (resource intensive), and they have
difficult
APIs.

I guess I should have added some smiley faces. :)

There is a whole area of computer security though, covert
signaling methods, where low bandwidth, unreliable communication paths
are still of interest.
 
D

Daniel DeLorme

Thanks for all the answers. I now have:

primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket
- IO.pipe
- named pipes
- shared memory (can't find low-level lib?)

libraries: (after some digging on rubyforge)
- DRb
- Event Machine
- ActiveMessaging
- Slave
- BackgroundDRb
- System V IPC
- POSIXIPC (not released)
- reliable-msg
- MPI Ruby
- stomp / stompserver / stompmessage
- AP4R (Asynchronous Processing for Ruby)

Frankly I'm still not entirely clear on what are the advantages of each,
both in terms of features and in terms of performance. I guess I'll just
have to write some code to find out...

Daniel
 
M

M. Edward (Ed) Borasky

Daniel said:
Thanks for all the answers. I now have:

primitives:
- TCPServer/TCPSocket
- UNIXServer/UNIXSocket
- IO.pipe
- named pipes
- shared memory (can't find low-level lib?)

libraries: (after some digging on rubyforge)
- DRb
- Event Machine
- ActiveMessaging
- Slave
- BackgroundDRb
- System V IPC
- POSIXIPC (not released)
- reliable-msg
- MPI Ruby
- stomp / stompserver / stompmessage
- AP4R (Asynchronous Processing for Ruby)

Frankly I'm still not entirely clear on what are the advantages of each,
both in terms of features and in terms of performance. I guess I'll just
have to write some code to find out...

Daniel

1. You missed Rinda, a Linda derivative layered on top of DRb.

2. There are basically two ways to do concurrency/parallelism: shared
memory and message passing. I'm not sure what the *real* tradeoffs are
-- I've pretty much had my brain hammered with "shared memory bad --
message passing good", but there obviously must be *some*
counter-arguments, or shared memory wouldn't exist. :)

3. System V IPC has three components -- message queues, semaphores, and
shared memory segments. So ... if you've found a System V IPC library,
you've found a shared memory library.
 
B

Bill Kelly

possibly mmap: http://moulon.inra.fr/ruby/mmap.html

1. You missed Rinda, a Linda derivative layered on top of DRb.

2. There are basically two ways to do concurrency/parallelism: shared
memory and message passing. I'm not sure what the *real* tradeoffs are
-- I've pretty much had my brain hammered with "shared memory bad --
message passing good", but there obviously must be *some*
counter-arguments, or shared memory wouldn't exist. :)

The way I look at it, Re: trade-offs, message passing is nice
because the API doesn't change whether the receiver is on localhost
or over the wire.

Shared memory is nice for very high bandwidth between processes;
say process A decodes a 300MB TIFF image into memory and wants
to make the bitmap available to process B.

3. System V IPC has three components -- message queues, semaphores, and
shared memory segments. So ... if you've found a System V IPC library,
you've found a shared memory library.


Regards,

Bill
 
J

Joel VanderWerf

Bill said:
From: "M. Edward (Ed) Borasky" <[email protected]> ...

The way I look at it, Re: trade-offs, message passing is nice
because the API doesn't change whether the receiver is on localhost
or over the wire.

Shared memory is nice for very high bandwidth between processes;
say process A decodes a 300MB TIFF image into memory and wants
to make the bitmap available to process B.

And shared memory might also be nice if you don't want queueing, you
just want the latest data. Sensors in a real-time system, for example.
 
B

Bill Kelly

(Just wanted to correct myself and say the API *need not* change,
rather than does not change... certainly some message passing
APIs only work on the local machine, but others are network-
transparent. :)
And shared memory might also be nice if you don't want queueing, you
just want the latest data. Sensors in a real-time system, for example.

Ah, yeah. Makes sense.

Indeed, this can also apply over the wire; for example: dealing with
a client->server->client remote mouse click-drag-release operation.
(Where, for example, a client is dragging a scrollbar thumb where
the UI is hosted on the server, and may be being rendered to multiple
clients.) One might transmit the initial click coordinate over TCP,
then the subsequent real-time drag telemetry over UDP, and finally the
terminating click-release coordinate again over TCP. (Such that, TCP
is used for reliable messages, UDP for unreliable streaming best-effort
real-time telemetry where one only wants the latest data.)


Regards,

Bill
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top