Inter-Process Messaging

Discussion in 'Ruby' started by Daniel DeLorme, Oct 10, 2007.

  1. What are the possibilities in ruby for passing information from one
    process to another? Specifically, I'm thinking of messaging between a
    parent process and its forked child.

    Right now I can think of some messaging primitives:
    - TCPServer/TCPSocket
    - UNIXServer/UNIXSocket (unstable?)
    - IO.pipe (doesn't need port#)
    - Process.kill (impossible to send data)

    And some messaging libraries:
    - DRb (standard)
    - eventmachine (efficient?)

    ...what else? And, if anyone has the experience, what are the
    advantages/disadvantages of each in terms of speed, reliability, system
    resources?

    Thanks,
    Daniel
     
    Daniel DeLorme, Oct 10, 2007
    #1
    1. Advertising

  2. Am Wed, 10 Oct 2007 16:11:15 +0900
    schrieb Daniel DeLorme <>:

    > What are the possibilities in ruby for passing information from one
    > process to another? Specifically, I'm thinking of messaging between a
    > parent process and its forked child.
    >
    > Right now I can think of some messaging primitives:
    > - TCPServer/TCPSocket
    > - UNIXServer/UNIXSocket (unstable?)
    > - IO.pipe (doesn't need port#)
    > - Process.kill (impossible to send data)
    >
    > And some messaging libraries:
    > - DRb (standard)
    > - eventmachine (efficient?)
    >
    > ...what else? And, if anyone has the experience, what are the
    > advantages/disadvantages of each in terms of speed, reliability,
    > system resources?
    >
    > Thanks,
    > Daniel
    >


    shared memory?

    I don't expect UNIXSocket to be unstable. I'm running over 2 years
    with using ruby's socket library without problems. Dont forget to
    remove the created socket if your process ends :)
     
    Markus Schirp, Oct 10, 2007
    #2
    1. Advertising

  3. Francis Cianfrocca wrote:
    > On 10/10/07, Daniel DeLorme <> wrote:
    >> What are the possibilities in ruby for passing information from one
    >> process to another? Specifically, I'm thinking of messaging between a
    >> parent process and its forked child.
    >>
    >> Right now I can think of some messaging primitives:
    >> - TCPServer/TCPSocket
    >> - UNIXServer/UNIXSocket (unstable?)
    >> - IO.pipe (doesn't need port#)
    >> - Process.kill (impossible to send data)
    >>
    >> And some messaging libraries:
    >> - DRb (standard)
    >> - eventmachine (efficient?)
    >>
    >> ...what else? And, if anyone has the experience, what are the
    >> advantages/disadvantages of each in terms of speed, reliability, system
    >> resources?

    >
    >
    >
    > Intramachine messaging with any of the technologies you mentioned will be
    > pretty lightweight. The usual thing one tries to do with related processes
    > is to open a pipe or a socketpair in the parent. Then after the fork, just
    > send data over the sockets.
    >


    There's also a Ruby MPI binding, but I'm not sure how well maintained it
    is. I had trouble tracking it down on the web. So I'd go with
    EventMachine, because that *is* well maintained.
     
    M. Edward (Ed) Borasky, Oct 10, 2007
    #3
  4. Markus Schirp wrote:
    >> What are the possibilities in ruby for passing information from one
    >> process to another? Specifically, I'm thinking of messaging between a
    >> parent process and its forked child.

    >
    > shared memory?


    Is there a ruby lib to handle shared memory? Also, I can see how it can
    be used for "passing information", but for messaging between processes
    wouldn't it require constant polling to check if a message is waiting in
    the shared memory?

    > I don't expect UNIXSocket to be unstable. I'm running over 2 years
    > with using ruby's socket library without problems. Dont forget to
    > remove the created socket if your process ends :)


    I just vaguely remember reading in some mailing list or blog something
    along the lines that "unix domain sockets are flaky". I might have
    misunderstood.

    Daniel
     
    Daniel DeLorme, Oct 10, 2007
    #4
  5. On Oct 10, 2007, at 9:39 AM, M. Edward (Ed) Borasky wrote:

    > So I'd go with EventMachine, because that *is* well maintained.


    I would go with a pipe, because it's easy and sounds like it's all
    that's needed here. I'm an EventMachine fan, but don't underestimate
    the trivial approach.

    James Edward Gray II
     
    James Edward Gray II, Oct 10, 2007
    #5
  6. James Edward Gray II wrote:
    > On Oct 10, 2007, at 9:39 AM, M. Edward (Ed) Borasky wrote:
    >
    >> So I'd go with EventMachine, because that *is* well maintained.

    >
    > I would go with a pipe, because it's easy and sounds like it's all
    > that's needed here. I'm an EventMachine fan, but don't underestimate
    > the trivial approach.
    >
    > James Edward Gray II
    >
    >


    Well, if the architecture is *always* going to be "parent-forked child
    message passing on a single machine", sure. But what if the task grows
    beyond the capabilities of that architecture? Remember, I'm one of those
    people who's in *favor* of "premature" optimization. :)

    But seriously, Ruby has so many user-friendly ways to do concurrency
    both inside a single machine and across machines that I wouldn't limit
    myself to parent-forked child message passing on a single machine. I'd
    use EventMachine because it's more flexible. And to the person who
    suggested "shared memory", I'd say simply, "Bah!" :)
     
    M. Edward (Ed) Borasky, Oct 10, 2007
    #6
  7. Daniel DeLorme

    ara.t.howard Guest

    On Oct 10, 2007, at 1:11 AM, Daniel DeLorme wrote:

    > What are the possibilities in ruby for passing information from one
    > process to another? Specifically, I'm thinking of messaging between
    > a parent process and its forked child.
    >
    > Right now I can think of some messaging primitives:
    > - TCPServer/TCPSocket
    > - UNIXServer/UNIXSocket (unstable?)
    > - IO.pipe (doesn't need port#)
    > - Process.kill (impossible to send data)
    >
    > And some messaging libraries:
    > - DRb (standard)
    > - eventmachine (efficient?)
    >
    > ...what else? And, if anyone has the experience, what are the
    > advantages/disadvantages of each in terms of speed, reliability,
    > system resources?
    >
    > Thanks,
    > Daniel


    this is, by far, the easiest approach - it addresses exactly your
    problem by sticking a drb server in a forked child. the child has
    arrangements made such that it can *never* outlive it's parent:

    cfp:~ > cat a.rb
    require 'slave' ### gem install slave

    class Child
    def pid
    Process.pid
    end
    def foobar
    42
    end
    end

    slave = Slave.new{ Child.new }
    child = slave.object

    p Process.pid
    p child.pid
    p child.foobar


    cfp:~ > ruby a.rb
    20309
    20310
    42


    also, if your arch changes it short work to rework such that the drb
    objects are not in a child process but, rather, are distributed
    across machines.

    this code is used *heavily* in many of our production systems.

    kind regards.

    * http://codeforpeople.com/lib/ruby/slave/slave-1.2.1/README


    a @ http://codeforpeople.com/
    --
    we can deny everything, except that we have the possibility of being
    better. simply reflect on that.
    h.h. the 14th dalai lama
     
    ara.t.howard, Oct 10, 2007
    #7
  8. Daniel DeLorme

    Eric Hodel Guest

    On Oct 10, 2007, at 24:11 , Daniel DeLorme wrote:
    > What are the possibilities in ruby for passing information from one
    > process to another? Specifically, I'm thinking of messaging between
    > a parent process and its forked child.
    >
    > Right now I can think of some messaging primitives:
    > - TCPServer/TCPSocket
    > - UNIXServer/UNIXSocket (unstable?)
    > - IO.pipe (doesn't need port#)
    > - Process.kill (impossible to send data)
    >
    > And some messaging libraries:
    > - DRb (standard)
    > - eventmachine (efficient?)
    >
    > ...what else? And, if anyone has the experience, what are the
    > advantages/disadvantages of each in terms of speed, reliability,
    > system resources?


    DRb will cause you to do the least amount of work. You only need to
    speak ruby, you don't need to do any serializing of data because it
    all gets handled for you.

    --
    Poor workers blame their tools. Good workers build better tools. The
    best workers get their tools to do the work for them. -- Syndicate Wars
     
    Eric Hodel, Oct 10, 2007
    #8
  9. Markus Schirp wrote:
    ...
    > I don't expect UNIXSocket to be unstable. I'm running over 2 years
    > with using ruby's socket library without problems. Dont forget to
    > remove the created socket if your process ends :)


    Ditto. I've been using DRb over UNIX sockets extensively for about 2
    years, and no problems here. Data volume and rate tend to be small in my
    case, though.

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Oct 10, 2007
    #9
  10. Daniel DeLorme wrote:
    > Right now I can think of some messaging primitives:
    > - TCPServer/TCPSocket
    > - UNIXServer/UNIXSocket (unstable?)
    > - IO.pipe (doesn't need port#)
    > - Process.kill (impossible to send data)


    For completeness:

    - named pipes (man fifo)

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Oct 10, 2007
    #10
  11. Daniel DeLorme

    Gary Wright Guest

    Daniel DeLorme wrote:
    > Right now I can think of some messaging primitives:
    > - TCPServer/TCPSocket
    > - UNIXServer/UNIXSocket (unstable?)
    > - IO.pipe (doesn't need port#)
    > - Process.kill (impossible to send data)


    Not that it would be useful but I guess that if you used
    two different signals (say SIGUSR1 and SIGUSR2) then you've
    got a method of communicating a sequence of binary digits
    to another process. If you've got a reasonably accurate
    way of timing things you could just manage this with a single
    signal because the absence of the signal could be detected.
     
    Gary Wright, Oct 10, 2007
    #11
  12. Gary Wright wrote:
    >
    > Daniel DeLorme wrote:
    >> Right now I can think of some messaging primitives:
    >> - TCPServer/TCPSocket
    >> - UNIXServer/UNIXSocket (unstable?)
    >> - IO.pipe (doesn't need port#)
    >> - Process.kill (impossible to send data)

    >
    > Not that it would be useful but I guess that if you used
    > two different signals (say SIGUSR1 and SIGUSR2) then you've
    > got a method of communicating a sequence of binary digits
    > to another process. If you've got a reasonably accurate
    > way of timing things you could just manage this with a single
    > signal because the absence of the signal could be detected.


    I smell a ruby quiz: implement drb messaging on top of sigusr[12].

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Oct 10, 2007
    #12
  13. Daniel DeLorme

    Gary Wright Guest

    On Oct 10, 2007, at 5:03 PM, Joel VanderWerf wrote:

    > Gary Wright wrote:
    >> Daniel DeLorme wrote:
    >>> Right now I can think of some messaging primitives:
    >>> - TCPServer/TCPSocket
    >>> - UNIXServer/UNIXSocket (unstable?)
    >>> - IO.pipe (doesn't need port#)
    >>> - Process.kill (impossible to send data)

    >> Not that it would be useful but I guess that if you used
    >> two different signals (say SIGUSR1 and SIGUSR2) then you've
    >> got a method of communicating a sequence of binary digits
    >> to another process. If you've got a reasonably accurate
    >> way of timing things you could just manage this with a single
    >> signal because the absence of the signal could be detected.

    >
    > I smell a ruby quiz: implement drb messaging on top of sigusr[12].


    Hmm. It would be difficult. Since signals are not reliable
    in general, you would probably first have to construct the
    equivalent of TCP/IP across signals. Drb messaging should be
    easy after that...

    Gary Wright
     
    Gary Wright, Oct 10, 2007
    #13
  14. Daniel DeLorme

    Gary Wright Guest

    On Oct 10, 2007, at 6:04 PM, Francis Cianfrocca wrote:
    > On 10/10/07, Gary Wright <> wrote:
    >>
    >>
    >> Daniel DeLorme wrote:
    >>> Right now I can think of some messaging primitives:
    >>> - TCPServer/TCPSocket
    >>> - UNIXServer/UNIXSocket (unstable?)
    >>> - IO.pipe (doesn't need port#)
    >>> - Process.kill (impossible to send data)

    >>
    >> Not that it would be useful but I guess that if you used
    >> two different signals (say SIGUSR1 and SIGUSR2) then you've
    >> got a method of communicating a sequence of binary digits
    >> to another process. If you've got a reasonably accurate
    >> way of timing things you could just manage this with a single
    >> signal because the absence of the signal could be detected.

    >
    > There's some cleverness to this idea but I would avoid it. Signals
    > interact
    > very badly with threads and other system facilities, they're not
    > deterministic, they have serious platform dependencies among different
    > Unixes, they're heavyweight (resource intensive), and they have
    > difficult
    > APIs.


    I guess I should have added some smiley faces. :)

    There is a whole area of computer security though, covert
    signaling methods, where low bandwidth, unreliable communication paths
    are still of interest.
     
    Gary Wright, Oct 11, 2007
    #14
  15. Gary Wright wrote:
    > I guess I should have added some smiley faces. :)


    : )

    > There is a whole area of computer security though, covert
    > signaling methods, where low bandwidth, unreliable communication paths
    > are still of interest.


    http://www.ietf.org/rfc/rfc1149.txt

    in case anyone hasn't seen that yet.

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Oct 11, 2007
    #15
  16. Thanks for all the answers. I now have:

    primitives:
    - TCPServer/TCPSocket
    - UNIXServer/UNIXSocket
    - IO.pipe
    - named pipes
    - shared memory (can't find low-level lib?)

    libraries: (after some digging on rubyforge)
    - DRb
    - Event Machine
    - ActiveMessaging
    - Slave
    - BackgroundDRb
    - System V IPC
    - POSIXIPC (not released)
    - reliable-msg
    - MPI Ruby
    - stomp / stompserver / stompmessage
    - AP4R (Asynchronous Processing for Ruby)

    Frankly I'm still not entirely clear on what are the advantages of each,
    both in terms of features and in terms of performance. I guess I'll just
    have to write some code to find out...

    Daniel
     
    Daniel DeLorme, Oct 12, 2007
    #16
  17. Daniel DeLorme wrote:
    > Thanks for all the answers. I now have:
    >
    > primitives:
    > - TCPServer/TCPSocket
    > - UNIXServer/UNIXSocket
    > - IO.pipe
    > - named pipes
    > - shared memory (can't find low-level lib?)
    >
    > libraries: (after some digging on rubyforge)
    > - DRb
    > - Event Machine
    > - ActiveMessaging
    > - Slave
    > - BackgroundDRb
    > - System V IPC
    > - POSIXIPC (not released)
    > - reliable-msg
    > - MPI Ruby
    > - stomp / stompserver / stompmessage
    > - AP4R (Asynchronous Processing for Ruby)
    >
    > Frankly I'm still not entirely clear on what are the advantages of each,
    > both in terms of features and in terms of performance. I guess I'll just
    > have to write some code to find out...
    >
    > Daniel
    >
    >


    1. You missed Rinda, a Linda derivative layered on top of DRb.

    2. There are basically two ways to do concurrency/parallelism: shared
    memory and message passing. I'm not sure what the *real* tradeoffs are
    -- I've pretty much had my brain hammered with "shared memory bad --
    message passing good", but there obviously must be *some*
    counter-arguments, or shared memory wouldn't exist. :)

    3. System V IPC has three components -- message queues, semaphores, and
    shared memory segments. So ... if you've found a System V IPC library,
    you've found a shared memory library.
     
    M. Edward (Ed) Borasky, Oct 12, 2007
    #17
  18. Daniel DeLorme

    Bill Kelly Guest

    From: "M. Edward (Ed) Borasky" <>
    > Daniel DeLorme wrote:
    >> Thanks for all the answers. I now have:
    >>
    >> primitives:
    >> - TCPServer/TCPSocket
    >> - UNIXServer/UNIXSocket
    >> - IO.pipe
    >> - named pipes
    >> - shared memory (can't find low-level lib?)


    possibly mmap: http://moulon.inra.fr/ruby/mmap.html


    >>
    >> libraries: (after some digging on rubyforge)
    >> - DRb
    >> - Event Machine
    >> - ActiveMessaging
    >> - Slave
    >> - BackgroundDRb
    >> - System V IPC
    >> - POSIXIPC (not released)
    >> - reliable-msg
    >> - MPI Ruby
    >> - stomp / stompserver / stompmessage
    >> - AP4R (Asynchronous Processing for Ruby)
    >>
    >> Frankly I'm still not entirely clear on what are the advantages of each,
    >> both in terms of features and in terms of performance. I guess I'll just
    >> have to write some code to find out...
    >>
    >> Daniel
    >>
    >>

    >
    > 1. You missed Rinda, a Linda derivative layered on top of DRb.
    >
    > 2. There are basically two ways to do concurrency/parallelism: shared
    > memory and message passing. I'm not sure what the *real* tradeoffs are
    > -- I've pretty much had my brain hammered with "shared memory bad --
    > message passing good", but there obviously must be *some*
    > counter-arguments, or shared memory wouldn't exist. :)


    The way I look at it, Re: trade-offs, message passing is nice
    because the API doesn't change whether the receiver is on localhost
    or over the wire.

    Shared memory is nice for very high bandwidth between processes;
    say process A decodes a 300MB TIFF image into memory and wants
    to make the bitmap available to process B.


    > 3. System V IPC has three components -- message queues, semaphores, and
    > shared memory segments. So ... if you've found a System V IPC library,
    > you've found a shared memory library.



    Regards,

    Bill
     
    Bill Kelly, Oct 12, 2007
    #18
  19. Bill Kelly wrote:
    >
    > From: "M. Edward (Ed) Borasky" <>

    ...
    >> 2. There are basically two ways to do concurrency/parallelism: shared
    >> memory and message passing. I'm not sure what the *real* tradeoffs are
    >> -- I've pretty much had my brain hammered with "shared memory bad --
    >> message passing good", but there obviously must be *some*
    >> counter-arguments, or shared memory wouldn't exist. :)

    >
    > The way I look at it, Re: trade-offs, message passing is nice
    > because the API doesn't change whether the receiver is on localhost
    > or over the wire.
    >
    > Shared memory is nice for very high bandwidth between processes;
    > say process A decodes a 300MB TIFF image into memory and wants
    > to make the bitmap available to process B.


    And shared memory might also be nice if you don't want queueing, you
    just want the latest data. Sensors in a real-time system, for example.

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Oct 12, 2007
    #19
  20. Daniel DeLorme

    Bill Kelly Guest

    From: "Joel VanderWerf" <>
    > Bill Kelly wrote:
    >>
    >> From: "M. Edward (Ed) Borasky" <>

    > ...
    >>> 2. There are basically two ways to do concurrency/parallelism: shared
    >>> memory and message passing. I'm not sure what the *real* tradeoffs are
    >>> -- I've pretty much had my brain hammered with "shared memory bad --
    >>> message passing good", but there obviously must be *some*
    >>> counter-arguments, or shared memory wouldn't exist. :)

    >>
    >> The way I look at it, Re: trade-offs, message passing is nice
    >> because the API doesn't change whether the receiver is on localhost
    >> or over the wire.


    (Just wanted to correct myself and say the API *need not* change,
    rather than does not change... certainly some message passing
    APIs only work on the local machine, but others are network-
    transparent. :)

    >>
    >> Shared memory is nice for very high bandwidth between processes;
    >> say process A decodes a 300MB TIFF image into memory and wants
    >> to make the bitmap available to process B.

    >
    > And shared memory might also be nice if you don't want queueing, you
    > just want the latest data. Sensors in a real-time system, for example.


    Ah, yeah. Makes sense.

    Indeed, this can also apply over the wire; for example: dealing with
    a client->server->client remote mouse click-drag-release operation.
    (Where, for example, a client is dragging a scrollbar thumb where
    the UI is hosted on the server, and may be being rendered to multiple
    clients.) One might transmit the initial click coordinate over TCP,
    then the subsequent real-time drag telemetry over UDP, and finally the
    terminating click-release coordinate again over TCP. (Such that, TCP
    is used for reliable messages, UDP for unreliable streaming best-effort
    real-time telemetry where one only wants the latest data.)


    Regards,

    Bill
     
    Bill Kelly, Oct 12, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Knute Johnson
    Replies:
    0
    Views:
    837
    Knute Johnson
    Jun 27, 2003
  2. Jon A. Cruz
    Replies:
    0
    Views:
    746
    Jon A. Cruz
    Jun 28, 2003
  3. Sudsy
    Replies:
    0
    Views:
    974
    Sudsy
    Jun 28, 2003
  4. Frank D. Greco
    Replies:
    1
    Views:
    2,217
    Keeger
    Jun 30, 2003
  5. Justin Kaeser

    Inter-Process JNI

    Justin Kaeser, Nov 12, 2003, in forum: Java
    Replies:
    0
    Views:
    438
    Justin Kaeser
    Nov 12, 2003
Loading...

Share This Page