[ANN] slave-1.1.0

Discussion in 'Ruby' started by Ara.T.Howard, Nov 28, 2006.

  1. Ara.T.Howard

    Ara.T.Howard Guest

    SYNOPSIS

    the Slave class forks a process and starts a drb server in the child using
    any object as the server. the process is detached so it is not required
    (nor possible) to wait on the child pid. a Heartbeat is set up between the
    parent and child processes so that the child will exit of the parent exits
    for any reason - preventing orphaned slaves from running indefinitely. the
    purpose of Slaves is to be able to easily set up a collection of objects
    communicating via drb protocols instead of having to use IPC.

    typical usage:

    slave = Slave::new{ AnyObject.new }

    slave.object #=> handle on drb object
    slave.uri #=> uri of the drb object
    slave.socket #=> unix domain socket path for drb object
    slave.psname #=> title shown in ps/top

    object = slave.object

    value = object.any_method #=> use the object normally

    slaves may be configured via the environment, the Slave class, or via the
    ctor for object itself. attributes which may be configured include

    * object : specify the slave object. otherwise block value is used.
    * socket_creation_attempts : specify how many attempts to create a unix domain socket will be made
    * debug : turn on some logging to STDERR
    * psname : specify the name that will appear in 'top' ($0)
    * at_exit : specify a lambda to be called in the *parent* when the child dies
    * dumped : specify that the slave object should *not* be DRbUndumped (default is DRbUndumped)
    * threadsafe : wrap the slave object with ThreadSafe to implement gross thread safety

    URIS

    http://rubyforge.org/projects/codeforpeople/
    http://codeforpeople.com/lib/ruby/slave

    HISTORY

    1.1.0:
    - replaced HeartBeat class with LifeLine.

    - __HUGE__ cleanup of file descriptor/fork management with tons of help
    from skaar and ezra. thanks guys!

    - introduced Slave.object method used to return any object directory from
    a child process. see samples/g.rb.

    - indroduced keyword to automatically make slave objects threadsafe.
    remember that your slave object must be threadsafe because they are
    being served via DRb!!!

    SAMPLES

    <========< samples/a.rb >========>

    ~ > cat samples/a.rb

    require 'slave'
    #
    # simple usage is simply to stand up a server object as a slave. you do not
    # need to wait for the server, join it, etc. it will die when the parent
    # process dies - even under 'kill -9' conditions
    #
    class Server
    def add_two n
    n + 2
    end
    end

    slave = Slave.new :eek:bject => Server.new

    server = slave.object
    p server.add_two(40) #=> 42

    slave.shutdown

    ~ > ruby samples/a.rb

    42


    <========< samples/b.rb >========>

    ~ > cat samples/b.rb

    require 'slave'
    #
    # if certain operations need to take place in the child only a block can be
    # used
    #
    class Server
    def connect_to_db
    "we only want to do this in the child process!"
    @connection = :postgresql
    end
    attr :connection
    end

    slave = Slave.new('object' => Server.new){|s| s.connect_to_db}

    server = slave.object

    p server.connection #=> :postgresql
    #
    # errors in the child are detected and raised in the parent
    #
    slave = Slave.new('object' => Server.new){|s| s.typo} #=> raises an error!

    ~ > ruby samples/b.rb

    :postgresql
    ./lib/slave.rb:460:in `initialize': undefined method `typo' for #<Server:0xb7565694> (NoMethodError)
    from samples/b.rb:22:in `new'
    from samples/b.rb:22


    <========< samples/c.rb >========>

    ~ > cat samples/c.rb

    require 'slave'
    #
    # if no slave object is given the block itself is used to contruct it
    #
    class Server
    def initialize
    "this is run only in the child"
    @pid = Process.pid
    end
    attr 'pid'
    end

    slave = Slave.new{ Server.new }
    server = slave.object

    p Process.pid
    p server.pid # not going to be the same as parents!
    #
    # errors are still detected though
    #
    slave = Slave.new{ fubar } # raises error in parent

    ~ > ruby samples/c.rb

    14387
    14388
    ./lib/slave.rb:460:in `initialize': undefined local variable or method `fubar' for main:Object (NameError)
    from samples/c.rb:21:in `new'
    from samples/c.rb:21


    <========< samples/d.rb >========>

    ~ > cat samples/d.rb

    require 'slave'
    #
    # at_exit hanlders are handled correctly in both child and parent
    #
    at_exit{ p 'parent' }
    slave = Slave.new{ at_exit{ p 'child' }; 'the server is this string' }
    #
    # this will print 'child', then 'parent'
    #

    ~ > ruby samples/d.rb

    "child"
    "parent"


    <========< samples/e.rb >========>

    ~ > cat samples/e.rb

    require 'slave'
    #
    # slaves never outlive their parent. if the parent exits, even under kill -9,
    # the child will die.
    #
    slave = Slave.new{ at_exit{ p 'child' }; 'the server is this string' }

    Process.kill brutal=9, the_parent_pid=Process.pid
    #
    # even though parent dies a nasty death the child will still print 'child'
    #

    ~ > ruby samples/e.rb

    "child"


    <========< samples/f.rb >========>

    ~ > cat samples/f.rb

    require 'slave'
    #
    # slaves created previously are visible to newly created slaves - in this
    # example the child process of slave_a communicates directly with the child
    # process of slave_a
    #
    slave_a = Slave.new{ Array.new }
    slave_b = Slave.new{ slave_a.object }

    a, b = slave_b.object, slave_a.object

    b << 42
    puts a #=> 42

    ~ > ruby samples/f.rb

    42


    <========< samples/g.rb >========>

    ~ > cat samples/g.rb

    require 'slave'
    #
    # Slave.object can used when you want to construct an object in another
    # process. in otherwords you want to fork a process and retrieve a single
    # returned object from that process as opposed to setting up a server.
    #
    this = Process.pid
    that = Slave.object{ Process.pid }

    p 'this' => this, 'that' => that

    #
    # any object can be returned and it can be returned asychronously via a thread
    #
    thread = Slave.object:)async => true){ sleep 2 and [ Process.pid, Time.now ] }
    this = [ Process.pid, Time.now ]
    that = thread.value

    p 'this' => this, 'that' => that

    ~ > ruby samples/g.rb

    {"that"=>14406, "this"=>14405}
    {"that"=>[14407, Tue Nov 28 09:47:31 MST 2006], "this"=>[14405, Tue Nov 28 09:47:29 MST 2006]}


    enjoy.

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    Ara.T.Howard, Nov 28, 2006
    #1
    1. Advertising

  2. Ara.T.Howard

    Tim Pease Guest

    On 11/28/06, Ara.T.Howard <> wrote:
    >
    >
    > SYNOPSIS
    >
    > the Slave class forks a process and starts a drb server in the child using
    > any object as the server. the process is detached so it is not required
    > (nor possible) to wait on the child pid. a Heartbeat is set up between the
    > parent and child processes so that the child will exit of the parent exits
    > for any reason - preventing orphaned slaves from running indefinitely. the
    > purpose of Slaves is to be able to easily set up a collection of objects
    > communicating via drb protocols instead of having to use IPC.
    >


    Does Slave work on the Win32 platform, or is this a UNIX only gem?

    TwP
     
    Tim Pease, Nov 28, 2006
    #2
    1. Advertising

  3. Ara.T.Howard

    Guest

    On Nov 28, 2006, at 1:49 PM, Tim Pease wrote:

    > On 11/28/06, Ara.T.Howard <> wrote:
    >>
    >>
    >> SYNOPSIS
    >>
    >> the Slave class forks a process and starts a drb server in the
    >> child using
    >> any object as the server. the process is detached so it is not
    >> required
    >> (nor possible) to wait on the child pid. a Heartbeat is set up
    >> between the
    >> parent and child processes so that the child will exit of the
    >> parent exits
    >> for any reason - preventing orphaned slaves from running
    >> indefinitely. the
    >> purpose of Slaves is to be able to easily set up a collection
    >> of objects
    >> communicating via drb protocols instead of having to use IPC.
    >>

    >
    > Does Slave work on the Win32 platform, or is this a UNIX only gem?


    Slave is UNIX only since it relies on fork(). I've looked at the
    code. Ara does a nice job of keeping the fork stuff encapsulated. If
    you have a Win32 mechanism that could be bolted on to Slave and
    replace fork(), you'd get the rest of the code for free.

    cr
     
    , Nov 28, 2006
    #3
  4. Ara.T.Howard

    Guest

    On Wed, 29 Nov 2006, Tim Pease wrote:

    > On 11/28/06, Ara.T.Howard <> wrote:
    >>
    >>
    >> SYNOPSIS
    >>
    >> the Slave class forks a process and starts a drb server in the child
    >> using
    >> any object as the server. the process is detached so it is not required
    >> (nor possible) to wait on the child pid. a Heartbeat is set up between
    >> the
    >> parent and child processes so that the child will exit of the parent
    >> exits
    >> for any reason - preventing orphaned slaves from running indefinitely.
    >> the
    >> purpose of Slaves is to be able to easily set up a collection of objects
    >> communicating via drb protocols instead of having to use IPC.
    >>

    >
    > Does Slave work on the Win32 platform, or is this a UNIX only gem?


    what is this "windows" you speak of?

    ;-)

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    , Nov 28, 2006
    #4
  5. Ara.T.Howard

    Guest

    On Wed, 29 Nov 2006, Tim Pease wrote:

    > On 11/28/06, Ara.T.Howard <> wrote:
    >>
    >>
    >> SYNOPSIS
    >>
    >> the Slave class forks a process and starts a drb server in the child
    >> using
    >> any object as the server. the process is detached so it is not required
    >> (nor possible) to wait on the child pid. a Heartbeat is set up between
    >> the
    >> parent and child processes so that the child will exit of the parent
    >> exits
    >> for any reason - preventing orphaned slaves from running indefinitely.
    >> the
    >> purpose of Slaves is to be able to easily set up a collection of objects
    >> communicating via drb protocols instead of having to use IPC.
    >>

    >
    > Does Slave work on the Win32 platform, or is this a UNIX only gem?


    in all seriousness i'd be happy if it could be made to work - have a look and
    let me know what you think if you get a chance.

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    , Nov 28, 2006
    #5
  6. Ara.T.Howard

    Tim Pease Guest

    On 11/28/06, <> wrote:
    > On Wed, 29 Nov 2006, Tim Pease wrote:
    >
    > > Does Slave work on the Win32 platform, or is this a UNIX only gem?

    >
    > in all seriousness i'd be happy if it could be made to work - have a look and
    > let me know what you think if you get a chance.
    >


    I think Daniel Berger is your man for getting this to work on Windows.
    Just one of those things that would be "nice to have" in the future.

    TwP
     
    Tim Pease, Nov 28, 2006
    #6
  7. Re: slave-1.1.0

    Tim Pease wrote:
    > On 11/28/06, <> wrote:
    > > On Wed, 29 Nov 2006, Tim Pease wrote:
    > >
    > > > Does Slave work on the Win32 platform, or is this a UNIX only gem?

    > >
    > > in all seriousness i'd be happy if it could be made to work - have a look and
    > > let me know what you think if you get a chance.
    > >

    >
    > I think Daniel Berger is your man for getting this to work on Windows.
    > Just one of those things that would be "nice to have" in the future.


    It's probably doable, but I'll have to look at the source code.

    I don't know much about slave, but one question I have right off the
    bat is why you would want one drb server per object, instead of just
    one drb server that stored objects in a hash table that you could push
    and pop at will. Maybe I need to go back and see Ara's use case for
    this...

    Regards,

    Dan
     
    Daniel Berger, Nov 28, 2006
    #7
  8. Ara.T.Howard

    Tim Pease Guest

    Re: slave-1.1.0

    On 11/28/06, Daniel Berger <> wrote:
    >
    > Tim Pease wrote:
    > > On 11/28/06, <> wrote:
    > > > On Wed, 29 Nov 2006, Tim Pease wrote:
    > > >
    > > > > Does Slave work on the Win32 platform, or is this a UNIX only gem?
    > > >
    > > > in all seriousness i'd be happy if it could be made to work - have a look and
    > > > let me know what you think if you get a chance.
    > > >

    > >
    > > I think Daniel Berger is your man for getting this to work on Windows.
    > > Just one of those things that would be "nice to have" in the future.

    >
    > It's probably doable, but I'll have to look at the source code.
    >
    > I don't know much about slave, but one question I have right off the
    > bat is why you would want one drb server per object, instead of just
    > one drb server that stored objects in a hash table that you could push
    > and pop at will. Maybe I need to go back and see Ara's use case for
    > this...
    >


    Slave forks off a ruby object in a separate process and then uses DRb
    for communication between the processes. Each forked process has its
    own DRb server.

    TwP
     
    Tim Pease, Nov 28, 2006
    #8
  9. Re: slave-1.1.0

    Tim Pease wrote:

    <snip>

    > > I don't know much about slave, but one question I have right off the
    > > bat is why you would want one drb server per object, instead of just
    > > one drb server that stored objects in a hash table that you could push
    > > and pop at will. Maybe I need to go back and see Ara's use case for
    > > this...
    > >

    >
    > Slave forks off a ruby object in a separate process and then uses DRb
    > for communication between the processes. Each forked process has its
    > own DRb server.
    >
    > TwP


    Yes, but why? I assume the purpose is IPC. If so, again I ask why you
    would need 1 drb server per object instead of one drb server with lots
    of objects. If its purspose isn't IPC, what is it?

    Thanks,

    Dan
     
    Daniel Berger, Nov 28, 2006
    #9
  10. Ara.T.Howard

    Guest

    Re: slave-1.1.0

    On Wed, 29 Nov 2006, Daniel Berger wrote:

    > Tim Pease wrote:
    >
    > <snip>
    >
    >>> I don't know much about slave, but one question I have right off the
    >>> bat is why you would want one drb server per object, instead of just
    >>> one drb server that stored objects in a hash table that you could push
    >>> and pop at will. Maybe I need to go back and see Ara's use case for
    >>> this...
    >>>

    >>
    >> Slave forks off a ruby object in a separate process and then uses DRb
    >> for communication between the processes. Each forked process has its
    >> own DRb server.
    >>
    >> TwP

    >
    > Yes, but why? I assume the purpose is IPC. If so, again I ask why you
    > would need 1 drb server per object instead of one drb server with lots
    > of objects. If its purspose isn't IPC, what is it?


    the purpose is two-fold:

    1) easy ipc
    2) scalling expensive operations across multiple processors easily

    you certainly could setup a hashtable of objects. but a few things to
    consider

    1) drop-in scalibility

    object = ExpensiveCalcuation.new

    # 1000 lines of code

    change to

    object = Slave.new{ ExpensiveCalcuation.new }

    # exact same 1000 lines of code

    and, whammo. you scale to multi-processor machines.


    2) DRbUndumped nuances

    let's say you build a table of objects like this:


    harp:~ > cat a.rb
    require "slave"
    require "yaml"

    slave = Slave.new{ Hash.new }
    table = slave.object

    array = []

    table["array"] = array

    table["array"] << Process.pid

    y "slave.pid" => slave.pid
    y "array" => array
    y "table['array']" => table["array"]


    what will happen?


    harp:~ > ruby a.rb
    slave.pid: 5258
    array: []
    table['array']: []


    yikes!!! what went wrong? well, consider:

    array = [] # array is in parent

    table["array"] = array # array is marshaled across to child as a copy

    table["array"] << Process.pid # array is marshaled back across to parent and this __local_copy__ is appended too

    p "array" => array # local copy empty

    p "table['hash']" => table["hash"] # remote copy empty


    here is a fix

    array = []
    array.extend DRbUndumped # the important part


    now, running again


    harp:~ > ruby a.rb
    slave.pid: 5302

    array:
    - 5301

    table['array']:
    - 5301


    but look carefully this example is even __more__ fubar: the array lives in
    the parent! we setup a table and sent only the proxy (DRbUndumped) across
    to the child! we've completely negated the entire benefit of running
    objects in another process - all we have is a lookup table for objects in
    our own address space.


    ok. take two. this time we'll try this:

    harp:~ > cat a.rb
    require "slave"
    require "yaml"

    slave = Slave.new{ # build table in child
    table = Hash.new
    array = []
    table["array"] = array
    table
    }

    table = slave.object

    table["array"] << Process.pid

    y "slave.pid" => slave.pid
    y "table['array']" => table["array"]


    we give a whirl:


    harp:~ > ruby a.rb
    slave.pid: 5358
    table['array']: []


    sigh. now our table exists only in the child, and so does all it's
    elements. why doesn't it work. DRbUndumped is screwing us again. here's
    why:


    table["array"] << Process.pid # here the child marshals a copy back to
    # parent. we append to the copy

    y "table['array']" => table["array"] # and here we get a fresh __empty__ copy


    dang. we know the fix though, in the child we do:

    array.extend DRbUndumped


    trying again

    harp:~ > ruby a.rb
    slave.pid: 5405
    table['array']: !ruby/object:DRb::DRbObject
    ref: -609755446
    uri: drbunix:///tmp/hash_-609755436_5404_5405_0

    wtf? well, since our object now lives in the child and we only get a drb
    proxy back some things work, like '<<', but other methods, like 'to_yaml', and
    'inspect' are handled locally by drb. to get around this 'to_yaml' bug we have
    to force a copy


    harp:~ > cat a.rb
    require "slave"
    require "yaml"

    slave = Slave.new{ # build table in child
    table = Hash.new
    array = []
    array.extend DRbUndumped
    table["array"] = array
    table
    }

    table = slave.object

    table["array"] << Process.pid

    y "slave.pid" => slave.pid
    y "table['array']" => table["array"].map # force local copy, neither dup nor clone will cut it!



    harp:~ > ruby a.rb
    slave.pid: 5465
    table['array']:
    - 5464


    ah, finally.


    so, you see, managing a hierarchy of object across drb is tricky. not that it
    can't be done - i just happen to think that managing __one__ object and it's
    DRbUndumped nuances is about all most hackers can manage - thus that's the
    default usage. nothing will stop you from making your server object a factory
    or table though - it just gets confusing fast.


    kind regards.

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    , Nov 28, 2006
    #10
  11. Ara.T.Howard

    Bil Kleb Guest

    wrote:
    > On Wed, 29 Nov 2006, Tim Pease wrote:
    >> Does Slave work on the Win32 platform, or is this a UNIX only gem?

    >
    > what is this "windows" you speak of?


    I think it might be what these GNU folks are talking about
    in their coding standards document,

    If you do support Windows, please do not abbreviate it as “win”.
    In hacker terminology, calling something a “win” is a form of
    praise. You're free to praise Microsoft Windows on your own
    if you want, but please don't do this in GNU packages. Instead
    of abbreviating “Windows” to “un”, you can write it in full or
    abbreviate it to “woe” or “w”. In GNU Emacs, for instance, we
    use ‘w32’ in file names of Windows-specific files, but the macro
    for Windows conditionals is called WINDOWSNT.

    http://www.gnu.org/prep/standards/standards.html#System-Portability

    Later,
    --
    Bil Kleb
    http://fun3d.larc.nasa.gov
     
    Bil Kleb, Nov 29, 2006
    #11
  12. Ara.T.Howard

    Guest

    On Thu, 30 Nov 2006, Bil Kleb wrote:

    > wrote:
    >> On Wed, 29 Nov 2006, Tim Pease wrote:
    >>> Does Slave work on the Win32 platform, or is this a UNIX only gem?

    >>
    >> what is this "windows" you speak of?

    >
    > I think it might be what these GNU folks are talking about
    > in their coding standards document,
    >
    > If you do support Windows, please do not abbreviate it as “win”.
    > In hacker terminology, calling something a “win” is a form of
    > praise. You're free to praise Microsoft Windows on your own
    > if you want, but please don't do this in GNU packages. Instead
    > of abbreviating “Windows” to “un”, you can write it in full or
    > abbreviate it to “woe” or “w”. In GNU Emacs, for instance, we
    > use ‘w32’ in file names of Windows-specific files, but the macro
    > for Windows conditionals is called WINDOWSNT.
    >
    > http://www.gnu.org/prep/standards/standards.html#System-Portability


    that's __awesome__. more humour in docs i say!

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    , Nov 29, 2006
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Colin Jackson

    VHDL for FPGA VME Slave

    Colin Jackson, Aug 15, 2003, in forum: VHDL
    Replies:
    2
    Views:
    4,953
    Tullio Grassi
    Aug 19, 2003
  2. [ANN] slave-0.2.0

    , Sep 23, 2006, in forum: Ruby
    Replies:
    3
    Views:
    129
    Ezra Zygmuntowicz
    Sep 24, 2006
  3. [ANN] slave-1.0.0

    , Oct 13, 2006, in forum: Ruby
    Replies:
    2
    Views:
    132
  4. [ANN] slave-1.2.0.rb

    , Dec 8, 2006, in forum: Ruby
    Replies:
    0
    Views:
    152
  5. Ara.T.Howard

    [ANN] slave-1.2.1

    Ara.T.Howard, Apr 27, 2007, in forum: Ruby
    Replies:
    1
    Views:
    113
    John Stoffel
    Feb 17, 2009
Loading...

Share This Page