[ANN] Packet : A Pure Ruby Library for Event Driven NetworkProgramming

Discussion in 'Ruby' started by hemant, Nov 20, 2007.

  1. hemant

    hemant Guest

    Hi Folks,

    I am pleased to announce "Packet", A pure ruby library for Event
    Driven Network Programming.

    Packet is a pure ruby library for writing network applications in Ruby.
    It follows Evented Model of network programming and implements almost all the
    features provided by EventMachine.

    It also provides real easy to user UNIX workers for concurrent programming.

    Its best to have some examples going:

    == Examples
    === A Simple Echo Server:
    require "rubygems"
    require "packet"

    class Foo
    def receive_data p_data
    send_data(p_data)
    end

    def post_init
    puts "Client connected"
    end

    def connection_completed
    puts "Whoa man"
    end

    def unbind
    puts "Client Disconnected"
    end
    end

    Packet::Reactor.run do |t_reactor|
    t_reactor.start_server("localhost",11006,Foo)
    end

    Those new to network programming with events and callbacks, will note that,
    each time a new client connects an instance of class Foo is instantiated.
    When client writes some data to the socket, receive_data method is invoked.

    Although Packet implements an API similar to EventMachine, but it differs
    slightly because of the fact that, for a packet app, there can be more than one
    reactor loop running and hence, we don't use Packet.start_server(...).

    === A Simple Http Client
    class WikiHandler
    def receive_data p_data
    p p_data
    end

    def post_init
    end

    def unbind
    end

    def connection_completed
    send_data("GET / \r\n")
    end
    end

    Packet::Reactor.run do |t_reactor|
    t_reactor.connect("en.wikipedia.org",80,WikiHandler)
    end

    === Using Callbacks and Deferables
    Documentation to come.

    === Using Workers
    Packet enables you to write simple workers, which will run in
    different process and gives you nice
    evented handle for concurrent execution of various tasks.

    When, you are writing a scalable networking application
    using Event Model of network programming,
    sometimes when processing of certain events take time,
    your event loop is stuck there. With green
    threads, you don't really have a way of paralleling
    your request processing. Packet library, allows
    you to write simple workers, for executing long
    running tasks. You can pass data and callbacks as an
    argument.

    When you are going to use workers in
    your application, you need to define
    constant WORKER_ROOT,
    which is the directory location, where
    your workers are located. All the workers defined in that directory
    will be automatically, picked and forked in a
    new process when your packet app starts. So, a typical
    packet_app, that wants to use workers, will look like this:

    packet_app_root
    |
    |__ lib
    |
    |___ worker
    |
    |___ config
    |
    |___ log

    You would define WORKER_ROOT = PACKET_APP_ROOT/worker

    All the workers must inherit class Packet::Worker, and hence a
    general skeleton of worker will look like:

    class FooWorker < Packet::Worker
    set_worker_name :foo_worker #=> This is necessary.
    def receive_data p_data
    end

    def connection_completed
    end

    def unbind
    end

    def post_init
    end
    end

    All the forked workers are connected to master via
    UNIX sockets, and hence messages passed to workers from master
    will be available in receive_data method. Also,
    when you are passing messages to workers, or worker is passing
    message to master ( in a nutshell, all the internal
    communication between workers and master ) directly takes
    place using ruby objects. All the passed ruby objects are
    dumped and marshalled across unix sockets in a non blocking
    manner. BinParser class parses dumped binary objects and
    makes sure, packets received at other end are complete.
    Usually, you wouldn't need to worry about this little detail.

    Packet provides various ways of interacting with
    workers. Usually, when a worker is instantiated, a proxy for
    that worker will also be instantiated at master
    process. Packet automatically provides a worker proxy(See meta_pimp.rb)
    for you, but if you need to multiplex/demultiplex
    requests based on certain criteria, you may as well define your
    own worker proxies. Code, would like something like this:

    class FooWorker < Packet::Worker
    set_worker_proxy :foo_handler
    end

    When you define, :foo_handler as a proxy for
    this worker, packet is gonna search for FooHandler class and
    instantiate it when the worker gets started. All
    the worker proxies must inherit from Packet::pimp.
    Have a look at, Packet::MetaPimp,
    which acts as a meta pimp for all the workers,
    which don't have a explicit worker proxy defined.

    === A complete Case :

    Just for kicks, lets write a sample server,
    which evals whatever clients send to it. But, assuming this 'eval' of
    client data can be potentially time/cpu
    consuming ( not to mention dangerous too ), we are gonna ask our
    eval_worker, to
    perform eval and return the result to master process, which in
    turn returns the result to happy client.

    # APP_ROOT/bin/eval_server.rb
    EVAL_APP_ROOT = File.expand_path(File.join(File.dirname(__FILE__) + "/.."))
    ["bin","worker","lib"].each { |x| $LOAD_PATH.unshift(EVAL_APP_ROOT
    + "/#{x}")}
    WORKER_ROOT = EVAL_APP_ROOT + "/worker"

    require "packet"
    class EvalServer
    def receive_data p_data
    ask_worker:)eval_worker,:data => p_data, :type => :request)
    end

    # will be called, when any worker sends data back to master process
    # it should be noted that, you may have several instances of
    eval_server in
    # your master, for each connected client, but worker_receive
    will be always
    # be invoked for the instance, which originally made the request.
    # If you need fine control, over this behaviour, you can
    implement a worker proxy
    # on the lines of meta_pimp class. This API will change in
    future perhaps, as i
    # expect, better ideas to come.
    def worker_receive p_data
    send_data "#{p_data[:data]}\n"
    end

    def show_result p_data
    send_data("#{p_data[:response]}\n")
    end

    def connection_completed
    end

    def post_init
    end

    def wow
    puts "Wow"
    end
    end

    Packet::Reactor.run do |t_reactor|
    t_reactor.start_server("localhost", 11006,EvalServer) do |instance|
    instance.wow
    end
    end

    # APP_ROOT/worker/eval_worker.rb
    class EvalWorker < Packet::Worker
    set_worker_name :eval_worker
    def worker_init
    p "Starting no proxy worker"
    end

    def receive_data data_obj
    eval_data = eval(data_obj[:data])
    data_obj[:data] = eval_data
    data_obj[:type] = :response
    send_data(data_obj)
    end
    end

    === Passing file descriptors to workers :
    Packet, allows this style of load
    distribution as well, you can use method send_fd to
    pass file descriptors to workers.
    WARNING: This feature hasn't been tested well.

    === Disable auto loading of certain workers:
    Sometimes, you would need to start a
    worker at runtime and don't want this pre-forking mechanism.
    Packet, allows this. You just need to define
    "set_no_auto_load true" in your worker class and worker
    will not be automatically forked. Although name is a bit misleading perhaps.

    Now, at runtime, you can call start_worker:)foo_worker, options)
    to start a worker as usual. It should
    be noted that, forking a worker, which is already
    forked can be disastrous, since worker names are being
    used as unique keys that represent a worker.

    == Performance:
    Although written in pure ruby, packet performs
    reasonably well. Mongrel, running on top of Packet is a tad
    slower than Mongrel running on top of EventMachine. More benchmarks
    coming soon.

    == Code:
    SVN Code for packet is on google code, svn repo is:
    http://packet.googlecode.com/svn/trunk/

    Rubyforge URL:
    http://rubyforge.org/projects/packet/


    == Credits
    Francis for awesome EventMachine lib, which has constantly acted as
    an inspiration.
    Ezra, for being a early user and porting mongrel to run on top of packet.

    --
    Let them talk of their oriental summer climes of everlasting
    conservatories; give me the privilege of making my own summer with my
    own coals.

    http://gnufied.org
     
    hemant, Nov 20, 2007
    #1
    1. Advertising

  2. hemant

    Andre Nathan Guest

    On Wed, 2007-11-21 at 04:14 +0900, hemant wrote:
    > It also provides real easy to user UNIX workers for concurrent programming.


    Hemant,

    In the eval server example, is a single worker process created,
    corresponding to eval_worker.rb, or is it possible to create a pool of
    pre-forked worker processes of the same class?

    Andre
     
    Andre Nathan, Nov 20, 2007
    #2
    1. Advertising

  3. hemant

    hemant Guest

    On Nov 21, 2007 1:25 AM, Andre Nathan <> wrote:

    > In the eval server example, is a single worker process created,
    > corresponding to eval_worker.rb, or is it possible to create a pool of
    > pre-forked worker processes of the same class?
    >


    Andre,

    Currently, for each defined worker, only one process is created. You
    can't have a pool of workers instantiated from class.
    I am quite inclined to implement this feature and hence will be coming
    soon. If you can provide a patch, it would be awesome.




    --
    Let them talk of their oriental summer climes of everlasting
    conservatories; give me the privilege of making my own summer with my
    own coals.

    http://gnufied.org
     
    hemant, Nov 20, 2007
    #3
  4. hemant

    Chuck Remes Guest

    On Nov 20, 2007, at 1:14 PM, hemant wrote:

    > Hi Folks,
    >
    > I am pleased to announce "Packet", A pure ruby library for Event
    > Driven Network Programming.
    >
    > Packet is a pure ruby library for writing network applications in
    > Ruby.
    > It follows Evented Model of network programming and implements
    > almost all the
    > features provided by EventMachine.


    Did you ever consider working on the EventMachine project so it could
    handle your requirements? While I'm happy to see more event-driven
    programming, I'm puzzled why you re-invented so much of the wheel for
    one or two features that EM didn't have.
     
    Chuck Remes, Nov 21, 2007
    #4
  5. hemant

    Chris Carter Guest

    On Nov 20, 2007 6:09 PM, Chuck Remes <> wrote:
    >
    > On Nov 20, 2007, at 1:14 PM, hemant wrote:
    >
    > > Hi Folks,
    > >
    > > I am pleased to announce "Packet", A pure ruby library for Event
    > > Driven Network Programming.
    > >
    > > Packet is a pure ruby library for writing network applications in
    > > Ruby.
    > > It follows Evented Model of network programming and implements
    > > almost all the
    > > features provided by EventMachine.

    >
    > Did you ever consider working on the EventMachine project so it could
    > handle your requirements? While I'm happy to see more event-driven
    > programming, I'm puzzled why you re-invented so much of the wheel for
    > one or two features that EM didn't have.
    >
    >
    >


    I think mainly that this is pure ruby, and event machine is a C++ backend.

    --
    Chris Carter
    concentrationstudios.com
    brynmawrcs.com
     
    Chris Carter, Nov 21, 2007
    #5
  6. hemant

    hemant Guest

    On Nov 21, 2007 5:56 AM, Chris Carter <> wrote:
    >
    > On Nov 20, 2007 6:09 PM, Chuck Remes <> wrote:
    > >
    > > On Nov 20, 2007, at 1:14 PM, hemant wrote:
    > >
    > > > Hi Folks,
    > > >
    > > > I am pleased to announce "Packet", A pure ruby library for Event
    > > > Driven Network Programming.
    > > >
    > > > Packet is a pure ruby library for writing network applications in
    > > > Ruby.
    > > > It follows Evented Model of network programming and implements
    > > > almost all the
    > > > features provided by EventMachine.

    > >
    > > Did you ever consider working on the EventMachine project so it could
    > > handle your requirements? While I'm happy to see more event-driven
    > > programming, I'm puzzled why you re-invented so much of the wheel for
    > > one or two features that EM didn't have.
    > >
    > >
    > >

    >
    > I think mainly that this is pure ruby, and event machine is a C++ backend.
    >


    I would have liked nothing more than having it in EventMachine. But as
    you see the biggest differentiator is
    ability to have process based workers, so I needed low level access of
    sockets. Once started as proof of concept,
    it took off on its own, and I didn't see any harm in having a pure
    ruby version.


    --
    Let them talk of their oriental summer climes of everlasting
    conservatories; give me the privilege of making my own summer with my
    own coals.

    http://gnufied.org
     
    hemant, Nov 21, 2007
    #6
  7. Re: Packet : A Pure Ruby Library for Event Driven NetworkProgramming

    Hi.
    I was trying out "packet" tonight, but failed at the starting block with

    c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:133:in
    `start_reactor': uninitialized constant
    Packet::Core::CommonMethods::UNIXSocket (NameError)
    from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:132:in
    `each'
    from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:132:in
    `start_reactor'
    from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:127:in
    `loop'
    from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:127:in
    `start_reactor'
    from
    c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/packet_master.rb:16:in
    `run'
    from wikipacket.rb:23

    I groped around and it seems there is no UNIXSocket support in my Ruby
    installation (ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32])
    on Windows XP.
    Any idea what I'm doing wrong ?

    Tad

    --
    Posted via http://www.ruby-forum.com/.
     
    Tadeusz Bochan, Dec 4, 2007
    #7
  8. Re: Packet : A Pure Ruby Library for Event Driven NetworkProgramming

    On Dec 4, 2007, at 2:22 PM, Tadeusz Bochan wrote:

    > Hi.
    > I was trying out "packet" tonight, but failed at the starting block
    > with
    >
    > c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:133:in
    > `start_reactor': uninitialized constant
    > Packet::Core::CommonMethods::UNIXSocket (NameError)
    > from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:132:in
    > `each'
    > from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:132:in
    > `start_reactor'
    > from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:127:in
    > `loop'
    > from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:127:in
    > `start_reactor'
    > from
    > c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/packet_master.rb:16:in
    > `run'
    > from wikipacket.rb:23
    >
    > I groped around and it seems there is no UNIXSocket support in my Ruby
    > installation (ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32])
    > on Windows XP.
    > Any idea what I'm doing wrong ?
    >
    > Tad
    >
    > --
    > Posted via http://www.ruby-forum.com/.
    >




    Windows does not support UnixSockets. Therefor packet does not run on
    windows.

    Cheers-
    - Ezra Zygmuntowicz
    -- Founder & Software Architect
    --
    -- EngineYard.com
     
    Ezra Zygmuntowicz, Dec 4, 2007
    #8
  9. Re: Packet : A Pure Ruby Library for Event Driven Network Programming

    Ezra Zygmuntowicz wrote:
    > Windows does not support UnixSockets. Therefor packet does not run on
    > windows.


    It wouldn't be hard to use Named Pipes instead, which are the Windows equivalent.

    Clifford Heath.
     
    Clifford Heath, Dec 5, 2007
    #9
  10. hemant

    hemant Guest

    Re: Packet : A Pure Ruby Library for Event Driven NetworkProgramming

    On Dec 5, 2007 8:15 AM, Clifford Heath <> wrote:
    > Ezra Zygmuntowicz wrote:
    > > Windows does not support UnixSockets. Therefor packet does not run on
    > > windows.

    >
    > It wouldn't be hard to use Named Pipes instead, which are the Windows equivalent.
    >


    Never used them, but I will have a look at see, what can be done about
    Windows support.


    --
    Let them talk of their oriental summer climes of everlasting
    conservatories; give me the privilege of making my own summer with my
    own coals.

    http://gnufied.org
     
    hemant, Dec 5, 2007
    #10
  11. Re: Packet : A Pure Ruby Library for Event Driven NetworkProgramming

    Note: parts of this message were removed by the gateway to make it a legal Usenet post.

    On Dec 5, 2007 3:06 AM, hemant <> wrote:

    > On Dec 5, 2007 8:15 AM, Clifford Heath <> wrote:
    > > Ezra Zygmuntowicz wrote:
    > > > Windows does not support UnixSockets. Therefor packet does not

    > run on
    > > > windows.

    > >
    > > It wouldn't be hard to use Named Pipes instead, which are the Windows

    > equivalent.
    > >

    >
    > Never used them, but I will have a look at see, what can be done about
    > Windows support.
    >
    >
    > --
    > Let them talk of their oriental summer climes of everlasting
    > conservatories; give me the privilege of making my own summer with my
    > own coals.
    >
    > http://gnufied.org
    >
    >

    You do know that there is a maintained pure ruby version of EventMachine
    sitting right next to the C++ extension, right?

    Jason
     
    Jason Roelofs, Dec 5, 2007
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Charles
    Replies:
    11
    Views:
    722
    John Fereira
    Dec 31, 2004
  2. Li Han
    Replies:
    2
    Views:
    508
    bobicanprogram
    Feb 9, 2009
  3. Gelonida N
    Replies:
    4
    Views:
    888
    Gelonida N
    Sep 11, 2011
  4. hemant
    Replies:
    0
    Views:
    177
    hemant
    Feb 11, 2008
  5. hemant
    Replies:
    0
    Views:
    100
    hemant
    Jul 13, 2008
Loading...

Share This Page