event driven framework for ruby

Discussion in 'Ruby' started by snacktime, Aug 9, 2005.

  1. snacktime

    snacktime Guest

    Is there an event driven framework for ruby? Something similar to POE
    in perl or twisted in python?

    Chris
     
    snacktime, Aug 9, 2005
    #1
    1. Advertising

  2. snacktime

    Guest

    Actually, yes. I'm working on the Ruby/Event library. It's a C extensio=
    n
    for libevent. I built a small framework on top of this called Myriad
    which is similar to twisted and POE. I used Myriad to write the SCGI
    client/server which can host Ruby on Rails applications.

    The framework is still pretty rough, but you can see the entire code for
    the SCGI server at:

    http://phpfi.com/73376

    Also the performance so far is pretty damn good. I'll be posting some
    metrics soon for it.

    You can download the last version from:

    http://www.zedshaw.com/

    I'm going to release a new version this week which fixes some memory
    issues and has better documentation.

    Feedback is welcome.

    Zed

    > Is there an event driven framework for ruby? Something similar to POE
    > in perl or twisted in python?
    >
    > Chris
    >
    >
     
    , Aug 9, 2005
    #2
    1. Advertising

  3. On Aug 9, 2005, at 12:23 PM, wrote:

    > Actually, yes. I'm working on the Ruby/Event library. It's a C
    > extension
    > for libevent. I built a small framework on top of this called Myriad
    > which is similar to twisted and POE. I used Myriad to write the SCGI
    > client/server which can host Ruby on Rails applications.


    Does this do all it's socket work in C, to bypass Ruby's Thread
    blocking issues?

    James Edward Gray II
     
    James Edward Gray II, Aug 9, 2005
    #3
  4. Hey Zed-
    I have been using and abusing you scgi server with lightppd
    1.3.15 for the last couple of weeks. It works great so far and
    performs very well. The few problems that I have had are as follows:
    With lighttpd in production mode it fails with an ENOENT error when
    caching is turned on. It seems that it is having problems writing the
    cache files. If I restart the same exact app with webrick or plain
    lighttpd/fcgi the caching works fine and my permission setup is OK.
    Also I have tried to set scgi up with apachge 1.3.33. Everything
    seems to work except the round trip back to apache. I mean I can
    request different URL's and I can see them get rendered in the
    develpment and production logs. But in the browser I just get an
    internal server error for any and all pages.

    I really like how easy it was to set up and manage though. Its
    great to not have to restart the webserver to restart my app. I'd
    love to be able to use this in production but the production caching
    issues are the main hold up right now. If you want any stack traces
    or log files form some of these errors let me know and I am more than
    happy to be a tester and troubleshooter. I would rerally like to see
    this project get up and running.

    Thanks for your hard work Zed-
    -Ezra Zygmuntowicz
    Yakima Herald-Republic
    WebMaster
    509-577-7732



    On Aug 9, 2005, at 10:23 AM, wrote:

    > Actually, yes. I'm working on the Ruby/Event library. It's a C
    > extension
    > for libevent. I built a small framework on top of this called Myriad
    > which is similar to twisted and POE. I used Myriad to write the SCGI
    > client/server which can host Ruby on Rails applications.
    >
    > The framework is still pretty rough, but you can see the entire
    > code for
    > the SCGI server at:
    >
    > http://phpfi.com/73376
    >
    > Also the performance so far is pretty damn good. I'll be posting some
    > metrics soon for it.
    >
    > You can download the last version from:
    >
    > http://www.zedshaw.com/
    >
    > I'm going to release a new version this week which fixes some memory
    > issues and has better documentation.
    >
    > Feedback is welcome.
    >
    > Zed
    >
    >
    >> Is there an event driven framework for ruby? Something similar to
    >> POE
    >> in perl or twisted in python?
    >>
    >> Chris
    >>
    >>
    >>

    >
    >
    >
     
    Ezra Zygmuntowicz, Aug 9, 2005
    #4
  5. James Edward Gray II <> writes:

    > On Aug 9, 2005, at 12:23 PM, wrote:
    >
    >> Actually, yes. I'm working on the Ruby/Event library. It's a C
    >> extension
    >> for libevent. I built a small framework on top of this called Myriad
    >> which is similar to twisted and POE. I used Myriad to write the SCGI
    >> client/server which can host Ruby on Rails applications.

    >
    > Does this do all it's socket work in C, to bypass Ruby's Thread
    > blocking issues?


    What blocking issue?

    In any case, ruby itself uses bsd's select which, in most
    implementations, degrades linearly wrt number of sockets to be
    monitored.

    libevent allows you to take advantage of OS-specific event monitoring
    interface that has better degradation characteristics than select()
    under high load.

    YS.
     
    Yohanes Santoso, Aug 9, 2005
    #5
  6. On Aug 9, 2005, at 12:57 PM, Yohanes Santoso wrote:

    > James Edward Gray II <> writes:
    >
    >
    >> Does this do all it's socket work in C, to bypass Ruby's Thread
    >> blocking issues?
    >>

    >
    > What blocking issue?


    Writing a large chunk of data to a file, for example. Ruby will
    block until the write completes.

    James Edward Gray II
     
    James Edward Gray II, Aug 9, 2005
    #6
  7. James Edward Gray II <> writes:

    > On Aug 9, 2005, at 12:57 PM, Yohanes Santoso wrote:
    >
    >> James Edward Gray II <> writes:
    >>
    >>
    >>> Does this do all it's socket work in C, to bypass Ruby's Thread
    >>> blocking issues?
    >>>

    >>
    >> What blocking issue?

    >
    > Writing a large chunk of data to a file, for example. Ruby will
    > block until the write completes.



    Disk I/O will always block the process executing it. only way to avoid
    that is to have another process do the disk I/O (at least on linux,
    *bsd; there used to have a unix that does not have this limitation,
    but i can't remember the name). Using native thread also does not help
    since the whole process is blocked.


    YS.
     
    Yohanes Santoso, Aug 9, 2005
    #7
  8. On Aug 9, 2005, at 2:22 PM, Yohanes Santoso wrote:

    > Disk I/O will always block the process executing it. only way to avoid
    > that is to have another process do the disk I/O (at least on linux,
    > *bsd; there used to have a unix that does not have this limitation,
    > but i can't remember the name). Using native thread also does not help
    > since the whole process is blocked.


    Non-blocking IO is another solution.

    James Edward Gray II
     
    James Edward Gray II, Aug 9, 2005
    #8
  9. snacktime

    rcoder Guest

    Actually, the entire point of "native" threading is to allow blocking
    actions to be performed in one thread while another runs -- hence their
    usual applications in user interfaces which must stay responsive while
    some blocking operation is happening, or in network servers where each
    client is assigned a thread, so that socket reads and writes don't stop
    the entire process.

    It's true that Linux and *BSD did not always have the best support for
    native threading, but at this point all the major distributions and
    variants seem to have pretty much figured it out. However, since the
    threading model was inconsistent from system to system, many developers
    simply used the MIT pthreads implementation, which was a purely
    user-space implementation, with the same limitations as Ruby's internal
    threading model.

    -Lennon
     
    rcoder, Aug 9, 2005
    #9
  10. James Edward Gray II <> writes:

    > On Aug 9, 2005, at 2:22 PM, Yohanes Santoso wrote:
    >
    >> Disk I/O will always block the process executing it. only way to avoid
    >> that is to have another process do the disk I/O (at least on linux,
    >> *bsd; there used to have a unix that does not have this limitation,
    >> but i can't remember the name). Using native thread also does not help
    >> since the whole process is blocked.

    >
    > Non-blocking IO is another solution.
    >
    > James Edward Gray II


    I know no POSIX system that honours O_NONBLOCK for disk I/O. That
    includes linux, *bsd.

    OTOH, AsyncIO is supposed to be a solution to this problem. But except
    for win32, AsyncIO support on posix systems is spotty.

    YS.
     
    Yohanes Santoso, Aug 9, 2005
    #10
  11. snacktime

    Guest

    Depending on the libevent API used, all of the actual socket
    reading/writing is either done with Ruby's Socket API, or it's done by
    libevent. The only additional bit of "magic" I do is to turn off
    blocking. Incidentally, none of this works with $stdout, $stdin, $stderr
    and probably doesn't work with files. Most likely that's because Ruby is
    messing with things it shouldn't be messing with <ehem>.

    This all works because libevent only deals with the OS level event API an=
    d
    the direct socket file descriptor (Socket.fileno). So, with non-blocking
    sockets combined with libevent, we only access the Ruby Sockets API when
    there's actually something to read.

    Additionally, Myriad is layered on top of the libevent bufferevent
    structure, so libevent actually does *all* of the IO and we just have to
    read chunks of memory data. This gets around a ton of performance
    bottle-necks and interference from Ruby Sockets.

    I'll hopefully have a more detailed explanation when I release later this
    week.

    Zed A. Shaw
    http://www.zedshaw.com/

    > On Aug 9, 2005, at 12:23 PM, wrote:
    >
    >> Actually, yes. I'm working on the Ruby/Event library. It's a C
    >> extension
    >> for libevent. I built a small framework on top of this called Myriad
    >> which is similar to twisted and POE. I used Myriad to write the SCGI
    >> client/server which can host Ruby on Rails applications.

    >
    > Does this do all it's socket work in C, to bypass Ruby's Thread
    > blocking issues?
    >
    > James Edward Gray II
    >
    >
    >
     
    , Aug 9, 2005
    #11
  12. snacktime

    Guest

    Hey Ezra, this stuff is actually still in my queue of things to get done.=
    =20
    The ENOENT is bizarre, but if you have traces for it then send them on.=20
    The Apache problem is mostly because I haven't pulled up the courage yet
    to don the S&M suit and fight the Apache config file monster. :)

    Keep bugging me though.

    > Hey Zed-
    > I have been using and abusing you scgi server with lightppd
    > 1.3.15 for the last couple of weeks. It works great so far and
    > performs very well. The few problems that I have had are as follows:
    > With lighttpd in production mode it fails with an ENOENT error when
    > caching is turned on. It seems that it is having problems writing the
    > cache files. If I restart the same exact app with webrick or plain
    > lighttpd/fcgi the caching works fine and my permission setup is OK.
    > Also I have tried to set scgi up with apachge 1.3.33. Everything
    > seems to work except the round trip back to apache. I mean I can
    > request different URL's and I can see them get rendered in the
    > develpment and production logs. But in the browser I just get an
    > internal server error for any and all pages.
    >
    > I really like how easy it was to set up and manage though. Its
    > great to not have to restart the webserver to restart my app. I'd
    > love to be able to use this in production but the production caching
    > issues are the main hold up right now. If you want any stack traces
    > or log files form some of these errors let me know and I am more than
    > happy to be a tester and troubleshooter. I would rerally like to see
    > this project get up and running.
    >
    > Thanks for your hard work Zed-
    > -Ezra Zygmuntowicz
    > Yakima Herald-Republic
    > WebMaster
    > 509-577-7732
    >
    >
    >
     
    , Aug 9, 2005
    #12
  13. Hello Yohanes,

    YS> James Edward Gray II <> writes:

    >> On Aug 9, 2005, at 2:22 PM, Yohanes Santoso wrote:
    >>
    >>> Disk I/O will always block the process executing it. only way to avoid
    >>> that is to have another process do the disk I/O (at least on linux,
    >>> *bsd; there used to have a unix that does not have this limitation,
    >>> but i can't remember the name). Using native thread also does not help
    >>> since the whole process is blocked.

    >>
    >> Non-blocking IO is another solution.
    >>
    >> James Edward Gray II


    YS> I know no POSIX system that honours O_NONBLOCK for disk I/O. That
    YS> includes linux, *bsd.

    YS> OTOH, AsyncIO is supposed to be a solution to this problem. But except
    YS> for win32, AsyncIO support on posix systems is spotty.

    It should work on linux even if the linux implementation is done
    really bad by starting one thread for each async operation (using a
    thread pool to speed this a little bit up) - at least this was how it
    was implemented a few years ago. But the API is there, stable and
    an offical POSIX conform standard.

    --
    Best regards, emailto: scholz at scriptolutions dot com
    Lothar Scholz http://www.ruby-ide.com
    CTO Scriptolutions Ruby, PHP, Python IDE 's
     
    Lothar Scholz, Aug 9, 2005
    #13
  14. On Aug 9, 2005, at 4:55 PM, wrote:

    > Additionally, Myriad is layered on top of the libevent bufferevent
    > structure, so libevent actually does *all* of the IO and we just
    > have to
    > read chunks of memory data. This gets around a ton of performance
    > bottle-necks and interference from Ruby Sockets.
    >
    > I'll hopefully have a more detailed explanation when I release
    > later this
    > week.


    This project interests me a great deal! Thanks for taking the time
    to explain these details. I'll watch for the promised documentation
    and dig a little deeper at that time...

    James Edward Gray II
     
    James Edward Gray II, Aug 9, 2005
    #14
  15. snacktime

    Tanaka Akira Guest

    In article <>,
    snacktime <> writes:

    > Is there an event driven framework for ruby? Something similar to POE
    > in perl or twisted in python?


    Why do you need an event driven framework?

    Ruby has its own event driven mechanism to implement threads.

    So, the thread mechanism should provide I/O multiplexing behaviour
    similar to other event driven frameworks.

    If you want event driven programming *style*, it doesn't help you,
    though.
    --
    Tanaka Akira
     
    Tanaka Akira, Aug 10, 2005
    #15
  16. snacktime

    Tanaka Akira Guest

    In article <>,
    James Edward Gray II <> writes:

    > Non-blocking IO is another solution.


    O_NONBLOCK is not always applicable.

    * read from socket: works well by default. (O_NONBLOCK is not required.)
    (UDP packet with wrong checksum on Linux 2.6 is an exception.)
    * write to socket: O_NONBLOCK is required.
    * read from pipe, console (and devices): works well by default on Unix.
    may not work on other platforms.
    * write to pipe, console (and devices): O_NONBLOCK is required on Unix.
    may not work on other platforms.
    * read from file: AIO or native thread is required. Ruby doesn't support yet.
    * write to file: AIO or native thread is required. Ruby doesn't support yet.
    * connect: works well by default. (Ruby use O_NONBLOCK internally.)
    * accept: works well by default. (Old BSD platform require O_NONBLOCK?)

    Note that Ruby 1.8.2 or former may lost data if O_NONBLOCK is set.
    It is fixed by Ruby 1.9. Ruby 1.8.3 can avoid the problem by
    IO#sync=true which is default for sockets.

    Also note that some methods behave differently when O_NONBLOCK is
    set. So O_NONBLOCK should be used carefully. I think O_NONBLOCK can
    be usable more easily if Ruby provides blocking methods and
    nonblocking methods. I proposed a method for nonblocking connect on
    ruby-dev but matz doesn't accept because its name is not good enough,
    though.
    --
    Tanaka Akira
     
    Tanaka Akira, Aug 10, 2005
    #16
  17. Tanaka Akira wrote:
    > In article <>,
    > James Edward Gray II <> writes:
    >
    >
    >>Non-blocking IO is another solution.

    >
    >
    > O_NONBLOCK is not always applicable.
    >
    > * read from socket: works well by default. (O_NONBLOCK is not required.)
    > (UDP packet with wrong checksum on Linux 2.6 is an exception.)


    Can you say a bit more about what happens differently on Linux 2.6 when
    a UDP packet has a bad checksum? Is it a ruby behavior or a linux behavior?

    (I'm working with wireless UDP on linux 2.6, having migrated from QNX,
    so it sounds important to me...)

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Aug 10, 2005
    #17
  18. snacktime

    Tanaka Akira Guest

    In article <>,
    Joel VanderWerf <> writes:

    >> * read from socket: works well by default. (O_NONBLOCK is not required.)
    >> (UDP packet with wrong checksum on Linux 2.6 is an exception.)

    >
    > Can you say a bit more about what happens differently on Linux 2.6 when
    > a UDP packet has a bad checksum? Is it a ruby behavior or a linux behavior?
    >
    > (I'm working with wireless UDP on linux 2.6, having migrated from QNX,
    > so it sounds important to me...)


    Linux 2.6's select notify readability by a UDP packet with a bad checksum.
    But read discards it and waits next packet with a correct checksum if
    O_NONBLOCK is clear. If the next packet is not available, the read
    hangs. Since O_NONBLOCK avoids the hang, it is possible to care the
    problem in application level.

    Linux-Kernel Archive: UDP recvmsg blocks after select(), 2.6 bug?
    http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1372.html

    Debian Bug report logs - #275585 - /usr/sbin/inetd: UDP builtins can be used to hang inetd
    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=275585&archive=yes

    select(2):
    Under Linux, select may report a socket file descriptor as "ready for
    reading", while nevertheless a subsequent read blocks. This could for
    example happen when data has arrived but upon examination has wrong
    checksum and is discarded. There may be other circumstances. Thus it
    may be safer to use O_NONBLOCK on sockets that should not block.

    [ruby-talk:148436] Nonblocking Sockets
    --
    Tanaka Akira
     
    Tanaka Akira, Aug 10, 2005
    #18
  19. snacktime

    Tanaka Akira Guest

    In article <-a-geek.org>,
    Yohanes Santoso <-a-geek.org> writes:

    > Disk I/O will always block the process executing it. only way to avoid
    > that is to have another process do the disk I/O (at least on linux,
    > *bsd; there used to have a unix that does not have this limitation,
    > but i can't remember the name). Using native thread also does not help
    > since the whole process is blocked.


    I think native (kernel level) thread doesn't block other threads.

    % uname -a
    Linux nute 2.6.8-2-686 #1 Thu May 19 17:53:30 JST 2005 i686 GNU/Linux
    % cat t.c
    #include <pthread.h>
    #include <unistd.h>
    #include <stdlib.h>
    #include <stdio.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    #include <fcntl.h>

    #define TMPFILE "/tmp/xx"

    void *f1(void *arg)
    {
    struct stat s;
    int i, ret;
    for (i = 0; i < 10; i++) {
    ret = stat(TMPFILE, &s);
    if (ret == -1)
    printf("%d: stat fail\n", i);
    else
    printf("%d: %x\n", i, s.st_size);
    usleep(100000);
    }
    }

    void *f2(void *arg)
    {
    #define SIZ 0x10000000
    char *buf = malloc(SIZ);
    int fd;
    int ret;
    if (buf == NULL) { perror("malloc"); return NULL; }
    fd = open(TMPFILE, O_WRONLY|O_CREAT|O_TRUNC, 0666);
    if (fd == -1) { perror("open"); return NULL; }
    printf("begin write\n");
    ret = write(fd, buf, SIZ);
    printf("end write %x\n", ret);
    return NULL;
    }

    int main()
    {
    pthread_t th1, th2;
    unlink(TMPFILE);
    pthread_create(&th1, NULL, f1, NULL);
    pthread_create(&th2, NULL, f2, NULL);
    pthread_join(th1, NULL);
    pthread_join(th2, NULL);
    return 0;
    }
    % gcc t.c -lpthread
    % ./a.out
    0: stat fail
    begin write
    1: 2143000
    2: 41a3000
    3: 61d9000
    4: 8264000
    5: a0e8000
    6: c08e000
    7: e0d3000
    end write 10000000
    8: 10000000
    9: 10000000
    --
    Tanaka Akira
     
    Tanaka Akira, Aug 10, 2005
    #19
  20. Tanaka Akira wrote:
    > In article <>,
    > Joel VanderWerf <> writes:
    >
    >
    >>>* read from socket: works well by default. (O_NONBLOCK is not required.)
    >>> (UDP packet with wrong checksum on Linux 2.6 is an exception.)

    >>
    >>Can you say a bit more about what happens differently on Linux 2.6 when
    >>a UDP packet has a bad checksum? Is it a ruby behavior or a linux behavior?
    >>
    >>(I'm working with wireless UDP on linux 2.6, having migrated from QNX,
    >>so it sounds important to me...)

    >
    >
    > Linux 2.6's select notify readability by a UDP packet with a bad checksum.
    > But read discards it and waits next packet with a correct checksum if
    > O_NONBLOCK is clear. If the next packet is not available, the read
    > hangs. Since O_NONBLOCK avoids the hang, it is possible to care the
    > problem in application level.
    >
    > Linux-Kernel Archive: UDP recvmsg blocks after select(), 2.6 bug?
    > http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1372.html
    >
    > Debian Bug report logs - #275585 - /usr/sbin/inetd: UDP builtins can be used to hang inetd
    > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=275585&archive=yes
    >
    > select(2):
    > Under Linux, select may report a socket file descriptor as "ready for
    > reading", while nevertheless a subsequent read blocks. This could for
    > example happen when data has arrived but upon examination has wrong
    > checksum and is discarded. There may be other circumstances. Thus it
    > may be safer to use O_NONBLOCK on sockets that should not block.
    >
    > [ruby-talk:148436] Nonblocking Sockets


    That post (148436) suggests that the native thread running the
    interpreter is not blocked, and the only danger of blocking is in the
    *ruby* thread reading the socket. Is that right? A bad UDP packet won't
    stop other ruby threads?

    I tried to hang ruby with

    hping2 --udp -p 9999 -b

    but the socket did not block.

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Aug 10, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Chrisman
    Replies:
    2
    Views:
    498
    Dan Gravell
    Apr 13, 2004
  2. jose luis fernandez diaz

    Using signals as a event driven framework

    jose luis fernandez diaz, May 3, 2004, in forum: C Programming
    Replies:
    6
    Views:
    410
    Xenos
    May 3, 2004
  3. Phillip B Oldham

    Event-driven framework (other than Twisted)?

    Phillip B Oldham, Oct 1, 2008, in forum: Python
    Replies:
    15
    Views:
    694
    James Mills
    Oct 2, 2008
  4. hemant
    Replies:
    10
    Views:
    244
    Jason Roelofs
    Dec 5, 2007
  5. Wyatt Biker
    Replies:
    4
    Views:
    153
    Chuck Remes
    Oct 1, 2009
Loading...

Share This Page