queue management with "application failure management"

Discussion in 'Java' started by pouet, Jul 30, 2004.

  1. pouet

    pouet Guest

    hi,

    I am looking for something which looks like the message queue of the IPC
    systemV with java.

    my requirements are:

    -management of a queue. one application adds data in this queue and
    another gets data from this queue.

    -simple : no server like JMS. (btw if you know some JMS implementation
    without a server it could be what I want). could be in a file.

    -no data lost if one application crashes. ie if the application which
    gets data from the queue crashes, the other can continue to add data to
    this queue and the processing will continue without losing data once the
    crashed application is restarted.

    do somebody know some libs which implements this?

    thanks.
    pouet, Jul 30, 2004
    #1
    1. Advertising

  2. pouet

    Bryce Guest

    On Fri, 30 Jul 2004 12:44:07 +0200, pouet <> wrote:

    >hi,
    >
    >I am looking for something which looks like the message queue of the IPC
    >systemV with java.
    >
    >my requirements are:
    >
    >-management of a queue. one application adds data in this queue and
    >another gets data from this queue.


    Google for ThreadPools. You can use a Collection class inside a
    ThreadPool, and threads act on items it its queue:

    See here for a good example:
    http://www-106.ibm.com/developerworks/java/library/j-jtp0730.html

    >-simple : no server like JMS. (btw if you know some JMS implementation
    >without a server it could be what I want). could be in a file.


    >-no data lost if one application crashes. ie if the application which
    >gets data from the queue crashes, the other can continue to add data to
    >this queue and the processing will continue without losing data once the
    >crashed application is restarted.


    There are many strategies for handling this. Having your Queue use a
    database to persist its data, using transactions...

    >do somebody know some libs which implements this?
    >
    >thanks.


    Sounds like you need a JMS implementation.

    --
    now with more cowbell
    Bryce, Jul 30, 2004
    #2
    1. Advertising

  3. pouet

    Will Hartung Guest

    "pouet" <> wrote in message
    news:ced8pf$d07$...
    > hi,
    >
    > I am looking for something which looks like the message queue of the IPC
    > systemV with java.
    >
    > my requirements are:
    >
    > -management of a queue. one application adds data in this queue and
    > another gets data from this queue.
    >
    > -simple : no server like JMS. (btw if you know some JMS implementation
    > without a server it could be what I want). could be in a file.
    >
    > -no data lost if one application crashes. ie if the application which
    > gets data from the queue crashes, the other can continue to add data to
    > this queue and the processing will continue without losing data once the
    > crashed application is restarted.
    >
    > do somebody know some libs which implements this?


    Pretty vague.

    I'm assuming you're talking about applications that are running in seperate
    processes? Since you're talking SysV IPC, you're talking about processes on
    the same machine I reckon.

    As for libraries, no I don't have any links at all.

    One thing you could do, of course, is JNI some wrappers to the SysV IPC
    calls, getting essentially exactly what you asked for.

    Another thing you could do is implement this on top of the file system. This
    could work very well. The trick would be to write the file to a temp area,
    then "rename" the file to the queue area, as that's typically an atomic
    operation on most filesystems. If you don't do that, the file "appears" in
    the directory before it's actually done being written, and you don't want
    that.

    The client would start up, read the ENTIRE directory of the queue, process
    each element and then delete the respective file. When it ran out of things
    to process, it would read the directory again. If nothing was there, it
    should sleep (1 second, 10 seonds, whatever is appropriate for your task)
    and then start over.

    You have the potential of processing a queue element twice if your queue
    client fails after it has finished processing the queue item, but before it
    deletes the item. If practical, you can put code to check if an item has
    been processed (through an external mechanism) before processing it again.
    But this is a race condition you should certainly consider. Most distributed
    queueing system have this problem.

    By reading the entire directory, you gain efficiency of only have to read
    the directory once for each block of items, but you lose the ability to
    "dequeue" a queued item. Also you'll have a blip on startup if you have a
    LOT of items queued up. On the other hand, if you have a lot of items in the
    queue, you don't want to read the entire directory every time just to get
    the next item in the queue (and you pretty much have to do that as
    directories typically do not sort their entries, so there is no guarantee
    that the first element in the directory is indeed the "oldest" item in the
    queue).

    Also by using a filesystems, there is really no efficient wait to throttle
    people inserting items in the queue, or limiting its contents (like setting
    a statistic of "at most 100 items in the queue").

    Another thing that you can use is a named pipe, particularly if your items
    are small. Have your client program simply read the named pipe (for example,
    it could be reading file names, or actual data). But the named pipe messages
    have to be "small", and your injectors will block if the pipe fills up. I
    think the total size for a pipe is only 4K bytes, but that's enough for ~200
    filenames, say. The pipe should survive application restarts, but not system
    restarts. You also will lose the item that is being worked on if the queue
    client fails after it has extracted the item. The high side is that it's
    really low weight to implement (simply reading and writing a stream), but it
    may be too fragile or underperformant for your needs, depending on your
    transaction load. You also can't "see what's in the queue", as it's buried
    in the pipe.

    As someone else mentioned you could easily do this with a database as well.

    Finally, you could play with shared memory through mapped files, but I'd use
    the directory method before I went that direction.

    Regards,

    Will Hartung
    ()
    Will Hartung, Jul 30, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Paul L. Du Bois

    Queue.Queue-like class without the busy-wait

    Paul L. Du Bois, Mar 24, 2005, in forum: Python
    Replies:
    29
    Views:
    1,031
    Antoon Pardon
    Apr 4, 2005
  2. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    654
    Russell Warren
    Jun 27, 2006
  3. Kceiw
    Replies:
    3
    Views:
    973
    Jim Langston
    Mar 14, 2006
  4. Gabriel Rossetti
    Replies:
    3
    Views:
    486
    Jerry Hill
    Apr 25, 2008
  5. Kris
    Replies:
    0
    Views:
    424
Loading...

Share This Page