queue management with "application failure management"

P

pouet

hi,

I am looking for something which looks like the message queue of the IPC
systemV with java.

my requirements are:

-management of a queue. one application adds data in this queue and
another gets data from this queue.

-simple : no server like JMS. (btw if you know some JMS implementation
without a server it could be what I want). could be in a file.

-no data lost if one application crashes. ie if the application which
gets data from the queue crashes, the other can continue to add data to
this queue and the processing will continue without losing data once the
crashed application is restarted.

do somebody know some libs which implements this?

thanks.
 
B

Bryce

hi,

I am looking for something which looks like the message queue of the IPC
systemV with java.

my requirements are:

-management of a queue. one application adds data in this queue and
another gets data from this queue.

Google for ThreadPools. You can use a Collection class inside a
ThreadPool, and threads act on items it its queue:

See here for a good example:
http://www-106.ibm.com/developerworks/java/library/j-jtp0730.html
-simple : no server like JMS. (btw if you know some JMS implementation
without a server it could be what I want). could be in a file.
-no data lost if one application crashes. ie if the application which
gets data from the queue crashes, the other can continue to add data to
this queue and the processing will continue without losing data once the
crashed application is restarted.

There are many strategies for handling this. Having your Queue use a
database to persist its data, using transactions...
do somebody know some libs which implements this?

thanks.

Sounds like you need a JMS implementation.
 
W

Will Hartung

pouet said:
hi,

I am looking for something which looks like the message queue of the IPC
systemV with java.

my requirements are:

-management of a queue. one application adds data in this queue and
another gets data from this queue.

-simple : no server like JMS. (btw if you know some JMS implementation
without a server it could be what I want). could be in a file.

-no data lost if one application crashes. ie if the application which
gets data from the queue crashes, the other can continue to add data to
this queue and the processing will continue without losing data once the
crashed application is restarted.

do somebody know some libs which implements this?

Pretty vague.

I'm assuming you're talking about applications that are running in seperate
processes? Since you're talking SysV IPC, you're talking about processes on
the same machine I reckon.

As for libraries, no I don't have any links at all.

One thing you could do, of course, is JNI some wrappers to the SysV IPC
calls, getting essentially exactly what you asked for.

Another thing you could do is implement this on top of the file system. This
could work very well. The trick would be to write the file to a temp area,
then "rename" the file to the queue area, as that's typically an atomic
operation on most filesystems. If you don't do that, the file "appears" in
the directory before it's actually done being written, and you don't want
that.

The client would start up, read the ENTIRE directory of the queue, process
each element and then delete the respective file. When it ran out of things
to process, it would read the directory again. If nothing was there, it
should sleep (1 second, 10 seonds, whatever is appropriate for your task)
and then start over.

You have the potential of processing a queue element twice if your queue
client fails after it has finished processing the queue item, but before it
deletes the item. If practical, you can put code to check if an item has
been processed (through an external mechanism) before processing it again.
But this is a race condition you should certainly consider. Most distributed
queueing system have this problem.

By reading the entire directory, you gain efficiency of only have to read
the directory once for each block of items, but you lose the ability to
"dequeue" a queued item. Also you'll have a blip on startup if you have a
LOT of items queued up. On the other hand, if you have a lot of items in the
queue, you don't want to read the entire directory every time just to get
the next item in the queue (and you pretty much have to do that as
directories typically do not sort their entries, so there is no guarantee
that the first element in the directory is indeed the "oldest" item in the
queue).

Also by using a filesystems, there is really no efficient wait to throttle
people inserting items in the queue, or limiting its contents (like setting
a statistic of "at most 100 items in the queue").

Another thing that you can use is a named pipe, particularly if your items
are small. Have your client program simply read the named pipe (for example,
it could be reading file names, or actual data). But the named pipe messages
have to be "small", and your injectors will block if the pipe fills up. I
think the total size for a pipe is only 4K bytes, but that's enough for ~200
filenames, say. The pipe should survive application restarts, but not system
restarts. You also will lose the item that is being worked on if the queue
client fails after it has extracted the item. The high side is that it's
really low weight to implement (simply reading and writing a stream), but it
may be too fragile or underperformant for your needs, depending on your
transaction load. You also can't "see what's in the queue", as it's buried
in the pipe.

As someone else mentioned you could easily do this with a database as well.

Finally, you could play with shared memory through mapped files, but I'd use
the directory method before I went that direction.

Regards,

Will Hartung
([email protected])
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top