Worker pool - what to use?

[Christoph B.]

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker "frameworks" come with
their own queue... however a few concerns:

- does putting msgs from one into another queue make sense?
- how reliable is resque? I cannot lose mails
- how to handle confirmation on the OpenMQ? if I insert the msgs into
resque

So I wondered how you would do this...
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue.. which is not really needed in this
scenario as I confirm messages on the OpenMQ..

I would appreciate any suggestions!
Thanks, Christoph

 

[Robert Klemme]

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker "frameworks" come with
their own queue... however a few concerns:

- does putting msgs from one into another queue make sense?
- how reliable is resque? I cannot lose mails
- how to handle confirmation on the OpenMQ? if I insert the msgs into
resque

So I wondered how you would do this...
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue.. which is not really needed in this
scenario as I confirm messages on the OpenMQ..

I would appreciate any suggestions!

<disclaimer>I don't know stomp and I have only rudimentary knowledge
of OpenMQ,</disclaimer>

It seems the simplest solution would be to create a number of Ruby
processes that each individually fetch data from your OpenMQ one after
another and process it. You could add a controlling master process
that restarts child processes if they die. Sample:

https://gist.github.com/794746

Cheers

robert

 

[Tim Pease]

=20
<disclaimer>I don't know stomp and I have only rudimentary knowledge
of OpenMQ,</disclaimer>
=20
It seems the simplest solution would be to create a number of Ruby
processes that each individually fetch data from your OpenMQ one after
another and process it. You could add a controlling master process
that restarts child processes if they die. Sample:
=20
https://gist.github.com/794746
=20

Here is an example that will spawn 7 worker process and connect to a =
beanstalkd job queue. Replace beanstalkd with an OpenMQ connection if =
you want. However beanstalkd is very simple, reliable, requeues jobs if =
they fail, etc.

https://github.com/TwP/servolux/blob/master/examples/beanstalk.rb

http://kr.github.com/beanstalkd/

Blessings,
TwP

 

[Christoph Blank]

Hey, thanks a lot for the input!

I will give those examples a try, looks pretty easy.
I also thought about just creating a thread pool.

One more thing:
How would you wrap this into a daemon, e.g. to have an executable with
start/stop etc commands?

I remember there was something like this, saw an example of a self
contained sinatra app, but I don't remember it anymore...

I would need that for monitoring with monit

Thanks for your help,
Christoph