queuing work in a web app (sinatra/webrick)

[Gurpal 2000]

Hi

I want to make a webapp that allows users to queue bit of work
(converting video for example). They are also able to see the queue at
any time and check the progress (queued|processing|done) and are able to
REORDER items in the queue.
The queue can be stopped (ie. paused) and resumed in between conversions
of course.

I'm using sintatra and the LoopingThread example from here
http://codeidol.com/other/rubyckbk/Multitasking-and-Multithreading/Terminating-a-Thread/

Im sure there's ppl out there who've done something similar. I can do
this in java no problem (synchronized lists etc) but not sure how to do
this in ruby as im sort of a newbie.

I have a working bit of code but im sure im missing something (memory
leak, synchorizing issues etc). Can post more code (very small) if
someone's willing to help.

Thanks!

 

[Gurpal 2000]

Hi

I want to make a webapp that allows users to queue bit of work
(converting video for example). They are also able to see the queue at
any time and check the progress (queued|processing|done) and are able to
REORDER items in the queue.
The queue can be stopped (ie. paused) and resumed in between conversions
of course.

I'm using sintatra and the LoopingThread example from here
http://codeidol.com/other/rubyckbk/Multitasking-and-Multithreading/Terminating-a-Thread/

Im sure there's ppl out there who've done something similar. I can do
this in java no problem (synchronized lists etc) but not sure how to do
this in ruby as im sort of a newbie.

I have a working bit of code but im sure im missing something (memory
leak, synchorizing issues etc). Can post more code (very small) if
someone's willing to help.

Thanks!

Oh and i forgot to mention that the queue can be persisted if u want - i
dont care at this stage. But im essentially keeping the queue in a flat
file on disk to hopefully get some 'cheap locking'. I'd like to do this
without a DB so to speak.

thx

 

[Gurpal 2000]

snip..

OK i altered my code to use Sequel.sqlite. So now i have my "list" in
memory. So now the queue implementation is the issue. thx

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

OK i altered my code to use Sequel.sqlite. So now i have my "list" in
memory. So now the queue implementation is the issue. thx

Are you wanting to farm out work among a distributed network of agents?
Nanite + RabbitMQ comes to mind:

http://github.com/ezmobius/nanite

 

[Gurpal 2000]

Are you wanting to farm out work among a distributed network of agents?
Nanite + RabbitMQ comes to mind:

http://github.com/ezmobius/nanite

not really farm out, i'll check this out but mq sounds over the top for
what's essentially a small home project (for now)

 

[Richard Conroy]

[Note: parts of this message were removed to make it a legal post.]

Oh and i forgot to mention that the queue can be persisted if u want - i
dont care at this stage. But im essentially keeping the queue in a flat
file on disk to hopefully get some 'cheap locking'. I'd like to do this
without a DB so to speak.

There are numerous solutions for this kind of task: BackgroundJob,
DelayedJob,
Workling etc. The list is quite extensive.

My personal opinion is that spinning off threads to do this work is not the
way to
go about it. To my mind this is an activity that should be spun off in a
separate
process.

While you want to avoid using the DB for this, I think you should
reconsider:
DB backed scheduling gets your locking and monitoring via
transactions/persistence and
it doesn't have to be difficult either. I would look for a background
scheduling tool
that implicitly uses your DB so that you don't get your hands dirty.

 

[Gurpal 2000]

OK i managed to use Sequel DB. But the backgroundjob api etc seems
overkill. This LoopingThread works below, but i noticed 100% usgae on
ruby 1.8.6. Is this normal?

class LoopingThread < Thread
def initialize
@stopped = false
@paused = false
super do
before_loop
until @stopped
yield
Thread.stop if @paused
end
after_loop
end
end

def before_loop
end

def after_loop
end

def stop
@stopped = true
end

def paused=(paused)
@paused = paused
run if !paused
end
end

 

[Robert Gleeson]

Hi,

I would guess your culprit for the spike to 100% CPU usage may be your
infinite loop:

until @stopped
yield
Thread.stop if @paused
end

Look at using sleep() if you want to decrease CPU usage. Even sleeping
for one second should decrease your CPU usage dramatically in my
experience(maybe with a rare spike).

 

[Gurpal 2000]

Hi,

I would guess your culprit for the spike to 100% CPU usage may be your
infinite loop:



Look at using sleep() if you want to decrease CPU usage. Even sleeping
for one second should decrease your CPU usage dramatically in my
experience(maybe with a rare spike).

Sure, i got this code from the Oreilly Ruby Cookbook. Is it bad code
then?
Again this doesn't happen when i use similar code in Java...

 

[Robert Gleeson]

Sure, i got this code from the Oreilly Ruby Cookbook. Is it bad code

then?

The book seems to be a little outdated - for example, i rarely see
people defining setters themselves nowadays, especially when you can use
attr_writer ause . I'm not sure why they didn't pick up on what could
be an infinite loop, or at least a loop that can repeat itself many,
many times.

Again this doesn't happen when i use similar code in Java...

I don't know why that is in Java - in C, Ruby, an infinite loop is going
to consume a lot of CPU power unless you take precautions.