Deferred jobs server as backend for web application

[Sean Davis]

In the past, I have put together web applications that process tasks
serially, either with short algorithms to manipulate user-submitted
data or to return database queries. However, now I am faced with the
task of having a long-running process being started by a web
submission. I want to process an uploaded file. One way to do this
is to simply start a process each time someone submits a job and then
email when complete. Instead, I would like to have a way of
submitting the job to a persistent backend queue that can process the
job and answer queries about the status of the job (if it is still
running) or return results. I have looked at threadpool, which seems
fine if I a want to submit jobs from a single process (such as a Qt
application, etc), but it won't work directly for a web platform where
I will likely have multiple threads/processes handling http requests.
Any suggestions on how to go about this?

Thanks,
Sean

 

[Kushal Kumaran]

In the past, I have put together web applications that process tasks
serially, either with short algorithms to manipulate user-submitted
data or to return database queries. However, now I am faced with the
task of having a long-running process being started by a web
submission. I want to process an uploaded file. One way to do this
is to simply start a process each time someone submits a job and then
email when complete. Instead, I would like to have a way of
submitting the job to a persistent backend queue that can process the
job and answer queries about the status of the job (if it is still
running) or return results. I have looked at threadpool, which seems
fine if I a want to submit jobs from a single process (such as a Qt
application, etc), but it won't work directly for a web platform where
I will likely have multiple threads/processes handling http requests.
Any suggestions on how to go about this?

Thanks,
Sean

I recently discovered this:
http://vicerveza.homeunix.net/~viric/soft/ts/

Might be what you need.