Task Engine Framework?


N

Nate

Hello,

I'm in the process of developing a task engine / workflow module for
my Python application and I'm wondering if anyone knows of existing
code that could be used or adapted. Since I know that's far too
generic a question, let me share my goals:
1) Support long running operations (think backing up millions of
files) where:
- The operation can be paused (application closed) and the
operation resumed later.
- Individual tasks can be chained, run in parallel, or looped over
(the workflow part)
2) Would like to graph each defined operation (task A starts task B
with parameters... ) for documenting algorithms in Software Design
Document
3) Each individual task in the operation would a self-contained
class. I'd imagine implementing its action by defining a doTask()
method.

Hopefully that's clear. I just feel like someone must have already
solved this elegantly. I greatly enjoy Python and I look forward to
proving its use as a valuable language for a Masters student even
though everyone thinks I should use C# :).

Thanks!
-Nate
Masters Student at Eastern Washington University
 
Ad

Advertisements

A

Adam Tauno Williams

Hello,
I'm in the process of developing a task engine / workflow module for
my Python application and I'm wondering if anyone knows of existing
code that could be used or adapted. Since I know that's far too
generic a question, let me share my goals:
1) Support long running operations (think backing up millions of
files) where:
- The operation can be paused (application closed) and the
operation resumed later.
- Individual tasks can be chained, run in parallel, or looped over
(the workflow part)

We have something like that in OIE (OpenGroupware Integration Engine).
<http://sourceforge.net/projects/coils/>. These things tend to turn out
to be quite specific [and thus not generic]. But if you have any
questions feel free to ask. The focus in OIE was the ability to
describe processes in BPML and facilitate process management [creating,
queuing, parking (stopping for later resume) of business / ETL tasks.
Parts of the code aren't especially elegant but it does move a fairly
large amount of data every day.
 
N

Nate

Hello,
I'm in the process of developing a task engine / workflow module for
my Python application and I'm wondering if anyone knows of existing
code that could be used or adapted.  Since I know that's far too
generic a question, let me share my goals:
1) Support long running operations (think backing up millions of
files) where:
   - The operation can be paused (application closed) and the
operation resumed later.
   - Individual tasks can be chained, run in parallel, or looped over
(the workflow part)

We have something like that in OIE (OpenGroupware Integration Engine).
<http://sourceforge.net/projects/coils/>.  These things tend to turn out
to be quite specific [and thus not generic].  But if you have any
questions feel free to ask.  The focus in OIE was the ability to
describe processes in BPML and facilitate process management [creating,
queuing, parking (stopping for later resume) of business / ETL tasks.
Parts of the code aren't especially elegant but it does move a fairly
large amount of data every day.






2) Would like to graph each defined operation (task A starts task B
with parameters... ) for documenting algorithms in Software Design
Document
3) Each individual task in the operation would a self-contained
class.  I'd imagine implementing its action by defining a doTask()
method
Hopefully that's clear.  I just feel like someone must have already
solved this elegantly.  I greatly enjoy Python and I look forward to
proving its use as a valuable language for a Masters student even
though everyone thinks I should use C# :).

Thank you,

I'll take a look at the project. At the very least, seeing someone
else's solution would be helpful. I'm trying desperately hard to keep
the code simple :)

-Nate
 
Ad

Advertisements

T

Tim Chase

I'll take a look at the project. At the very least, seeing someone
else's solution would be helpful. I'm trying desperately hard to keep
the code simple :)

You might also want to look at Celery[1] which may do some (if
not all) of what you're looking for.

-tkc

[1]
http://celeryproject.org
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top