Task Engine Framework?

[Nate]

Hello,

I'm in the process of developing a task engine / workflow module for
my Python application and I'm wondering if anyone knows of existing
code that could be used or adapted. Since I know that's far too
generic a question, let me share my goals:
1) Support long running operations (think backing up millions of
files) where:
- The operation can be paused (application closed) and the
operation resumed later.
- Individual tasks can be chained, run in parallel, or looped over
(the workflow part)
2) Would like to graph each defined operation (task A starts task B
with parameters... ) for documenting algorithms in Software Design
Document
3) Each individual task in the operation would a self-contained
class. I'd imagine implementing its action by defining a doTask()
method.

Hopefully that's clear. I just feel like someone must have already
solved this elegantly. I greatly enjoy Python and I look forward to
proving its use as a valuable language for a Masters student even
though everyone thinks I should use C# .

Thanks!
-Nate
Masters Student at Eastern Washington University

 

[Adam Tauno Williams]

Hello,
I'm in the process of developing a task engine / workflow module for
my Python application and I'm wondering if anyone knows of existing
code that could be used or adapted. Since I know that's far too
generic a question, let me share my goals:
1) Support long running operations (think backing up millions of
files) where:
- The operation can be paused (application closed) and the
operation resumed later.
- Individual tasks can be chained, run in parallel, or looped over
(the workflow part)

We have something like that in OIE (OpenGroupware Integration Engine).
<http://sourceforge.net/projects/coils/>. These things tend to turn out
to be quite specific [and thus not generic]. But if you have any
questions feel free to ask. The focus in OIE was the ability to
describe processes in BPML and facilitate process management [creating,
queuing, parking (stopping for later resume) of business / ETL tasks.
Parts of the code aren't especially elegant but it does move a fairly
large amount of data every day.

 

[Nate]

Hello,
I'm in the process of developing a task engine / workflow module for
my Python application and I'm wondering if anyone knows of existing
code that could be used or adapted.  Since I know that's far too
generic a question, let me share my goals:
1) Support long running operations (think backing up millions of
files) where:
   - The operation can be paused (application closed) and the
operation resumed later.
   - Individual tasks can be chained, run in parallel, or looped over
(the workflow part)

We have something like that in OIE (OpenGroupware Integration Engine).
<http://sourceforge.net/projects/coils/>.  These things tend to turn out
to be quite specific [and thus not generic].  But if you have any
questions feel free to ask.  The focus in OIE was the ability to
describe processes in BPML and facilitate process management [creating,
queuing, parking (stopping for later resume) of business / ETL tasks.
Parts of the code aren't especially elegant but it does move a fairly
large amount of data every day.

2) Would like to graph each defined operation (task A starts task B
with parameters... ) for documenting algorithms in Software Design
Document
3) Each individual task in the operation would a self-contained
class.  I'd imagine implementing its action by defining a doTask()
method
Hopefully that's clear.  I just feel like someone must have already
solved this elegantly.  I greatly enjoy Python and I look forward to
proving its use as a valuable language for a Masters student even
though everyone thinks I should use C# .

Thank you,

I'll take a look at the project. At the very least, seeing someone
else's solution would be helpful. I'm trying desperately hard to keep
the code simple

-Nate

 

[Tim Chase]

I'll take a look at the project. At the very least, seeing someone
else's solution would be helpful. I'm trying desperately hard to keep
the code simple

You might also want to look at Celery[1] which may do some (if
not all) of what you're looking for.

-tkc

[1]
http://celeryproject.org