Task Engine Framework?

Discussion in 'Python' started by Nate, Dec 6, 2010.

  1. Nate

    Nate Guest

    Hello,

    I'm in the process of developing a task engine / workflow module for
    my Python application and I'm wondering if anyone knows of existing
    code that could be used or adapted. Since I know that's far too
    generic a question, let me share my goals:
    1) Support long running operations (think backing up millions of
    files) where:
    - The operation can be paused (application closed) and the
    operation resumed later.
    - Individual tasks can be chained, run in parallel, or looped over
    (the workflow part)
    2) Would like to graph each defined operation (task A starts task B
    with parameters... ) for documenting algorithms in Software Design
    Document
    3) Each individual task in the operation would a self-contained
    class. I'd imagine implementing its action by defining a doTask()
    method.

    Hopefully that's clear. I just feel like someone must have already
    solved this elegantly. I greatly enjoy Python and I look forward to
    proving its use as a valuable language for a Masters student even
    though everyone thinks I should use C# :).

    Thanks!
    -Nate
    Masters Student at Eastern Washington University
    Nate, Dec 6, 2010
    #1
    1. Advertising

  2. On Mon, 2010-12-06 at 15:11 -0800, Nate wrote:
    > Hello,
    > I'm in the process of developing a task engine / workflow module for
    > my Python application and I'm wondering if anyone knows of existing
    > code that could be used or adapted. Since I know that's far too
    > generic a question, let me share my goals:
    > 1) Support long running operations (think backing up millions of
    > files) where:
    > - The operation can be paused (application closed) and the
    > operation resumed later.
    > - Individual tasks can be chained, run in parallel, or looped over
    > (the workflow part)


    We have something like that in OIE (OpenGroupware Integration Engine).
    <http://sourceforge.net/projects/coils/>. These things tend to turn out
    to be quite specific [and thus not generic]. But if you have any
    questions feel free to ask. The focus in OIE was the ability to
    describe processes in BPML and facilitate process management [creating,
    queuing, parking (stopping for later resume) of business / ETL tasks.
    Parts of the code aren't especially elegant but it does move a fairly
    large amount of data every day.

    > 2) Would like to graph each defined operation (task A starts task B
    > with parameters... ) for documenting algorithms in Software Design
    > Document
    > 3) Each individual task in the operation would a self-contained
    > class. I'd imagine implementing its action by defining a doTask()
    > method
    > Hopefully that's clear. I just feel like someone must have already
    > solved this elegantly. I greatly enjoy Python and I look forward to
    > proving its use as a valuable language for a Masters student even
    > though everyone thinks I should use C# :).
    Adam Tauno Williams, Dec 7, 2010
    #2
    1. Advertising

  3. Nate

    Nate Guest

    On Dec 7, 8:32 am, Adam Tauno Williams <> wrote:
    > On Mon, 2010-12-06 at 15:11 -0800, Nate wrote:
    > > Hello,
    > > I'm in the process of developing a task engine / workflow module for
    > > my Python application and I'm wondering if anyone knows of existing
    > > code that could be used or adapted.  Since I know that's far too
    > > generic a question, let me share my goals:
    > > 1) Support long running operations (think backing up millions of
    > > files) where:
    > >    - The operation can be paused (application closed) and the
    > > operation resumed later.
    > >    - Individual tasks can be chained, run in parallel, or looped over
    > > (the workflow part)

    >
    > We have something like that in OIE (OpenGroupware Integration Engine).
    > <http://sourceforge.net/projects/coils/>.  These things tend to turn out
    > to be quite specific [and thus not generic].  But if you have any
    > questions feel free to ask.  The focus in OIE was the ability to
    > describe processes in BPML and facilitate process management [creating,
    > queuing, parking (stopping for later resume) of business / ETL tasks.
    > Parts of the code aren't especially elegant but it does move a fairly
    > large amount of data every day.
    >
    >
    >
    >
    >
    >
    >
    > > 2) Would like to graph each defined operation (task A starts task B
    > > with parameters... ) for documenting algorithms in Software Design
    > > Document
    > > 3) Each individual task in the operation would a self-contained
    > > class.  I'd imagine implementing its action by defining a doTask()
    > > method
    > > Hopefully that's clear.  I just feel like someone must have already
    > > solved this elegantly.  I greatly enjoy Python and I look forward to
    > > proving its use as a valuable language for a Masters student even
    > > though everyone thinks I should use C# :).


    Thank you,

    I'll take a look at the project. At the very least, seeing someone
    else's solution would be helpful. I'm trying desperately hard to keep
    the code simple :)

    -Nate
    Nate, Dec 7, 2010
    #3
  4. Nate

    Tim Chase Guest

    On 12/07/2010 10:56 AM, Nate wrote:
    > On Dec 7, 8:32 am, Adam Tauno Williams<> wrote:
    >>> I'm in the process of developing a task engine / workflow module for
    >>> my Python application and I'm wondering if anyone knows of existing
    >>> code that could be used or adapted. Since I know that's far too
    >>> generic a question, let me share my goals:
    >>> 1) Support long running operations (think backing up millions of
    >>> files) where:
    >>> - The operation can be paused (application closed) and the
    >>> operation resumed later.
    >>> - Individual tasks can be chained, run in parallel, or looped over
    >>> (the workflow part)

    >>
    >> We have something like that in OIE (OpenGroupware Integration Engine).
    >> <http://sourceforge.net/projects/coils/>.

    >
    > I'll take a look at the project. At the very least, seeing someone
    > else's solution would be helpful. I'm trying desperately hard to keep
    > the code simple :)


    You might also want to look at Celery[1] which may do some (if
    not all) of what you're looking for.

    -tkc

    [1]
    http://celeryproject.org
    Tim Chase, Dec 7, 2010
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Victor
    Replies:
    0
    Views:
    8,819
    Victor
    Sep 1, 2004
  2. krabhi
    Replies:
    1
    Views:
    9,262
    Marco Meschieri
    Aug 9, 2006
  3. teggy
    Replies:
    0
    Views:
    804
    teggy
    May 29, 2007
  4. Mike
    Replies:
    1
    Views:
    1,919
    GArlington
    May 12, 2008
  5. Stéphane Wirtel
    Replies:
    3
    Views:
    355
    Stephane Wirtel
    Jun 15, 2007
Loading...

Share This Page