Parallelization with Python: which, where, how?

Discussion in 'Python' started by Mathias, Dec 20, 2004.

  1. Mathias

    Mathias Guest

    Dear NG,

    I have a (pretty much) "emberassingly parallel" problem and look for the
    right toolbox to parallelize it over a cluster of homogenous linux
    workstations. I don't need automatic loop-parallelization or the like
    since I prefer to prepare the work packets "by hand".
    I simply need
    - to specify a list of clients
    - a means of sending a work packet to a free client and receiving the
    result (hopefully automatically without need to login to each one)
    - optionally a timeout mechanism if a client doesn't respond
    - optionally help for debugging of remote clients

    So far I've seen scipy's COW (cluster of workstation) package, but
    couldn't find documentation or even examples for it (and the small
    example in the code crashes...).
    I've noticed PYRO as well, but didn't look too far yet.

    Can someone recommend a parallelization approach? Are there examples or
    documentation? Has someone got experience with stability and efficiency?

    Thanks a lot,
    Mathias
    Mathias, Dec 20, 2004
    #1
    1. Advertising

  2. "Mathias" <no_sp@m_please.cc> wrote:

    > I have a (pretty much) "emberassingly parallel" problem and look for the right toolbox to
    > parallelize it over a cluster of homogenous linux workstations. I don't need automatic
    > loop-parallelization or the like since I prefer to prepare the work packets "by hand".
    > I simply need
    > - to specify a list of clients
    > - a means of sending a work packet to a free client and receiving the
    > result (hopefully automatically without need to login to each one)
    > - optionally a timeout mechanism if a client doesn't respond
    > - optionally help for debugging of remote clients
    >
    > So far I've seen scipy's COW (cluster of workstation) package, but couldn't find documentation or
    > even examples for it (and the small example in the code crashes...).
    > I've noticed PYRO as well, but didn't look too far yet.
    >
    > Can someone recommend a parallelization approach? Are there examples or documentation? Has someone
    > got experience with stability and efficiency?


    googling for "parallel python" brings up lots of references; tools like

    http://pympi.sourceforge.net/
    http://datamining.anu.edu.au/~ole/pypar/

    (see https://geodoc.uchicago.edu/climatewiki/DiscussPythonMPI for
    a comparision)

    seem to be commonly used.

    </F>
    Fredrik Lundh, Dec 20, 2004
    #2
    1. Advertising

  3. Mathias

    Paul Rubin Guest

    Mathias <no_sp@m_please.cc> writes:
    > Can someone recommend a parallelization approach? Are there examples
    > or documentation? Has someone got experience with stability and
    > efficiency?


    In the "persistent objects" thread someone mentioned a very cool package
    called POSH:

    http://poshmodule.sourceforge.net/posh/html/posh.html
    Paul Rubin, Dec 20, 2004
    #3
  4. Mathias

    Ganesan R Guest

    >>>>> "Mathias" == Mathias <no_sp@m_please.cc> writes:

    > Dear NG,
    > I have a (pretty much) "emberassingly parallel" problem and look for
    > the right toolbox to parallelize it over a cluster of homogenous linux
    > workstations. I don't need automatic loop-parallelization or the like
    > since I prefer to prepare the work packets "by hand".
    > I simply need
    > - to specify a list of clients
    > - a means of sending a work packet to a free client and receiving the
    > result (hopefully automatically without need to login to each one)
    > - optionally a timeout mechanism if a client doesn't respond
    > - optionally help for debugging of remote clients


    pypvm or pympi? See http://pypvm.sourceforge.net/ and
    http://pympi.sourceforge.net/.

    Ganesan
    Ganesan R, Dec 20, 2004
    #4
  5. Mathias wrote:
    > I have a (pretty much) "emberassingly parallel" problem and look for the
    > right toolbox to parallelize it over a cluster of homogenous linux
    > workstations.


    We have a >1000-node cluster here and use the commercial Platform LSF to
    manage it. My Poly package
    <http://www.ebi.ac.uk/~hoffman/software/poly/> makes that trivial to use
    from Python and also avoids many of the pitfalls of programming farms
    that large, such as accidental distributed denial of service attacks on
    your own fileserver ;)

    Due to the cost and difficulty of setup, LSF is probably not what you
    want, or you would already have it. But MPI is probably not what you
    want if you are doing embarassingly parallelizable problems. I would
    look into OpenPBS <http://www.openpbs.org/>. If you want to write a Poly
    plugin for OpenPBS, I would be happy to accept it. ;)
    --
    Michael Hoffman
    Michael Hoffman, Dec 20, 2004
    #5
  6. On Mon, 20 Dec 2004 14:03:09 +0100, Mathias <no_sp@m_please.cc> wrote:
    > Can someone recommend a parallelization approach? Are there examples or
    > documentation? Has someone got experience with stability and efficiency?


    If you think a light-weight approach of distributing work and collecting
    the output afterwards (using ssh/rsh) fits your problem, send me an
    email.

    Albert
    --
    Unlike popular belief, the .doc format is not an open publically available format.
    Albert Hofkamp, Jan 4, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. P.M.
    Replies:
    60
    Views:
    1,237
    Alex Martelli
    Oct 26, 2004
  2. P.M.
    Replies:
    0
    Views:
    301
  3. Mike M?ller
    Replies:
    0
    Views:
    343
    Mike M?ller
    Dec 21, 2004
  4. Jp Calderone
    Replies:
    1
    Views:
    399
    Irmen de Jong
    Dec 21, 2004
  5. Robert Dailey

    Parallelization in Python 2.6

    Robert Dailey, Aug 18, 2009, in forum: Python
    Replies:
    16
    Views:
    997
    Neal Becker
    Aug 19, 2009
Loading...

Share This Page