Advice needed on parallel processing in python

Discussion in 'Python' started by Glazner, Mar 24, 2010.

  1. Glazner

    Glazner Guest

    Hi!

    I need to replace an app that does number crunching over a local
    network.
    it have about 50 computers as slaves
    each computer needs to run COM that will do the "job"
    right now the system uses MFC threads and DCOM to distribute the load.

    as i said, Now i'm trying to replace this system with python.
    I already use win32all and read about Pareller Python and Pyro.

    a requirement is that messages can be sent to the remote worker.

    If anyone can share experience I'll love to hear


    Many Thanks,
    Yoav Glazner
     
    Glazner, Mar 24, 2010
    #1
    1. Advertisements

  2. Glazner

    Jon Clements Guest

    On 24 Mar, 15:27, Glazner <> wrote:
    > Hi!
    >
    > I need to replace an app that does number crunching over a local
    > network.
    > it have about 50 computers as slaves
    > each computer needs to run COM that will do the "job"
    > right now the system uses MFC threads and DCOM to distribute the load.
    >
    > as i said, Now i'm trying to replace this system with python.
    > I already use win32all and read about Pareller Python and Pyro.
    >
    > a requirement is that messages can be sent to the remote worker.
    >
    > If anyone can share experience I'll love to hear
    >
    > Many Thanks,
    > Yoav Glazner


    Would Celery suit? http://celeryproject.org/

    hth

    Jon.
     
    Jon Clements, Mar 24, 2010
    #2
    1. Advertisements

  3. On Mar 24, 1:13 pm, Jon Clements <> wrote:
    > On 24 Mar, 15:27, Glazner <> wrote:
    >
    >
    >
    > > Hi!

    >
    > > I need to replace an app that does number crunching over a local
    > > network.
    > > it have about 50 computers as slaves
    > > each computer needs to run COM that will do the "job"
    > > right now the system uses MFC threads and DCOM to distribute the load.

    >
    > > as i said, Now i'm trying to replace this system with python.
    > > I already use win32all and read about Pareller Python and Pyro.

    >
    > > a requirement is that messages can be sent to the remote worker.

    >
    > > If anyone can share experience I'll love to hear

    >
    > > Many Thanks,
    > > Yoav Glazner

    >
    > Would Celery suit?http://celeryproject.org/
    >
    > hth
    >
    > Jon.



    If you can add just one Linux server onto that network you could use
    SIMPL-Python to do what you want.

    http://www.icanprogram.com/06py/lesson1/lesson1.html

    ie. SIMPL messages going seamlessly from Python (Windows) to Python
    (Windows) with the SIMPL sandbox being hosted on the Linux node.

    Of course if you wanted to add Linux nodes as processing nodes.
    Python (Windows) modules could seamlessly exchange messages with those
    as well. Same would be true if you wanted to distribute part of your
    app into the cloud. All your Python code would look virtually
    identical in each of these cases.

    As an example of an inexpensive Linux node the SIMPL toolkit has been
    ported to the ~$100 Sheeva Plug computer (http://www.icanprogram.com/
    simpl/plugsimplbin.self.html) (http://www.plugcomputer.org).

    bob
     
    bobicanprogram, Mar 25, 2010
    #3
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mike
    Replies:
    10
    Views:
    9,453
    Cristian
    Feb 6, 2009
  2. Andrei D.
    Replies:
    2
    Views:
    518
    Grant Edwards
    Aug 6, 2003
  3. bearophile

    Parallel processing

    bearophile, Nov 17, 2004, in forum: Python
    Replies:
    5
    Views:
    676
    Josiah Carlson
    Nov 18, 2004
  4. howa
    Replies:
    8
    Views:
    2,775
  5. Michel Albert

    parallel csv-file processing

    Michel Albert, Nov 9, 2007, in forum: Python
    Replies:
    3
    Views:
    428
    Paul Boddie
    Nov 9, 2007
  6. Robert Kern
    Replies:
    4
    Views:
    350
    Robert Kern
    Dec 29, 2007
  7. Soren
    Replies:
    4
    Views:
    1,604
    c d saunter
    Feb 14, 2008
  8. Vivek Menon
    Replies:
    5
    Views:
    3,947
    Paul Uiterlinden
    Jun 8, 2011
Loading...