Request for comments on python distributed technologies

Discussion in 'Python' started by Carl J. Van Arsdall, May 19, 2006.

  1. Hey everyone, another question for the list. In particular i'm looking
    for comments on some of the distributed technologies supported in
    python. Specifically, I'm looking at XML-RPC, RPyC, CORBA, and Twisted.

    Before you offer any comments let me talk about what i'm doing a little
    bit. Right now I'm creating an execution framework. The framework aims
    to allow users to create a modular system by which engineers can
    mitigate some of the issues seen in large complex systems. By using
    modules the system hopes to allow the system to change but allow
    engineers to re-use as much code as possible. By using the modules in a
    framework we hope to encapsulate the entire system to that it is
    controllable in the sense that we can generate and catch errors anywhere
    in the system and exit gracefully, things like that. There's a lot more
    to it but the motivation spawn from our system where the system kept
    needing to adapt to support changes in our company. These changes over
    time have resulted in a large number of "hacks". The hope of the
    modules is to make the code a lot cleaner and organized so that the
    system can operate in whatever way it needs to without having so many
    hacks. This system I use also operates in a primitive distributed
    environment. Currently, these are ssh calls to other nodes that then
    execute other python scripts and shell commands. This is the next piece
    I'm interested in moving away from.

    With all that being said, I've come across those four technologies so
    far and I've been looking into them. I was wondering if anyone had any
    comments, pros/cons, or suggestions of other distributed technologies to
    take a look at. I intend to integrate the distributed technology
    directly into the framework making it easy for developers to distribute
    tasks. Anyhow, any comments would be greatly appreciated! I also hope
    to release this framework as an open-source project when its finished,
    so again I community input is important.

    Thanks!

    ..c


    --

    Carl J. Van Arsdall

    Build and Release
    MontaVista Software
    Carl J. Van Arsdall, May 19, 2006
    #1
    1. Advertising

  2. Carl J. Van Arsdall

    Paul McGuire Guest

    "Carl J. Van Arsdall" <> wrote in message
    news:...
    > Hey everyone, another question for the list. In particular i'm looking
    > for comments on some of the distributed technologies supported in
    > python. Specifically, I'm looking at XML-RPC, RPyC, CORBA, and Twisted.
    >


    What you describe sounds very much like pyro, which should probably be added
    to your list.

    BTW, distributed frameworks are not inherently more robust or more easily
    maintained than monolithic systems. Unless you establish dynamic
    replication and/or relocation of services, you can easily end up with one or
    more single points of failure. Services should be self-registering in order
    to make finding them an automatic process, without having to maintain global
    configuration/locator files or services (which themselves can become
    bottlenecks), or distributed lookup files that must be propagated when
    changes occur (with appropriate change collision management). You also want
    to incorporate some form of isolation in your framework, to try to avoid
    permitting ill-behaved clients from crashing services. A global logging
    service for all services is handy - its a bear traipsing through and
    interleaving among log files from each separate service node - but this too
    can be a drag on performance and robustness. And also, don't forget admin
    and monitoring utilities - some op center somewhere is going to have to
    monitor this whole mess^H^H^H^Hsystem, and would like an early warning
    before the one critical bottleneck resource decides to croak at 2a.m. on a
    Sunday.

    -- Paul
    Paul McGuire, May 19, 2006
    #2
    1. Advertising

  3. Paul McGuire wrote:
    >>

    >
    > What you describe sounds very much like pyro, which should probably be added
    > to your list.
    >

    Pyro looks nothing short of amazing. I will definitely spend some time
    studying this technology as well. Does anyone here use pyro or any of
    the other technologies enough to comment on them?

    ..c


    --

    Carl J. Van Arsdall

    Build and Release
    MontaVista Software
    Carl J. Van Arsdall, May 19, 2006
    #3
  4. >>>>> "Carl J. Van Arsdall" <> (CJVA) wrote:

    >CJVA> Hey everyone, another question for the list. In particular i'm
    >CJVA> looking for comments on some of the distributed technologies
    >CJVA> supported in python. Specifically, I'm looking at XML-RPC, RPyC,
    >CJVA> CORBA, and Twisted.


    Although it doesn't cover exactly the technologies mentioned above
    the following article may be interesting for you:

    Python Web services developer: Messaging technologies compared
    http://www.ibm.com/developerworks/library/ws-pyth9/

    --
    Piet van Oostrum <>
    URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
    Private email:
    Piet van Oostrum, May 25, 2006
    #4
  5. Carl J. Van Arsdall

    Guest

    Piet> Python Web services developer: Messaging technologies compared
    Piet> http://www.ibm.com/developerworks/library/ws-pyth9/

    Note a couple things. One, the article is four years old. You can't assume
    the various technologies have remained static since then. Two, the authors
    apparently didn't use sgmlop (at least it's not mentioned) to boost the
    decode performance of XML-RPC.

    Skip
    , May 25, 2006
    #5
  6. Carl J. Van Arsdall

    Guest

    Carl J. Van Arsdall wrote:
    > Hey everyone, another question for the list. In particular i'm looking
    > for comments on some of the distributed technologies supported in
    > python. Specifically, I'm looking at XML-RPC, RPyC, CORBA, and Twisted.
    >
    > Before you offer any comments let me talk about what i'm doing a little
    > bit. Right now I'm creating an execution framework. The framework aims
    > to allow users to create a modular system by which engineers can
    > mitigate some of the issues seen in large complex systems. By using
    > modules the system hopes to allow the system to change but allow
    > engineers to re-use as much code as possible. By using the modules in a
    > framework we hope to encapsulate the entire system to that it is
    > controllable in the sense that we can generate and catch errors anywhere
    > in the system and exit gracefully, things like that. There's a lot more
    > to it but the motivation spawn from our system where the system kept
    > needing to adapt to support changes in our company. These changes over
    > time have resulted in a large number of "hacks". The hope of the
    > modules is to make the code a lot cleaner and organized so that the
    > system can operate in whatever way it needs to without having so many
    > hacks. This system I use also operates in a primitive distributed
    > environment. Currently, these are ssh calls to other nodes that then
    > execute other python scripts and shell commands. This is the next piece
    > I'm interested in moving away from.
    >
    > With all that being said, I've come across those four technologies so
    > far and I've been looking into them. I was wondering if anyone had any
    > comments, pros/cons, or suggestions of other distributed technologies to
    > take a look at. I intend to integrate the distributed technology
    > directly into the framework making it easy for developers to distribute
    > tasks. Anyhow, any comments would be greatly appreciated! I also hope
    > to release this framework as an open-source project when its finished,
    > so again I community input is important.
    >
    > Thanks!
    >
    > .c
    >
    >
    > --
    >
    > Carl J. Van Arsdall
    >
    > Build and Release
    > MontaVista Software


    RPyC !
    I personnaly used it (and its predecessor, pyinvoke) for:
    * a testing system that controls several computers
    * drawing graphs with matplotlib while avoiding its memory leaks
    * distributed calculation
    * controlling embedded python applications from my favorite python
    shell (not the embedded one), also allowing me to control several
    instances of the application.
    it's so easy to use that there's was no effort require to do any of
    those!

    it's pretty self-explanatory, so I'll only say this:
    it's Transparent. you run remote code and access remote objects just as
    you would locally. it's minimal, and complete.
    I highly recommend it!
    , May 26, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brian Birtle
    Replies:
    2
    Views:
    1,989
    John Saunders
    Oct 16, 2003
  2. Replies:
    0
    Views:
    1,091
  3. Monk
    Replies:
    10
    Views:
    1,433
    Michael Wojcik
    Apr 20, 2005
  4. Zed Shaw
    Replies:
    0
    Views:
    345
    Zed Shaw
    Sep 29, 2010
  5. Replies:
    4
    Views:
    572
    Dr John Stockton
    Jun 3, 2006
Loading...

Share This Page