RE: Embedding Python, threading and scalability

Discussion in 'Python' started by Simon Wittber (Maptek), Jul 11, 2003.

  1. [snip]
    >...the way to scale Python in a threaded environment is to call out to

    a C >extension that releases the GIL.
    [snip]

    To write scalable applications in Python, one must write the
    'scalabilty-required' parts n C.

    Does anyone else see this as a problem?

    Sw.
     
    Simon Wittber (Maptek), Jul 11, 2003
    #1
    1. Advertising

  2. Simon Wittber (Maptek)

    Donn Cave Guest

    Quoth "Simon Wittber (Maptek)" <>:
    | [snip]
    | >...the way to scale Python in a threaded environment is to call out to
    | a C >extension that releases the GIL.
    | [snip]
    |
    | To write scalable applications in Python, one must write the
    | 'scalabilty-required' parts n C.
    |
    | Does anyone else see this as a problem?

    Is it a problem?

    I don't know all the reasons why an application might want to
    compute in parallel on multiple CPUs, but I am guessing that
    ordinarily it's about processor intensive computations. That
    isn't really Python's strongest point - you want to write the
    heavy duty computing in C if you want speed.

    If it's any consolation, I believe the ocaml Objective CAML
    thread implementation has the same kind of global lock, even
    though it does compile to efficient native code and would
    otherwise be an attractive candidate for compute intensive
    applications. Don't take my word for it, but that's how it
    looks to me. Regardless of how grave the problem may be,
    if there's no practical fix, we live with it.

    Donn Cave,
     
    Donn Cave, Jul 11, 2003
    #2
    1. Advertising

  3. Simon Wittber (Maptek)

    Alan Kennedy Guest

    [Aahz]

    >>...the way to scale Python in a threaded environment is to call out
    >> to a C extension that releases the GIL.


    Simon Wittber (Maptek) wrote:

    > To write scalable applications in Python, one must write the
    > 'scalabilty-required' parts n C.


    Or Pyrex? Or Boost?

    Do either of these products release the GIL when control is
    transferred to the C/C++ extensions?

    > Does anyone else see this as a problem?


    Comparatively, perhaps. But certainly less of a problem than writing
    your entire program in C++, for example.

    Perhaps easiest to write the whole thing in jython, if the jython free
    threading claim is true? Ype? Aahz? I'd really like to know the truth
    of the assertion that jython will utilise all processors in a
    multi-processor box.

    regards,

    --
    alan kennedy
    -----------------------------------------------------
    check http headers here: http://xhaus.com/headers
    email alan: http://xhaus.com/mailto/alan
     
    Alan Kennedy, Jul 11, 2003
    #3
  4. Simon Wittber (Maptek)

    Aahz Guest

    In article <>,
    Alan Kennedy <> wrote:
    >Simon Wittber (Maptek) wrote:
    >>
    >> To write scalable applications in Python, one must write the
    >> 'scalabilty-required' parts n C.

    >
    >Or Pyrex? Or Boost?
    >
    >Do either of these products release the GIL when control is
    >transferred to the C/C++ extensions?


    Not automatically, that's for certain; dunno if either provides any
    features to make the job easier.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Not everything in life has a clue in front of it...." --JMS
     
    Aahz, Jul 11, 2003
    #4
  5. Simon Wittber (Maptek)

    Aahz Guest

    In article <>,
    Simon Wittber (Maptek) <> wrote:
    >
    >To write scalable applications in Python, one must write the
    >'scalabilty-required' parts n C.
    >
    >Does anyone else see this as a problem?


    Not particularly. Most threading at the application level is done for
    one or more of three purposes:

    * Allowing background work and fast response in a GUI application

    * Scalable I/O

    * Autonomous sections of code for algorithmic simplicity (e.g.
    simulations)

    Python does quite well at all three out of the box (the second because
    all standard Python I/O modules release the GIL, as do most 3rd-party
    extensions that deal with I/O (e.g. mxODBC)). The only thing Python
    doesn't do is computational threading, and Python's overhead makes it a
    poor choice for that purpose. Finally, there are so many distributed
    computing solutions that multiple processes are a viable technique for
    managing computational threading.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Not everything in life has a clue in front of it...." --JMS
     
    Aahz, Jul 11, 2003
    #5
  6. Simon Wittber (Maptek)

    Paul Rubin Guest

    (Aahz) writes:
    > Not particularly. Most threading at the application level is done for
    > one or more of three purposes:
    >
    > * Allowing background work and fast response in a GUI application
    >
    > * Scalable I/O
    >
    > * Autonomous sections of code for algorithmic simplicity (e.g.
    > simulations)


    Um, concurrent access by multiple clients for server applications?

    > Python does quite well at all three out of the box (the second because
    > all standard Python I/O modules release the GIL, as do most 3rd-party
    > extensions that deal with I/O (e.g. mxODBC)). The only thing Python
    > doesn't do is computational threading, and Python's overhead makes it a
    > poor choice for that purpose. Finally, there are so many distributed
    > computing solutions that multiple processes are a viable technique for
    > managing computational threading.


    Yeah, but now you need complicated and possibly slow mechanisms for
    sharing global state between processes.
     
    Paul Rubin, Jul 11, 2003
    #6
  7. Paul Rubin wrote:
    > (Aahz) writes:
    >
    >>Not particularly. Most threading at the application level is done for
    >>one or more of three purposes:
    >>
    >>* Allowing background work and fast response in a GUI application
    >>
    >>* Scalable I/O
    >>
    >>* Autonomous sections of code for algorithmic simplicity (e.g.
    >>simulations)

    >
    >
    > Um, concurrent access by multiple clients for server applications?


    Exactly, that is what I use threads for in Pyro. Without threads,
    each of the client's remote method invocations has to wait in line
    before being accepted and processed (there can be many clients for
    one server object).


    --Irmen
     
    Irmen de Jong, Jul 11, 2003
    #7
  8. Simon Wittber (Maptek)

    Aahz Guest

    In article <>,
    Paul Rubin <http://> wrote:
    > (Aahz) writes:
    >>
    >> Not particularly. Most threading at the application level is done for
    >> one or more of three purposes:
    >>
    >> * Allowing background work and fast response in a GUI application
    >>
    >> * Scalable I/O
    >>
    >> * Autonomous sections of code for algorithmic simplicity (e.g.
    >> simulations)

    >
    >Um, concurrent access by multiple clients for server applications?


    That's functionally an I/O thing, combined possibly with background
    work.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Not everything in life has a clue in front of it...." --JMS
     
    Aahz, Jul 11, 2003
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. tharma
    Replies:
    4
    Views:
    508
    Jason Kester
    Sep 13, 2005
  2. Wenning Qiu
    Replies:
    7
    Views:
    528
  3. Jeff Epler
    Replies:
    3
    Views:
    338
    Afanasiy
    Jul 10, 2003
  4. David Harrison
    Replies:
    3
    Views:
    327
    David Harrison
    May 10, 2005
  5. Replies:
    0
    Views:
    339
Loading...

Share This Page