Re: Embedding Python, threading and scalability

Discussion in 'Python' started by Jeff Epler, Jul 10, 2003.

  1. Jeff Epler

    Jeff Epler Guest

    On Thu, Jul 10, 2003 at 11:14:47AM +0800, Simon Wittber (Maptek) wrote:
    > Seriously though, this is an issue, which is a major hurdle Python *has*
    > to cross if it is ever going to be seriously considered for use on large
    > projects, on large SMP hardware.


    Has anybody proposed "how to get there from here" for this problem
    (useful multithreading of non-blocking pure Python code)? I'm not
    bright enough to see how, that's for sure. Especially if you are
    talking about an incremental approach, not the mythical "Python 3000".
    What I mean is that you have to work this magic *and* somehow let
    existing modules written in C work, with at worst a recompile. (as
    someone who doesn't *need* threads, but works on a project with piles of
    Python modules written in C, that's my bias anyway)

    Someone nearby mentioned lua, but I don't know what it did for threading.
    Perl and tcl both seem to have taken the approach of having each thread
    be a distinct interpreter with nothing shared. While this means you
    never have to worry about locking against a reader or modifier in another
    thread, it means you might as well be using the processes that the angry
    Unix Gods gave us in the first place. <1/3 overstatement> I'm pretty sure
    that this approach has been explicitly ruled out by that other angry God,
    Guido van Rossum, anyway.

    I've written Python programs that use threads in a way that was expedient
    to get a user interface running without long seizures, while I've never
    written a thread in tcl or perl. OTOH if I'd had to treat everything
    as explicit message passing (a la tcl threading), I'd have just found
    another way (like "after idle" and "update" in tcl)

    Jython will give you Java's thread model today, for Python code, won't
    it? Back when I benchmarked it, Jython code and Python code ran at
    fairly similar speeds (in pybench), if only you had a machine that could
    keep the Jython interpreter from swapping...

    Jeff
     
    Jeff Epler, Jul 10, 2003
    #1
    1. Advertising

  2. Jeff Epler

    Donn Cave Guest

    Quoth Jeff Epler <>:
    | On Thu, Jul 10, 2003 at 11:14:47AM +0800, Simon Wittber (Maptek) wrote:
    |> Seriously though, this is an issue, which is a major hurdle Python *has*
    |> to cross if it is ever going to be seriously considered for use on large
    |> projects, on large SMP hardware.
    |
    | Has anybody proposed "how to get there from here" for this problem
    | (useful multithreading of non-blocking pure Python code)? I'm not
    | bright enough to see how, that's for sure. Especially if you are
    | talking about an incremental approach, not the mythical "Python 3000".
    | What I mean is that you have to work this magic *and* somehow let
    | existing modules written in C work, with at worst a recompile. (as
    | someone who doesn't *need* threads, but works on a project with piles of
    | Python modules written in C, that's my bias anyway)

    Ha - I knew I'd find an answer for this if I searched for "free threading" -
    and among other hits, I found this very thread! So in Andrew Dalke's words:

    > Ahh, the Global Interpreter Lock (GIL).
    >
    > Years ago, Greg Stein had a version of Python 1.4 running with no GIL.
    >
    > http://www.python.org/ftp/python/contrib-09-Dec-1999/System/threading.README
    >
    > Search for "free threading" to get more hits on this topic.


    At any rate, it sure isn't because Guido can't scare up an SMP machine.

    | I've written Python programs that use threads in a way that was expedient
    | to get a user interface running without long seizures, while I've never
    | written a thread in tcl or perl. OTOH if I'd had to treat everything
    | as explicit message passing (a la tcl threading), I'd have just found
    | another way (like "after idle" and "update" in tcl)

    Hm, not sure what you're alluding to here. I actually kind of like an
    I/O thread dispatch model myself, but I don't think Python cares either
    way nor would it if the GIL were abolished.

    | Jython will give you Java's thread model today, for Python code, won't
    | it? Back when I benchmarked it, Jython code and Python code ran at
    | fairly similar speeds (in pybench), if only you had a machine that could
    | keep the Jython interpreter from swapping...

    Don't know, but I noticed in another thread among the hits that Ype Kingma
    asserts that Jython does indeed support fine grained multithreading.

    Donn Cave,
     
    Donn Cave, Jul 10, 2003
    #2
    1. Advertising

  3. Jeff Epler

    Peter Hansen Guest

    Afanasiy wrote:
    >
    > On Thu, 10 Jul 2003 05:26:14 -0000, "Donn Cave" <> wrote:
    >
    > >Quoth Jeff Epler <>:

    >
    > >At any rate, it sure isn't because Guido can't scare up an SMP machine.

    >
    > Is this the absolute truth or just something people like to say?
    >
    > If it is true I am very surprised... I think getting Guido an SMP
    > machine will not be very difficult, and could well be very cheap.


    Note the double negative in the phrase "isn't because Guido can't".

    That means in effect "Guido can". Nobody's disagreeing with that.

    -Peter
     
    Peter Hansen, Jul 10, 2003
    #3
  4. Jeff Epler

    Afanasiy Guest

    On Thu, 10 Jul 2003 15:36:04 -0400, Peter Hansen <>
    wrote:

    >Afanasiy wrote:
    >>
    >> On Thu, 10 Jul 2003 05:26:14 -0000, "Donn Cave" <> wrote:
    >>
    >> >Quoth Jeff Epler <>:

    >>
    >> >At any rate, it sure isn't because Guido can't scare up an SMP machine.

    >>
    >> Is this the absolute truth or just something people like to say?
    >>
    >> If it is true I am very surprised... I think getting Guido an SMP
    >> machine will not be very difficult, and could well be very cheap.

    >
    >Note the double negative in the phrase "isn't because Guido can't".
    >
    >That means in effect "Guido can". Nobody's disagreeing with that.


    Yes, I read a previous post which said "Give Guido a SMP machine."
    and read this one as the same, my mistake.
     
    Afanasiy, Jul 10, 2003
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. tharma
    Replies:
    4
    Views:
    504
    Jason Kester
    Sep 13, 2005
  2. Wenning Qiu
    Replies:
    7
    Views:
    527
  3. Simon Wittber (Maptek)

    RE: Embedding Python, threading and scalability

    Simon Wittber (Maptek), Jul 11, 2003, in forum: Python
    Replies:
    7
    Views:
    388
  4. David Harrison
    Replies:
    3
    Views:
    325
    David Harrison
    May 10, 2005
  5. Replies:
    0
    Views:
    339
Loading...

Share This Page