Re: [Python-Dev] Making python C-API thread safe (try 2)

Discussion in 'Python' started by Christopher A. Craig, Sep 12, 2003.

  1. Harri Pesonen <> writes:

    > After sleeping over night, I think that I got it. :) The simple solution is,
    > that each thread created in Python gets its own independent interpreter state
    > as well. And there could be a separate thread-global interpreter state for
    > shared memory access. Access to this global state would always be
    > synchronized. There could even be multiple named global states, so that the
    > thread interlocking could be minimized. The python syntax for creating objects
    > in this global state should be invented:
    >
    >
    > synchronize a = "abcd"
    >
    > Also when creating the new thread, it's arguments would be copied from the
    > creating state to the new state.
    >
    >
    > What does it sound? Of course it would be incompatible with the current
    > threading system in Python, but it would be totally multithreading, no global
    > interpreter lock needed. It would be faster than current Python, there would
    > be no need to free or acquire the lock when calling OS functions, and no need
    > to check how many byte codes have been processed, etc.
    >


    Couldn't you do this now with multiple processes and the shm module?

    --
    Christopher A. Craig <>
    "I affirm brethren by the boasting in you which I have in Christ Jesus
    our Lord, I die daily" I Cor 15:31 (NASB)
     
    Christopher A. Craig, Sep 12, 2003
    #1
    1. Advertising

  2. Christopher A. Craig

    Alan Kennedy Guest

    Harri Pesonen <> writes:
    >> The simple solution is,
    >> that each thread created in Python gets its own independent
    >> interpreter state as well. And there could be a separate
    >> thread-global interpreter state for shared memory access. Access
    >> to this global state would always be
    >> synchronized.


    "Christopher A. Craig" wrote:
    > Couldn't you do this now with multiple processes and the shm module?


    Yes, you can store representations of python objects in shared memory,
    e.g. pickles, etc: an in-memory database, with indices, etc.

    So when you want to access a "shared" object, you go to the shared
    store, retrieve the pickle, unpickle it and use it, making sure to
    repickle any changes and store them back in the shared memory. Which
    might be cumbersome and inefficient, particularly if your goal is to
    maximise efficiency in a multi-processor situation.

    But if you want to store actual python objects in shared memory, and
    avoid all the pickling etc,then you have to change the interpreter so
    that it obtains the memory for new python objects from the shared
    memory pool instead of "local" memory.

    Which leads to problems with reference counting and garbage
    collection. These would have to take multiple processes into account:
    what happens when a process goes down? Should the objects allocated by
    it be destroyed? Or remain persistent? Until when? If a process died
    in an unclean fashion, it might not delete its references to objects
    cleanly.

    And it gets more complex again when you're dealing with users and
    permissions.

    I think that there are so many questions associated with the approach
    of sharing python objects through shared memory that it will probably
    remain an "application specific" technique for some time to come.

    Though doubtless some more knowledgable person than I will now
    contradict me. Please. :)

    --
    alan kennedy
    -----------------------------------------------------
    check http headers here: http://xhaus.com/headers
    email alan: http://xhaus.com/mailto/alan
     
    Alan Kennedy, Sep 12, 2003
    #2
    1. Advertising

  3. Alan Kennedy wrote:
    > Which leads to problems with reference counting and garbage
    > collection. These would have to take multiple processes into account:
    > what happens when a process goes down? Should the objects allocated by
    > it be destroyed? Or remain persistent? Until when? If a process died
    > in an unclean fashion, it might not delete its references to objects
    > cleanly.
    >
    > And it gets more complex again when you're dealing with users and
    > permissions.
    >
    > I think that there are so many questions associated with the approach
    > of sharing python objects through shared memory that it will probably
    > remain an "application specific" technique for some time to come.
    >
    > Though doubtless some more knowledgable person than I will now
    > contradict me. Please. :)


    I'd just like to point out that Steffen Viken Valvag has just recently
    (March 2003) researched this, found solutions, and written an
    implementation.

    http://poshmodule.sourceforge.net/posh/html/node4.html

    That page leaves a few questions about the distributed garbage
    collection unanswered, but it's a nice high-level overview.

    I don't know how stable Posh is, but if nothing else it is a convincing
    proof of concept.

    Shane
     
    Shane Hathaway, Sep 12, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shane Hathaway
    Replies:
    3
    Views:
    340
    Martin v. =?iso-8859-15?q?L=F6wis?=
    Sep 12, 2003
  2. Harri Pesonen
    Replies:
    32
    Views:
    853
    Christopher A. Craig
    Sep 17, 2003
  3. Brian Quinlan
    Replies:
    12
    Views:
    658
    Skip Montanaro
    Sep 17, 2003
  4. Gabriel Rossetti
    Replies:
    0
    Views:
    1,389
    Gabriel Rossetti
    Aug 29, 2008
  5. John Nagle
    Replies:
    5
    Views:
    509
    John Nagle
    Mar 12, 2012
Loading...

Share This Page