Problem with embedded python

Discussion in 'Python' started by Ugo Di Girolamo, Apr 26, 2005.

  1. I have the following code, that seems to make sense to me.


    However, it crashes about 1/3 of the times.


    My platform is Python 2.4.1 on WXP (I tried the release version from
    the msi and the debug version built by me, both downloaded today to
    have the latest version).


    The crash happens while the main thread is in Py_Finalize.
    I traced the crash to _Py_ForgetReference(op) in object.c at line 1847,
    where I have op->_ob_prev == NULL.


    What am I doing wrong? I'm definitely not too sure about the way I'm
    handling the GIL.


    Thanks in adv for any suggestion/ comment


    Cheers and ciao


    Ugo


    ////////////////////////// TestPyThreads.py //////////////////////////
    #include <windows.h>
    #include "Python.h"


    int main()
    {
    PyEval_InitThreads();
    Py_Initialize();
    PyGILState_STATE main_restore_state = PyGILState_UNLOCKED;
    PyGILState_Release(main_restore_state);


    // start the thread
    {
    PyGILState_STATE state = PyGILState_Ensure();
    int trash = PyRun_SimpleString(
    "import thread\n"
    "import time\n"
    "def foo():\n"
    " f = open('pippo.out', 'w', 0)\n"
    " i = 0;\n"
    " while 1:\n"
    " f.write('%d\\n'%i)\n"
    " time.sleep(0.01)\n"
    " i += 1\n"
    "t = thread.start_new_thread(foo, ())\n"
    );
    PyGILState_Release(state);
    }


    // wait 300 ms
    Sleep(300);


    PyGILState_Ensure();
    Py_Finalize();
    return 0;

    }
     
    Ugo Di Girolamo, Apr 26, 2005
    #1
    1. Advertising

  2. Ugo Di Girolamo

    Tom Cocagne Guest

    From looking at your example, it looks like you're making the problem FAR
    more difficult than it needs to be. The main thing to keep in mind is that
    Python threads do not correspond to operating system threads. In an
    application using a single OS-level thread, you can use as many Python
    threads as you like. The thread-switching mechanism in Python is to
    transfer control between threads after the execution of 100 bytecodes. The
    GIL is used to insure that only 1 operating system thread is executing
    bytecodes at any given time. This allows os threads to sleep on blocking
    calls (usually I/O) without necessarily stalling the interpreter.

    Your example appears to be written under the assumption that the spawned
    Python thread will operate concurrently with your main thread. That will
    not be the case. The PyRun_SimpleString() call will operate entirely within
    the main thread and will not return until the Python thread has completed
    operation (so no GIL manipulation is needed). The subsequent Sleep() call
    will simply halt the process for 300ms.

    Regular threading is difficult enough to get right without throwing the
    Python-thread / OS-thread interaction into the mix. It's a tough nut to
    crack so don't feel bad if it takes a while to get everything straight ;-)
    The quickest and most practical approach would probably be to read through
    some code that's similar to what you're shooting for. I'd suggest taking a
    look at some of the modules in the Python source code (particularly the
    network related ones) to get a better handle on how threads are managed.

    Cheers,

    Cocagne


    Ugo Di Girolamo wrote:

    > I have the following code, that seems to make sense to me.
    >
    >
    > However, it crashes about 1/3 of the times.
    >
    >
    > My platform is Python 2.4.1 on WXP (I tried the release version from
    > the msi and the debug version built by me, both downloaded today to
    > have the latest version).
    >
    >
    > The crash happens while the main thread is in Py_Finalize.
    > I traced the crash to _Py_ForgetReference(op) in object.c at line 1847,
    > where I have op->_ob_prev == NULL.
    >
    >
    > What am I doing wrong? I'm definitely not too sure about the way I'm
    > handling the GIL.
    >
    >
    > Thanks in adv for any suggestion/ comment
    >
    >
    > Cheers and ciao
    >
    >
    > Ugo
    >
    >
    > ////////////////////////// TestPyThreads.py //////////////////////////
    > #include <windows.h>
    > #include "Python.h"
    >
    >
    > int main()
    > {
    > PyEval_InitThreads();
    > Py_Initialize();
    > PyGILState_STATE main_restore_state = PyGILState_UNLOCKED;
    > PyGILState_Release(main_restore_state);
    >
    >
    > // start the thread
    > {
    > PyGILState_STATE state = PyGILState_Ensure();
    > int trash = PyRun_SimpleString(
    > "import thread\n"
    > "import time\n"
    > "def foo():\n"
    > " f = open('pippo.out', 'w', 0)\n"
    > " i = 0;\n"
    > " while 1:\n"
    > " f.write('%d\\n'%i)\n"
    > " time.sleep(0.01)\n"
    > " i += 1\n"
    > "t = thread.start_new_thread(foo, ())\n"
    > );
    > PyGILState_Release(state);
    > }
    >
    >
    > // wait 300 ms
    > Sleep(300);
    >
    >
    > PyGILState_Ensure();
    > Py_Finalize();
    > return 0;
    >
    > }
     
    Tom Cocagne, Apr 27, 2005
    #2
    1. Advertising

  3. Ugo Di Girolamo

    ugodiggi Guest

    Tom Cocagne wrote:
    > From looking at your example, it looks like you're making the problem

    FAR
    > more difficult than it needs to be. The main thing to keep in mind is

    that
    > Python threads do not correspond to operating system threads. In an
    > application using a single OS-level thread, you can use as many

    Python
    > threads as you like. The thread-switching mechanism in Python is to
    > transfer control between threads after the execution of 100

    bytecodes. The
    > GIL is used to insure that only 1 operating system thread is

    executing
    > bytecodes at any given time. This allows os threads to sleep on

    blocking
    > calls (usually I/O) without necessarily stalling the interpreter.
    >
    > Your example appears to be written under the assumption that the

    spawned
    > Python thread will operate concurrently with your main thread. That

    will
    > not be the case. The PyRun_SimpleString() call will operate entirely

    within
    > the main thread and will not return until the Python thread has

    completed
    > operation (so no GIL manipulation is needed). The subsequent Sleep()

    call
    > will simply halt the process for 300ms.
    >
    > Regular threading is difficult enough to get right without throwing

    the
    > Python-thread / OS-thread interaction into the mix. It's a tough nut

    to
    > crack so don't feel bad if it takes a while to get everything

    straight ;-)
    > The quickest and most practical approach would probably be to read

    through
    > some code that's similar to what you're shooting for. I'd suggest

    taking a
    > look at some of the modules in the Python source code (particularly

    the
    > network related ones) to get a better handle on how threads are

    managed.
    >
    > Cheers,
    >
    > Cocagne
    >

    <removed code>

    Tom,

    Thank you for your answer.

    This code is not the real "problem child", but a slimmed down version
    that seems to reproduce a problem that I see in a much larger system.

    My understanding of the python GIL works as follows:
    1) when I start the embedded python interpreter (Py_Initialize,
    PyEval_InitThreads), I own the GIL.
    So I need to release it for other threads to be able to do anything.
    I don't have other threads, so nothing is happening.

    2) when I want to use PyRun_SimpleString, I need to acquire the GIL
    before, and release it after. By doing this, no python thread will run
    while I'm running my simple string, regardless of anything because the
    C is owning the GIL.

    3) after I release the GIL, all the python threads are actually free to
    run, and they actually will. Since I'm on WXP, python threads ARE
    actually os threads, as I can see from my debugger.
    I do get my nice pippo.out file with the numbers from 0 to 29 or 30,
    which is coherent with the sleeps.

    4) when I want to shut down the whole thing, I acquire the GIL and then
    I call Py_Finalize, which should pretty much kill all the threads
    cleanly.

    The crash that I'm getting is during step 4, in the python thread,
    while in the main thread I'm calling Py_Finalize.
    Maybe I'm doing something wrong in one of the steps, but I cannot see
    what.

    My best guess is that the words from the ref manual:
    "The lock is also released and reacquired around potentially blocking
    I/O operations like reading or writing a file, so that other threads
    can run while the thread that requests the I/O is waiting for the I/O
    operation to complete. "
    are haunting me, but I'm not sure how.

    Thanks again

    Cheers & ciao

    Ugo
     
    ugodiggi, Apr 27, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. JM
    Replies:
    2
    Views:
    464
    Greg Chapman
    Aug 28, 2003
  2. Colin Paul Gloster
    Replies:
    48
    Views:
    1,892
    Colin Paul Gloster
    Apr 10, 2007
  3. Thomas Dodds

    Databind an embedded control in an embedded datagrid

    Thomas Dodds, Jul 26, 2004, in forum: ASP .Net Datagrid Control
    Replies:
    0
    Views:
    402
    Thomas Dodds
    Jul 26, 2004
  4. Trans
    Replies:
    11
    Views:
    303
    micathom
    Sep 5, 2007
  5. Num GG
    Replies:
    2
    Views:
    365
    Num GG
    Nov 17, 2008
Loading...

Share This Page