Multi-threaded printing and UIErrors in Windows

Discussion in 'Python' started by Roger Binns, Apr 22, 2004.

  1. Roger Binns

    Roger Binns Guest

    My program has a random smattering of print statements
    scattered throughout that are mainly a debugging aid.
    They are typically things like this:

    print "entering key mode"

    The code is multithreaded, and the printing typically
    happens in both the main thread and a worker thread
    doing serial port IO. The problem I am experiencing
    when the program is packaged up with py2exe is that
    exceptions happen:

    IOError: [Errno 9] Bad file descriptor

    This can happen in either thread. Why is this
    happening? It doesn't happen on Linux or Mac,
    and hasn't happened when not using py2exe.

    Python 2.3, wxPython 2.4, py2exe 0.5 (and the
    previous version).

    Roger
    Roger Binns, Apr 22, 2004
    #1
    1. Advertising

  2. Roger Binns

    David Fraser Guest

    Roger Binns wrote:
    > My program has a random smattering of print statements
    > scattered throughout that are mainly a debugging aid.
    > They are typically things like this:
    >
    > print "entering key mode"
    >
    > The code is multithreaded, and the printing typically
    > happens in both the main thread and a worker thread
    > doing serial port IO. The problem I am experiencing
    > when the program is packaged up with py2exe is that
    > exceptions happen:
    >
    > IOError: [Errno 9] Bad file descriptor
    >
    > This can happen in either thread. Why is this
    > happening? It doesn't happen on Linux or Mac,
    > and hasn't happened when not using py2exe.
    >
    > Python 2.3, wxPython 2.4, py2exe 0.5 (and the
    > previous version).
    >
    > Roger
    >
    >

    sys.stdout is not threadsafe - try wrapping each call with a mutext and
    see if that fixes it...
    David Fraser, Apr 23, 2004
    #2
    1. Advertising

  3. Roger Binns

    Roger Binns Guest

    Re: Multi-threaded printing and IOErrors in Windows

    David Fraser wrote:
    > sys.stdout is not threadsafe - try wrapping each call with a mutext and
    > see if that fixes it...


    Having to wrap hundreds of random print statements just using for
    debugging seems really silly. Note that I don't care if the
    output gets garbled. (It doesn't on Mac or Linux ever).

    And I am only hitting this problem on Windows when wrapped
    with py2exe. If running outside of py2exe there is never
    any problem.

    When using py2exe, the stdout doesn't go anywhere anyway
    (it is gui app not a console app).

    The bizarre thing is that the IOError is very infrequent,
    but very annoying when it happens.

    Roger
    Roger Binns, Apr 23, 2004
    #3
  4. Re: Multi-threaded printing and IOErrors in Windows

    "Roger Binns" <> writes:

    > David Fraser wrote:
    >> sys.stdout is not threadsafe - try wrapping each call with a mutext and
    >> see if that fixes it...

    >
    > Having to wrap hundreds of random print statements just using for
    > debugging seems really silly. Note that I don't care if the
    > output gets garbled. (It doesn't on Mac or Linux ever).
    >
    > And I am only hitting this problem on Windows when wrapped
    > with py2exe. If running outside of py2exe there is never
    > any problem.
    >
    > When using py2exe, the stdout doesn't go anywhere anyway
    > (it is gui app not a console app).
    >
    > The bizarre thing is that the IOError is very infrequent,
    > but very annoying when it happens.


    If you have a small program which allows to reproduce this, I'll look
    into it.

    Thomas
    Thomas Heller, Apr 23, 2004
    #4
  5. Roger Binns

    David Bolen Guest

    Re: Multi-threaded printing and IOErrors in Windows

    "Roger Binns" <> writes:

    > When using py2exe, the stdout doesn't go anywhere anyway
    > (it is gui app not a console app).
    >
    > The bizarre thing is that the IOError is very infrequent,
    > but very annoying when it happens.


    Have you tried replacing sys.stdout with your own object that just
    throws away anything it is given? If you're not getting the output
    anywhere anyway, it would avoid any attempt to actually use the
    underlying I/O facilities. You could conditionalize this code based
    on whether or not you were running under py2exe.

    I know there can be an issue under Windows under some startup
    scenarios where you don't actually have opened O/S filesystem handles
    for the handles underlying stdin/stdout (which can cause this sort of
    problem), such as running as a service, but haven't normally had this
    problem with GUI applications. Although it may matter on how you are
    starting the application (from a command line, the start menu, as a
    service, etc...), since it will inherit its parent environment.

    If that's the problem, I'd expect it to happen any time actual I/O was
    attempted. However, since there is probably some buffering going on
    beneath the level of print within the underlying C library, maybe the
    randomness of the error you are seeing is due to how long it takes to
    print enough data to cause the C library to attempt to flush the data
    to the underlying O/S filesystem handle. If that were the case, doing
    some test prints in your code of very long strings (the C buffer is
    probably anywhere from 2-8K) should be able to reproduce the problem
    more reliably.

    -- David
    David Bolen, Apr 23, 2004
    #5
  6. Roger Binns

    David Bolen Guest

    Re: Multi-threaded printing and IOErrors in Windows

    "Roger Binns" <> writes:

    > David Fraser wrote:
    > > sys.stdout is not threadsafe - try wrapping each call with a mutext and
    > > see if that fixes it...

    >
    > Having to wrap hundreds of random print statements just using for
    > debugging seems really silly. Note that I don't care if the
    > output gets garbled. (It doesn't on Mac or Linux ever).


    To follow up on this point to my previous reply - I've never found
    sys.stdout not to be threadsafe, at least in terms of how I mean
    "threadsafe" - that is generating exceptions or crashes.

    If instead, threadsafe is meant to imply atomic I/O (guaranteed output
    of entire I/O as a unit) which is a broader definition, then yes, it
    is definitely true that multiple threads all using sys.stdout may have
    their output interleaved, particularly when you are using the 'print'
    statement, since a single print statement generates multiple
    individual writes to sys.stdout, generally for each argument, as well
    as the trailing newline.

    -- David
    David Bolen, Apr 23, 2004
    #6
  7. Roger Binns

    Roger Binns Guest

    Re: Multi-threaded printing and IOErrors in Windows (py2exe)

    Thomas Heller wrote:
    > "Roger Binns" <> writes:
    > > The bizarre thing is that the IOError is very infrequent,
    > > but very annoying when it happens.

    >
    > If you have a small program which allows to reproduce this, I'll look
    > into it.


    I cannot get it to repeat with test programs I have written (and boy have
    I tried). Not very many users reported the problem. It is probably
    some sort of race condition somewhere, and possibly only occurs
    on slower machines. My real program has many wxPython OnIdle handlers
    which is predominantly where the error shows up combined with a background
    thread doing heavy serial port activity to a cell phone, also doing
    prints.

    Is it possible for py2exe to replace stdout with something that
    just ignores all writes, as opposed to None (or is that what
    is already happening).

    Roger
    Roger Binns, May 8, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. David
    Replies:
    0
    Views:
    530
    David
    Sep 24, 2003
  2. Johny Franslay
    Replies:
    1
    Views:
    707
    Steve Horsley
    Mar 1, 2004
  3. Tim Black
    Replies:
    1
    Views:
    1,080
    Alan Kennedy
    Aug 3, 2004
  4. ian douglas
    Replies:
    2
    Views:
    971
    Randy Howard
    Jul 30, 2004
  5. Tim Hollingsworth

    Multi-threaded C++, new, and xmalloc

    Tim Hollingsworth, Nov 23, 2006, in forum: Ruby
    Replies:
    4
    Views:
    117
    Paul Brannan
    Nov 24, 2006
Loading...

Share This Page