Buffering of sys.stdout and sys.stderr in python3 (and documentation)

Discussion in 'Python' started by Geoff Bache, Dec 9, 2011.

  1. Geoff Bache

    Geoff Bache Guest

    Hi all,

    Short version:

    I'm a bit confused in general as to the changes between python2 and
    python3 regarding how standard output and standard error do buffering.
    A few things seem to have changed and I've failed to find any
    documentation of how and why. Also, the meaning of "python -u" seems
    to have changed and the docs don't seem to reflect the new behaviour
    (and I can't find any docs about the change either)...

    Long version:

    From rude experiment it seems that:
    1) In Python 2.x, standard error was always unbuffered while standard
    output was buffered by default. In python3, both are buffered. In
    both cases, "buffered" means line-buffered when writing to the console
    and not line-buffered when redirected to files.
    2) In Python 2.x, the "-u" flag meant everything was totally
    unbuffered. In Python 3.x, it means that both stdout and stderr are
    line-buffered also when redirected to files.

    Are either of these changes documented anywhere? (1) seems important :
    it can lead to not seeing exception printouts, if stderr is redirected
    to a file and the program is subsequently terminated with SIGTERM. I
    just wasted quite a bit of time due to this situation...

    This is what the Python 3 docs have to say about the -u flag:

    "Force the binary layer of the stdin, stdout and stderr streams (which
    is available as their buffer attribute) to be unbuffered. The text I/O
    layer will still be line-buffered."

    The "still" seems misleading to me, as it is only relevant if writing
    to the console. It would be useful to contrast the behaviour with and
    without "-u" when writing to files I would say.

    Regards,
    Geoff Bache
     
    Geoff Bache, Dec 9, 2011
    #1
    1. Advertising

  2. Geoff Bache

    Terry Reedy Guest

    Re: Buffering of sys.stdout and sys.stderr in python3 (anddocumentation)

    On 12/9/2011 2:32 PM, Geoff Bache wrote:
    > Hi all,
    >
    > Short version:
    >
    > I'm a bit confused in general as to the changes between python2 and
    > python3 regarding how standard output and standard error do buffering.
    > A few things seem to have changed and I've failed to find any
    > documentation of how and why. Also, the meaning of "python -u" seems
    > to have changed and the docs don't seem to reflect the new behaviour
    > (and I can't find any docs about the change either)...
    >
    > Long version:
    >
    >> From rude experiment it seems that:

    > 1) In Python 2.x, standard error was always unbuffered while standard
    > output was buffered by default. In python3, both are buffered. In
    > both cases, "buffered" means line-buffered when writing to the console
    > and not line-buffered when redirected to files.
    > 2) In Python 2.x, the "-u" flag meant everything was totally
    > unbuffered. In Python 3.x, it means that both stdout and stderr are
    > line-buffered also when redirected to files.
    >
    > Are either of these changes documented anywhere? (1) seems important :
    > it can lead to not seeing exception printouts, if stderr is redirected
    > to a file and the program is subsequently terminated with SIGTERM. I
    > just wasted quite a bit of time due to this situation...
    >
    > This is what the Python 3 docs have to say about the -u flag:
    >
    > "Force the binary layer of the stdin, stdout and stderr streams (which
    > is available as their buffer attribute) to be unbuffered. The text I/O
    > layer will still be line-buffered."
    >
    > The "still" seems misleading to me, as it is only relevant if writing
    > to the console. It would be useful to contrast the behaviour with and
    > without "-u" when writing to files I would say.


    The difference from 2.x should be in What's New in 3.0, except that the
    new i/o module is in 2.6, so it was not exactly new. You might be able
    to find more in http://python.org/dev/peps/pep-3116/

    You *should* be able to find sufficient info in the 3.x docs. If, after
    you get other responses (or not), you think the docs need upgrading,
    open an issue on the tracker at bugs.python.org with suggestions as
    specific as possible, including changed or new lines of text based on
    your experience and experiments.

    --
    Terry Jan Reedy
     
    Terry Reedy, Dec 10, 2011
    #2
    1. Advertising

  3. Geoff Bache

    Geoff Bache Guest

    Hi Terry,

    > The difference from 2.x should be in What's New in 3.0, except that the
    > new i/o module is in 2.6, so it was not exactly new.


    The io module existed in 2.6, but it was not used by default for
    standard output and standard error. The only mention of this in
    "What's New in 3.0" is in the section marked for changes that were
    already in 2.6 (which is wrong in this case), and it notes only that
    io.TextIOWrapper is now used, but not what implications that has for
    its behaviour and backward compatibility.

    > You might be able
    > to find more inhttp://python.org/dev/peps/pep-3116/


    I skimmed through it but couldn't find anything relevant. It seems
    more "advanced" and implementation-focussed.

    >
    > You *should* be able to find sufficient info in the 3.x docs. If, after
    > you get other responses (or not), you think the docs need upgrading,
    > open an issue on the tracker at bugs.python.org with suggestions as
    > specific as possible, including changed or new lines of text based on
    > your experience and experiments.


    OK, I'll do that if nobody points me at some existing docs here.

    Regards,
    Geoff Bache
     
    Geoff Bache, Dec 10, 2011
    #3
  4. Geoff Bache

    Guest

    It's surprising and broken that stderr should be buffered in python3. python3 calls setvbuf(3) on stderr at startup to achieve this chuckle-headed behavior. It makes stderr line buffered if on a terminal, and fully bufferedif redirected to a log file. A fully buffered stderr is a very bad idea.

    This change goes against the C89 and C99 standards, 40+ years of precedent,and the behavior of nearly every other current programming language.

    Error messages will not show up in logs on time, and may be completely lostif the process is terminated before the buffer is flushed, e.g. with SIGTERM

    How about fixing this?

    Until then, we should write this in every script:

    sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)
     
    , Nov 12, 2013
    #4
  5. Geoff Bache

    Guest

    > sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)

    which unfortunately doesn't work! I guess will resort to python3 -u, although I don't want stdout to be unbuffered.
     
    , Nov 12, 2013
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Roman Neuhauser
    Replies:
    0
    Views:
    691
    Roman Neuhauser
    Apr 4, 2005
  2. Mitchell L Model

    sys.stdout vs. sys.stderr

    Mitchell L Model, Jan 11, 2010, in forum: Python
    Replies:
    2
    Views:
    600
    Nobody
    Jan 11, 2010
  3. Michel Albert
    Replies:
    5
    Views:
    1,507
    Daniel Dehennin
    Oct 24, 2012
  4. Frank Millman

    sys.stdout and Python3

    Frank Millman, Nov 23, 2013, in forum: Python
    Replies:
    0
    Views:
    126
    Frank Millman
    Nov 23, 2013
  5. Chris Angelico

    Re: sys.stdout and Python3

    Chris Angelico, Nov 23, 2013, in forum: Python
    Replies:
    2
    Views:
    166
    Chris Angelico
    Nov 24, 2013
Loading...

Share This Page