Reducing cache/buffer for faster display

Discussion in 'Python' started by Rikishi42, Sep 27, 2012.

  1. Rikishi42

    Rikishi42 Guest

    I have these 2 scripts that are very heavy on the file i/o, consume a very
    reasonable amount of cpu and output their counters at a - very - relaxed
    pace to the console. The output is very simply done using something like:

    print "files:", nFiles, "\r",


    Yet alltough there is no real reason for it, even a pace of a print every
    10-30 secs will be cached, only to actually show an output update every 1-2
    min or so.

    When I run the scripts with "python -u myscript.py", the output is complete,
    very speedy and without any kind of impact on the actual work being one.


    Can this option be called from within the script? Or is there another option
    to make the display "a bit" speedier ?


    Runnning Python 2.7.3, but it seems to me I've allready had this problem a
    long, long time ago with other releases.



    --
    When in doubt, use brute force.
    -- Ken Thompson
     
    Rikishi42, Sep 27, 2012
    #1
    1. Advertising

  2. On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <> wrote:
    > I have these 2 scripts that are very heavy on the file i/o, consume a very
    > reasonable amount of cpu and output their counters at a - very - relaxed
    > pace to the console. The output is very simply done using something like:
    >
    > print "files:", nFiles, "\r",
    >
    >
    > Yet alltough there is no real reason for it, even a pace of a print every
    > 10-30 secs will be cached, only to actually show an output update every 1-2
    > min or so.


    Yup! Just add a call to sys.stdout.flush() after each print.

    ChrisA
     
    Chris Angelico, Sep 27, 2012
    #2
    1. Advertising

  3. Rikishi42

    John Gordon Guest

    In <> Chris Angelico <> writes:

    > On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <> wrote:
    > > I have these 2 scripts that are very heavy on the file i/o, consume a very
    > > reasonable amount of cpu and output their counters at a - very - relaxed
    > > pace to the console. The output is very simply done using something like:
    > >
    > > print "files:", nFiles, "\r",
    > >
    > >
    > > Yet alltough there is no real reason for it, even a pace of a print every
    > > 10-30 secs will be cached, only to actually show an output update every 1-2
    > > min or so.


    > Yup! Just add a call to sys.stdout.flush() after each print.


    Isn't terminal output line-buffered? I don't understand why there would
    be an output delay. (Unless the "\r" is messing things up...)

    --
    John Gordon A is for Amy, who fell down the stairs
    B is for Basil, assaulted by bears
    -- Edward Gorey, "The Gashlycrumb Tinies"
     
    John Gordon, Sep 27, 2012
    #3
  4. On Fri, Sep 28, 2012 at 8:25 AM, John Gordon <> wrote:
    > Isn't terminal output line-buffered? I don't understand why there would
    > be an output delay. (Unless the "\r" is messing things up...)


    This is a classic progress-indication case, which does indeed mess up
    line-buffering. The carriage return (and no line feed, done in the
    Python 2 style of a trailing comma) puts the cursor back to the
    beginning of the line, ready to overwrite, and ripe for one of those
    old favorite incomplete overwrite errors - if nFiles monotonically
    increases, it's fine, but if it decreases, the display can get ugly.

    ChrisA
     
    Chris Angelico, Sep 27, 2012
    #4
  5. Rikishi42

    Rikishi42 Guest

    On 2012-09-27, Chris Angelico <> wrote:
    > On Fri, Sep 28, 2012 at 8:25 AM, John Gordon <> wrote:
    >> Isn't terminal output line-buffered? I don't understand why there would
    >> be an output delay. (Unless the "\r" is messing things up...)

    >
    > This is a classic progress-indication case, which does indeed mess up
    > line-buffering. The carriage return (and no line feed, done in the
    > Python 2 style of a trailing comma) puts the cursor back to the
    > beginning of the line, ready to overwrite, and ripe for one of those
    > old favorite incomplete overwrite errors - if nFiles monotonically
    > increases, it's fine, but if it decreases, the display can get ugly.


    True, but that wasn't the problem here. The updates where. Thanks for the
    given answer, I'll try it.

    The scripts in question only increase numbers. But should that not be the
    case, solutions are simple enough. The numbers can be formatted to have a
    fixed size. In the case of random line contents (a list of filesnames, say)
    it's enough to create an output function that is aware of the length of the
    previously printed line, and add enough spaces to the current one to wipe
    exess content.


    Thanks again for the suggestion.


    --
    When in doubt, use brute force.
    -- Ken Thompson
     
    Rikishi42, Sep 28, 2012
    #5
  6. Rikishi42

    Rikishi42 Guest

    On 2012-09-27, Chris Angelico <> wrote:
    > On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <> wrote:
    >> I have these 2 scripts that are very heavy on the file i/o, consume a very
    >> reasonable amount of cpu and output their counters at a - very - relaxed
    >> pace to the console. The output is very simply done using something like:
    >>
    >> print "files:", nFiles, "\r",
    >>
    >>
    >> Yet alltough there is no real reason for it, even a pace of a print every
    >> 10-30 secs will be cached, only to actually show an output update every 1-2
    >> min or so.

    >
    > Yup! Just add a call to sys.stdout.flush() after each print.


    Update: tried it, ran it, I love it.

    Thanks !


    --
    When in doubt, use brute force.
    -- Ken Thompson
     
    Rikishi42, Sep 28, 2012
    #6
  7. On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <>
    declaimed the following in gmane.comp.python.general:

    >
    > Isn't terminal output line-buffered? I don't understand why there would
    > be an output delay. (Unless the "\r" is messing things up...)


    It's the trailing , The \r is being used to reset to the
    beginning of the console line, but the comma "says" more output for
    /this/ line will be coming... So no output until explicitly flushed, or
    a new-line is issued.
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
     
    Dennis Lee Bieber, Sep 28, 2012
    #7
  8. On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 <> wrote:
    > The scripts in question only increase numbers. But should that not be the
    > case, solutions are simple enough. The numbers can be formatted to have a
    > fixed size. In the case of random line contents (a list of filesnames, say)
    > it's enough to create an output function that is aware of the length of the
    > previously printed line, and add enough spaces to the current one to wipe
    > exess content.


    Yep, that's a pretty effective way to do it. One simple method to it
    is to format the whole string as a single whole, then left justify it
    in a field of (say) 79 characters, and output that:

    msg = "Progress: %d%% (%d/%d)... %s" % (done*100/total, done, total,
    current_file)
    print msg.ljust(79)+"\r",
    sys.stdout.flush()

    ChrisA
     
    Chris Angelico, Sep 28, 2012
    #8
  9. Rikishi42

    Rikishi42 Guest

    On 2012-09-28, Dennis Lee Bieber <> wrote:
    > On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <>
    > declaimed the following in gmane.comp.python.general:
    >
    >>
    >> Isn't terminal output line-buffered? I don't understand why there would
    >> be an output delay. (Unless the "\r" is messing things up...)

    >
    > It's the trailing , The \r is being used to reset to the
    > beginning of the console line, but the comma "says" more output for
    > /this/ line will be coming... So no output until explicitly flushed, or
    > a new-line is issued.


    Well, the \r seems to be the problem, allright.
    But output was not completely blocked, just delayed a very long time.

    So perhaps flushing and a sending a newline aren't the only triggers for
    output. Perhaps there's a maximum delay or a maximum cumulated size, and
    the output is flushed when such a limit is reached.

    Anyway, that's mainly academic. I doubt there will be a correction to
    that behaviour.

    --
    When in doubt, use brute force.
    -- Ken Thompson
     
    Rikishi42, Sep 29, 2012
    #9
  10. Rikishi42

    Rikishi42 Guest

    On 2012-09-28, Chris Angelico <> wrote:
    > On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 <> wrote:
    >> The scripts in question only increase numbers. But should that not be the
    >> case, solutions are simple enough. The numbers can be formatted to have a
    >> fixed size. In the case of random line contents (a list of filesnames, say)
    >> it's enough to create an output function that is aware of the length of the
    >> previously printed line, and add enough spaces to the current one to wipe
    >> exess content.

    >
    > Yep, that's a pretty effective way to do it. One simple method to it
    > is to format the whole string as a single whole, then left justify it
    > in a field of (say) 79 characters, and output that:
    >
    > msg = "Progress: %d%% (%d/%d)... %s" % (done*100/total, done, total,
    > current_file)
    > print msg.ljust(79)+"\r",
    > sys.stdout.flush()


    Mmm, I allmost went for that. It's elegant, simple and clear. But there's
    one drawback: I usually reduce the terminal's window to take up less desktop
    surface during those long runs.
    So fixing it to 79 chars won't do. And I'm not even tempted to go for a
    detection of the width of the terminal from within the script. The idea is
    after all to keep the scripts simple (syntax) and light (execution).

    Well, good night everyone.

    --
    When in doubt, use brute force.
    -- Ken Thompson
     
    Rikishi42, Sep 29, 2012
    #10
  11. Rikishi42

    Hans Mulder Guest

    On 29/09/12 02:20:50, Rikishi42 wrote:
    > On 2012-09-28, Dennis Lee Bieber <> wrote:
    >> On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <>
    >> declaimed the following in gmane.comp.python.general:
    >>
    >>>
    >>> Isn't terminal output line-buffered? I don't understand why there would
    >>> be an output delay. (Unless the "\r" is messing things up...)

    >>
    >> It's the trailing , The \r is being used to reset to the
    >> beginning of the console line, but the comma "says" more output for
    >> /this/ line will be coming... So no output until explicitly flushed, or
    >> a new-line is issued.

    >
    > Well, the \r seems to be the problem, allright.
    > But output was not completely blocked, just delayed a very long time.
    >
    > So perhaps flushing and a sending a newline aren't the only triggers for
    > output. Perhaps there's a maximum delay or a maximum cumulated size, and
    > the output is flushed when such a limit is reached.


    There's a maximum cumulated size; it's called the buffer size.
    Output goes into a buffer, and when the buffer is full, it's
    printed all at once.

    One way to avoid it, is to use an unbuffered stream.

    Another, more efficient, way to avoid it, is to invoke the
    stream's .flush() method after writing to it.

    > Anyway, that's mainly academic. I doubt there will be a correction to
    > that behaviour.


    It's an optimization. When it was invented, 40 years ago, it was a
    really necessary to do this, to get something resembling performance.

    The performance of a system without stream buffering would probably
    be tolerable on modern hardware. But the people maintaining Python
    are unlikely to cut out buffering, because few people would benefit
    (yours is pretty much the only use case where buffereing hurts) and
    some would suffer (those who write many short strings to a disk file).


    Hope this helps,

    -- HansM
     
    Hans Mulder, Sep 29, 2012
    #11
  12. On Sat, 29 Sep 2012 02:20:50 +0200, Rikishi42 <>
    declaimed the following in gmane.comp.python.general:


    >
    > Well, the \r seems to be the problem, allright.
    > But output was not completely blocked, just delayed a very long time.
    >

    Of course -- once the output buffer was filled, the system would
    output the data...
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
     
    Dennis Lee Bieber, Sep 29, 2012
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Just D.

    CACHE, BUFFER, system bug or what?

    Just D., Sep 7, 2005, in forum: ASP .Net
    Replies:
    9
    Views:
    899
    Damien
    Sep 8, 2005
  2. Raja
    Replies:
    12
    Views:
    24,723
    John Harrison
    Jun 21, 2004
  3. Replies:
    2
    Views:
    639
    sergejusz
    Mar 26, 2007
  4. Neal Becker

    buffer creates only read-only buffer?

    Neal Becker, Jan 8, 2009, in forum: Python
    Replies:
    0
    Views:
    429
    Neal Becker
    Jan 8, 2009
  5. xingye
    Replies:
    9
    Views:
    301
    Michael Lu
    Apr 19, 2004
Loading...

Share This Page