Problems with sys.stout.flush()

[Joel Ross]

Hi all,

I'm using python 2.5 and trying to flush the sys.stout buffer with
sys.stout.flush(), but doesn't seem to work. Each time a line is printed
it appends the one before it I need to clear the output and write a
new output without appending the previous one. I have tried the -u
(unbuffered) option for python and a few examples from this site
http://stackoverflow.com/questions/107705/python-output-buffering

Code:

import sys
from time import sleep

def progress(number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
if percentage > 0:
char = char * percentage
sys.stdout.write(char)
sys.stdout.flush() #Not working correctly
sleep(2)

progress(40, 50, "*")
progress(30, 50, "*")
progress(20, 50, "*")
progress(10, 50, "*")
progress(2, 50, "*")

Regards

jross

 

[Carl Banks]

Hi all,

I'm using python 2.5 and trying to flush the sys.stout buffer with
sys.stout.flush(), but doesn't seem to work. Each time a line is printed
   it appends the one before it I need to clear the output and write a
new output without appending the previous one.

That's not how streams work, chief. Once you output (and flush)
something you can't un-output it.

What you probably want to do is to write a carriage return ("\r")
which usually causes the cursor to return to the beginning of the
line, so that any new text you write overwrites the old text.

This has nothing to do with flushing; flushing doesn't erase or clear
the old input. Flushing is usually needed for a different reason,
however, namely standard output doesn't actually get sent to the
console until it sees a newline ("\n") unless you flush the buffer.

Try to adapt this example to your problem:

for i in xrange(11):
sys.stdout.write('*'*(10-i) + ' '*i + '\r')
sys.stdout.flush()
time.sleep(2)


Notice that you have to print out spaces as well, to overwrite the
previous characters on the line (well, it might not be strictly needed
for a progress bar that only grows, but it's normally a good idea.

I have tried the -u
(unbuffered) option for python and a few examples from this sitehttp://stackoverflow.com/questions/107705/python-output-buffering

Code:

import sys
from time import sleep

def progress(number, total,  char):

     percentage = float(number*100)/total
     percentage = int(round(percentage))
     percentage = int(100 - percentage)
     if percentage > 0:
         char = char * percentage
         sys.stdout.write(char)
         sys.stdout.flush()      #Not working correctly
         sleep(2)

progress(40, 50, "*")
progress(30, 50, "*")
progress(20, 50, "*")
progress(10, 50, "*")
progress(2, 50, "*")

Regards

jross

Carl Banks

 

[Joel Ross]

That's not how streams work, chief. Once you output (and flush)
something you can't un-output it.

What you probably want to do is to write a carriage return ("\r")
which usually causes the cursor to return to the beginning of the
line, so that any new text you write overwrites the old text.

This has nothing to do with flushing; flushing doesn't erase or clear
the old input. Flushing is usually needed for a different reason,
however, namely standard output doesn't actually get sent to the
console until it sees a newline ("\n") unless you flush the buffer.

Try to adapt this example to your problem:

for i in xrange(11):
sys.stdout.write('*'*(10-i) + ' '*i + '\r')
sys.stdout.flush()
time.sleep(2)

Your example prints a new line each time, Doesn't help me with a
progress bar, do you know of anyway doing it with the print command? My
progress bars works fine this is the only problem im having with it at
the moment. any help would be appreciated.

cheers

 

[Carl Banks]

Your example prints a new line each time,

Did you run it and observe what happened, or did you just guess what
it actually did? On my system it doesn't print new lines.

Doesn't help me with a
progress bar, do you know of anyway doing it with the print command? My
progress bars works fine this is the only problem im having with it at
the moment. any help would be appreciated.

Well, based on the code you posted it doesn't look like your progress
bar is fine.

Anyway, you haven't described the problem you are having very well,
and we can guess what might be wrong but if you want better help
you're going to have to describe your problem more coherently. Give
us the

1. What code are you actually running (cut-and-paste, please, don't
retype).
2. What do you expect the code to output (type in actual output you
expect to see, not merely a description).
3. What does the code actually output when you run it (cut-and paste,
please, don't just describe).
4. Include any tracebacks if there are any.

In the meantime, I repeat my suggestion that you take my example and
adapt it to what you are doing.

One other note: the print statement is unsuitable for this task
because it always prints a trailing whitespace. Use sys.stdout.write
for it.


Carl Banks

 

[Joel Ross]

Did you run it and observe what happened, or did you just guess what
it actually did? On my system it doesn't print new lines.



Well, based on the code you posted it doesn't look like your progress
bar is fine.

Anyway, you haven't described the problem you are having very well,
and we can guess what might be wrong but if you want better help
you're going to have to describe your problem more coherently. Give
us the

1. What code are you actually running (cut-and-paste, please, don't
retype).
2. What do you expect the code to output (type in actual output you
expect to see, not merely a description).
3. What does the code actually output when you run it (cut-and paste,
please, don't just describe).
4. Include any tracebacks if there are any.

In the meantime, I repeat my suggestion that you take my example and
adapt it to what you are doing.

One other note: the print statement is unsuitable for this task
because it always prints a trailing whitespace. Use sys.stdout.write
for it.


Carl Banks

Hey,

The only reason I didn't supply full details is because I thought it may
just be something simple I was missing. Sorry for the confusing. Well
here's the full code.

import sys, os
from time import sleep


class progress:

def progressbar(self, number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
self.f=sys.stdout
if percentage > 0:
char = char * percentage
self.f.write(char)
self.f.flush()
sleep(0.2)


def countlines(self, file):

lineCount = 0
f = open(file)
it = iter(f)
try:
while it.next():
lineCount += 1
except StopIteration:
pass
return lineCount


def main():

p = progress()
lines = 5
#lines = p.countlines("/tmp/junk")
count = lines
for i in xrange(lines):
p.progressbar(count, lines, "*")
count -=1
print "Finished"

if __name__ == "__main__":
main()

For now we can ignore the countlines() function. I need a nice progress
display e.g. ********************** continuing on til it reaches 100% I
need this to print out on one line only. At the moment it does do this,
only problem is its appends the print out from the last print out. I
need it to clear the last print out and then print the new print out
without appending the last.

So when it prints a progress at 50% it will print 50 (*) characters and
if the next progress is 51% it will print 51 (*) characters including
the last 50 (*) characters, so instead on ending up with 100 (*)
characters I end up with a shit load of them depending on how many lines
I pass to the progressbar() function.

Needed Output:
***************************************************************************************************Finished


Current Output:
********************************************************************************************************************************************************************************************************Finished

Note: As you can see I'm only sending 5 lines to the progressbar()
function and It prints this many characters, so you could imagine if I
have 10000 lines I would end up with a heap of * characters.

Regards
jross

 

[Carl Banks]

     def progressbar(self, number, total,  char):

         percentage = float(number*100)/total
         percentage = int(round(percentage))
         percentage = int(100 - percentage)
         self.f=sys.stdout
         if percentage > 0:
             char = char * percentage
             self.f.write(char)
             self.f.flush()            
             sleep(0.2) [snip]
So when it prints a progress at 50% it will print 50 (*) characters and
if the next progress is 51% it will print 51 (*) characters including
the last 50 (*) characters, so instead on ending up with 100 (*)
characters I end up with a shit load of them depending on how many lines
I pass to the progressbar() function.

Here's why: when you write "char = char * percentage", you are setting
char to a string, so on the next iteration char is the string you
printed last time. Thus char grows factorially with iterations. Not
good.

Instead, change char to some other name so you don't ever overwrite
it; char should always be just one character.

bar = char * percentage
self.f.write(bar)


Carl Banks

 

[Joel Ross]

def progressbar(self, number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
self.f=sys.stdout
if percentage > 0:
char = char * percentage
self.f.write(char)
self.f.flush()
sleep(0.2) [snip]
So when it prints a progress at 50% it will print 50 (*) characters and
if the next progress is 51% it will print 51 (*) characters including
the last 50 (*) characters, so instead on ending up with 100 (*)
characters I end up with a shit load of them depending on how many lines
I pass to the progressbar() function.

Here's why: when you write "char = char * percentage", you are setting
char to a string, so on the next iteration char is the string you
printed last time. Thus char grows factorially with iterations. Not
good.

Instead, change char to some other name so you don't ever overwrite
it; char should always be just one character.

bar = char * percentage
self.f.write(bar)


Carl Banks

Your a life saver. Always good to get someone's else view on it

Thanks a heap man appreciated

jross

 

[Joel Ross]

def progressbar(self, number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
self.f=sys.stdout
if percentage > 0:
char = char * percentage
self.f.write(char)
self.f.flush()
sleep(0.2) [snip]
So when it prints a progress at 50% it will print 50 (*) characters and
if the next progress is 51% it will print 51 (*) characters including
the last 50 (*) characters, so instead on ending up with 100 (*)
characters I end up with a shit load of them depending on how many lines
I pass to the progressbar() function.

Here's why: when you write "char = char * percentage", you are setting
char to a string, so on the next iteration char is the string you
printed last time. Thus char grows factorially with iterations. Not
good.

Instead, change char to some other name so you don't ever overwrite
it; char should always be just one character.

bar = char * percentage
self.f.write(bar)


Carl Banks

Still having the same problem if I pass it 1000 lines it will printout
1000 asterisks when I say lines I mean the argument <number> for the
progress() function. I only want to printout 100 asterisks no matter how
many lines I pass to the progress() function, that's why I need the
printout to overwrite the last printout instead of appending to the last
printout. I know there must be a way to do it. I'll keep trying and
eventually get it. Any ideas would be helpful

jross

 

[Peter Otten]

class progress:

def progressbar(self, number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
self.f=sys.stdout
if percentage > 0:
char = char * percentage

Carl probably overlooked that you didn't follow his advice. If you want to
overwrite the current line you have to send a "\r" to the terminal. Change

self.f.write(char)

to
self.f.write("\r" + char)

 

[Dave Angel]

Hey,

The only reason I didn't supply full details is because I thought it
may just be something simple I was missing. Sorry for the confusing.
Well here's the full code.

import sys, os
from time import sleep


class progress:

def progressbar(self, number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
self.f=sys.stdout
if percentage > 0:
char = char * percentage
self.f.write(char)
self.f.flush()
sleep(0.2)


def countlines(self, file):

lineCount = 0
f = open(file)
it = iter(f)
try:
while it.next():
lineCount += 1
except StopIteration:
pass
return lineCount


def main():

p = progress()
lines = 5
#lines = p.countlines("/tmp/junk")
count = lines
for i in xrange(lines):
p.progressbar(count, lines, "*")
count -=1
print "Finished"

if __name__ == "__main__":
main()

For now we can ignore the countlines() function. I need a nice
progress display e.g. ********************** continuing on til it
reaches 100% I need this to print out on one line only. At the moment
it does do this, only problem is its appends the print out from the
last print out. I need it to clear the last print out and then print
the new print out without appending the last.

So when it prints a progress at 50% it will print 50 (*) characters
and if the next progress is 51% it will print 51 (*) characters
including the last 50 (*) characters, so instead on ending up with 100
(*) characters I end up with a shit load of them depending on how many
lines I pass to the progressbar() function.

Needed Output:
***************************************************************************************************Finished



Current Output:
********************************************************************************************************************************************************************************************************Finished


Note: As you can see I'm only sending 5 lines to the progressbar()
function and It prints this many characters, so you could imagine if I
have 10000 lines I would end up with a heap of * characters.

Regards
jross

You don't tell your environment. Like, what version of Python, and more
importantly what operating system? You also don't say how you're
testing the system, nor where it's intended to finally run.

There is no portable way to say "erase those charcters I sent earlier, I
didn't mean it." flush() does nothing even close to that.

In a standard console (at least on Windows), adding a simple \r (*NOT*
\n) would work easily, as Carl Banks already suggested. However, in
some shells (I test with Komodo), it does not.

So, you have a couple of choices, depending on your environment. The
simplest is to do a cursor-position operation, specific to your
particular output "device." Nobody can reliably help you with that
without knowing what the environment is.

More complex for a forward-moving progress bar is to simply keep track
of how many stars have already been output, and only add the additional
ones you want. in your current incarnation, the percentage value shows
20, then 40, then 60, then 80 (it never reaches 100, but maybe that's
okay). Currently you're printing all those stars over again. This
solution would be to print only the difference, or 20 more stars each time.

A way to accomplish that would be to create a counter in the progress
class, that knows how many stars have already been output. Something like:

class progress:

def __init__(self):
self.already = 0 #number of stars already displayed

def progressbar(self, number, total, char):

percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
self.f=sys.stdout
if percentage > 0:
char = char * (percentage-self.already)
self.already = percentage
self.f.write(char + "X") #note: remove the "X" when
you're sure it's working
self.f.flush()
sleep(0.2)

Clearly the self.write() function should change to simply
self.write(char) when you're confident it's working. For now, it gives
you a marker to see how it's being pieced together.

Incidentally, Carl's other suggestion, that you use a new variable
instead of overwriting 'char', is a red herring. It's good advice to
never change your parameters in place, and for mutable data types, could
be a problem. But because char is a string, this is not your problem.

 

[Steven D'Aprano]

Still having the same problem if I pass it 1000 lines it will printout
1000 asterisks when I say lines I mean the argument <number> for the
progress() function. I only want to printout 100 asterisks no matter how
many lines I pass to the progress() function, that's why I need the
printout to overwrite the last printout instead of appending to the last
printout.

No, you're confusing two different problems:

(1) Scale the width of the progress bar to 100 characters;

(2) Make the progress bar appear on one line instead of multiple lines.


To solve the *second* problem, use the hint already given by Carl Banks.

To solve the *first* problem, use mathematics, then re-do your method,
because your method is wrong. Before you can write code to solve this
problem, you first have to get the mathematics right: you want to scale
the width of the progress bar to be 100 asterisks no matter how many
lines you have.

(a) If you have exactly 100 lines, and you draw one asterisk per line,
then you get a progress bar of exactly 100 asterisks.

(b) If you have 50 lines, and you want a progress bar of exactly 100
asterisks, you need to draw *two* asterisks per line.

(c) If you have 200 lines, and you want a progress bar of exactly 100
asterisks, you need to draw *one half* an asterisk per line (or one
asterisk per *two* lines).

(d) If you have N lines, and you want a progress bar of exactly 100
asterisks, each line must be equivalent to [____] asterisks.

Once you have filled in the blank, then go back to writing code.


Some further hints: I've added some commentary to your code:

def progressbar(self, number, total, char):
percentage = float(number*100)/total
percentage = int(round(percentage))
percentage = int(100 - percentage)
# Why not combine all of the above into one line?
# percentage = int(100 - round(number*100.0/total))

self.f=sys.stdout
# Why are you storing stdout as an attribute to the method? Unless
# you will need it again elsewhere, this is a waste.

if percentage > 0:
char = char * percentage
# Carl Banks has already pointed out the problem here.

self.f.write(char)
self.f.flush()
# Better written like this:
# sys.stdout.write(char)
# sys.stdout.flush()

sleep(0.2)


Finally, here's a function that should do what you ask for:

from __future__ import division
import sys

def progressbar(number, total, char='*', maxwidth=100):
done = number/total
if not 0 <= done <= 1:
raise ValueError('number or total out of range')
# Scale done=1 => maxwidth chars.
s = int(done*maxwidth)*char
sys.stdout.write(s + '\r')
sys.stdout.flush()
sleep(0.2)

This should work in "normal" terminal windows (tested on Linux, should
work on Windows and Mac too) but some IDEs may mess it up.

 

[Joel Ross]

Thanks for all the help guys. I got it to work correctly with this

class progress:

def __init__(self):
self.already = 0

def progressbar(self, number, total, char):

percentage = int(100 - round(number*100.0/total))
if percentage > 0:
xchar = char * (percentage-self.already)
self.already = percentage
sys.stdout.write(xchar)
sys.stdout.flush()
sleep(1)

Keeping track of the characters that had already been printed solved my
main problem thanks Dave and everyone else for helping me out with this
issue. Now I can move onto next one.

jross

 

[Dennis Lee Bieber]

So when it prints a progress at 50% it will print 50 (*) characters and
if the next progress is 51% it will print 51 (*) characters including
the last 50 (*) characters, so instead on ending up with 100 (*)
characters I end up with a shit load of them depending on how many lines
I pass to the progressbar() function.

So... Don't send the WHOLE "percentage", send the DIFFERENCE between
the current % and the previously sent value (take into account if you
find a 0 difference that you don't update the "previously sent" value.

So if you've displayed 50 *s, and now need 51 *s...

prvs = 50
....
now = 51

delta = now - prvs
if delta > 0:
sys.stdout.write("*" * delta)
sys.stdout.flush()
prvs = now


flush() is working perfectly fine -- it says transmit any data still
held within internal buffers. It is NOT a "clear screen", "clear line"
terminal command.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Carl Banks]

Incidentally, Carl's other suggestion, that you use a new variable
instead of overwriting 'char', is a red herring.

Yeah, I dropped the ball on this one, I misread his code as being in a
loop instead of being called repeatedly. And after criticizing him
for not running mine....

Sorry bout that.


Carl Banks

 

[Rhodri James]

Now I can move onto next one.

Except that you still have the interesting issue that your environment
isn't responding to '\r' correctly, which worries me rather. Or did
you never test that?

 

[Joel Ross]

Except that you still have the interesting issue that your environment
isn't responding to '\r' correctly, which worries me rather. Or did
you never test that?

Yeah I gave the "\r" a go and it kept printing out on a new line I will
look into it.

Thanks for the heads up

 

[Joel Ross]

flush() is working perfectly fine -- it says transmit any data still
held within internal buffers. It is NOT a "clear screen", "clear line"
terminal command.

I was mistaken about the sys.stout.flush(). I understand it a little
more now thanks

 

[Mel]

Rhodri James wrote: [ ... ]

Except that you still have the interesting issue that your environment
isn't responding to '\r' correctly, which worries me rather. Or did
you never test that?

Yeah I gave the "\r" a go and it kept printing out on a new line I will
look into it.

Are you running an Apple computer?

Mel.

 

[Joel Ross]

Rhodri James wrote: [ ... ]

Except that you still have the interesting issue that your environment
isn't responding to '\r' correctly, which worries me rather. Or did
you never test that?

Yeah I gave the "\r" a go and it kept printing out on a new line I will
look into it.

Are you running an Apple computer?

Mel.

No running a Linux fedora 9 box

 

[AK]

Yeah I gave the "\r" a go and it kept printing out on a new line I will
look into it.

Thanks for the heads up

What happens when you run this little program:

import time, sys

print "ONE",
sys.stdout.flush()
time.sleep(0.5)
print "\rTWO",
sys.stdout.flush()
time.sleep(0.5)


The idea is that it should print ONE, flush it, sleep .5 sec so
that you can see 'ONE', then return to start of line, print TWO,
flush it so you can read it, and sleep another 0.5 sec.

NOTE comma after print statement.

Does that work for you?

-ak