Can someone explain this unexpected raw_input behavior?

[Mike Kent]

It's often useful for debugging to print something to stderr, and to
route the error output to a file using '2>filename' on the command
line.

However, when I try that with a python script, all prompt output from
raw_input goes to stderr. Consider the following test program:

=== Start test.py ===
import sys

def main():
print "Hello world"
raw_input("Press ENTER")
print "Goodbye world"

if __name__ == "__main__":
main()
=== End test.py ===

If I run it with the command line 'python2.5 test.py', I get the
following output:

Hello world
Press ENTER <=== This appeared,the program paused, I press ENTER,
and the program continued
Goodbye world

However, if I run it with the command line 'python2.5 test.py 2>/dev/
null' (I'm routing stderr output to /dev/null), I instead get:

Hello world
<=== No output appeared, the program paused, I
press ENTER, and the program continued
Goodbye world

This indicates to me that the prompt output of raw_input is being sent
to stderr. I did check the source code for raw_input, and it appears
to be sending it to stdout as expected.

I get this behavior on multiple OS platforms, with multiple versions
of Python. I am building python on these platforms myself, but to my
knowledge, I am not doing anything special which could account for
this behavior.

Any suggestions or pointers on how to get the expected behavior out of
raw_input?

 

[Gabriel Genellina]

It's often useful for debugging to print something to stderr, and to
route the error output to a file using '2>filename' on the command
line.

However, when I try that with a python script, all prompt output from
raw_input goes to stderr. Consider the following test program:
[...]
This indicates to me that the prompt output of raw_input is being sent
to stderr. I did check the source code for raw_input, and it appears
to be sending it to stdout as expected.

Surely you've seen that in bltinmodule.c, builtin_raw_input calls
PyOS_Readline(PyFile_AsFile(fin), PyFile_AsFile(fout), prompt);
where fin and fout are sys.stdin and sys.stdout respectively.
That function is defined in Parser/myreadline.c, and eventually calls
PyOS_StdioReadline with the same arguments. But PyOS_StdioReadline doesn't
use its sys_stdout parameter to output the prompt; instead, it always uses
stderr, no matter what arguments it receives.
Looking at the svn history, PyOS_StdioReadline always has used stderr; got
its current signature in rev 29400 (2002).

Perhaps that behavior is based on the reverse of your use case, which may
be more common. A script writes most of its output to stdout, and you
capture that using >filename. If raw_input prompted the user using stdout,
that would not be visible in this case, so using stderr is more useful.

 

[Mike Kent]

Gabriel, thank you for clarifying the source of this behavior. Still,
I'm surprised it would be hard-coded into Python. Consider an
interactive program, that asks the user several questions, and
displays paragraphs of information based on those questions. The
paragraphs are output using print, and the questions are asked via
raw_input. You want to do some simple debugging of the program by
printing some debugging statements via 'print >>sys.stderr', and you
don't want the debug output mixed in with the normal output on the
screen, so you try to route the debugging output to a file by adding
'2>filename' to the end of the command line.

Unfortunately, you will no longer see any of the questions being
printed via raw_input. The rest of the output will be fine, but the
questions disappear. Your program just stops, without asking
anything... you have to guess what should be there.

I'm surprised that Python hard-codes this behavior without giving the
programmer any way to change it. It leaves me with two options: to
either always use the logging module for debugging messages (which is
not odious at all, it's just that the code in question predates the
logging module, which is why debugging was done as it is), or change
the program so that raw_input is never used with a prompt parameter;
the prompt must always be printed separately.

<shrug> At least I now know I'm not crazy... regarding this, anyway.

 

[Ben Finney]

Consider an interactive program, that asks the user several
questions, and displays paragraphs of information based on those
questions. The paragraphs are output using print, and the questions
are asked via raw_input.

Okay so far.

You want to do some simple debugging of the program by printing some
debugging statements via 'print >>sys.stderr', and you don't want
the debug output mixed in with the normal output on the screen, so
you try to route the debugging output to a file by adding
'2>filename' to the end of the command line.

This issue isn't specific to Python. "Program stops to ask questions
from the user" is not compatible with "Can safely redirect output of
the program to a file".

Either one, or both, of those requirements will have to be
compromised, or dropped completely.

 

[Gabriel Genellina]

Gabriel, thank you for clarifying the source of this behavior. Still,
I'm surprised it would be hard-coded into Python. Consider an
interactive program, that asks the user several questions, and
displays paragraphs of information based on those questions. The
paragraphs are output using print, and the questions are asked via
raw_input. You want to do some simple debugging of the program by
printing some debugging statements via 'print >>sys.stderr', and you
don't want the debug output mixed in with the normal output on the
screen, so you try to route the debugging output to a file by adding
'2>filename' to the end of the command line.

Unfortunately, you will no longer see any of the questions being
printed via raw_input. The rest of the output will be fine, but the
questions disappear. Your program just stops, without asking
anything... you have to guess what should be there.

You have one console, two streams to output data (stdout and stderr), and
three data sources (program output, user prompt, and debugging messages).
Someone has to give.
I'm now convinced that the current behavior is rather reasonable...

I'm surprised that Python hard-codes this behavior without giving the
programmer any way to change it. It leaves me with two options: to
either always use the logging module for debugging messages (which is
not odious at all, it's just that the code in question predates the
logging module, which is why debugging was done as it is), or change
the program so that raw_input is never used with a prompt parameter;
the prompt must always be printed separately.

Perhaps raw_input could have a use_stderr=True parameter; with False would
display the prompt on stdout.

 

[BJörn Lindqvist]

You have one console, two streams to output data (stdout and stderr), and
three data sources (program output, user prompt, and debugging messages).
Someone has to give.
I'm now convinced that the current behavior is rather reasonable...

If it weren't for the documentation...

"If the prompt argument is present, it is written to *standard output*
without a trailing newline."

 

[Mike Kent]

If it weren't for the documentation...

"If the prompt argument is present, it is written to *standard output*
without a trailing newline."

I have reported this issue to the python-dev mailing list, and Guido
agrees that this is a bug in Python. It turns out that the key is
that my site does not have GNU readline installed, so Python falls
back to its own implementation of readline. Using GNU readline,
raw_input will write its prompt to stdout. Python's own
implementation of readline sends the output to stderr. As Bjorn
states, the documentation for raw_input says it writes its prompt to
stdout. A bug issue has been opened in the Python Trac system for
this.

 

[Ben Finney]

A bug issue has been opened in the Python Trac system for this.

Wouldn't it be better to report it in the official Python bug tracker
<URL:http://bugs.python.org/>, which is Roundup, not Trac?

 

[Terry Reedy]

|
| > A bug issue has been opened in the Python Trac system for this.
|
| Wouldn't it be better to report it in the official Python bug tracker
| <URL:http://bugs.python.org/>, which is Roundup, not Trac?

Has been by Skip: http://bugs.python.org/issue1927

 

[Mike Kent]

|
| > A bug issue has been opened in the Python Trac system for this.
|
| Wouldn't it be better to report it in the official Python bug tracker
| <URL:http://bugs.python.org/>, which is Roundup, not Trac?

Has been by Skip: http://bugs.python.org/issue1927

My mistake in using the word 'Trac'.