codecs.EncodedFile

[Neil Cerutti]

Perhaps I'm just bad at searching for bugs, but anyhow, I wanted
to know what you all thought about the following behavior.

A quick search of pydev archives yielded a nice wrapper to apply
to streams to perform decoding and encoding behind the scenes.
Assuming I get the correct encodings from somewhere (that's a
whole 'nother thread):

Here's the docs:

EncodedFile( file, input[, output[, errors]])

Return a wrapped version of file which provides transparent
encoding translation.

Strings written to the wrapped file are interpreted according
to the given input encoding and then written to the original
file as strings using the output encoding. The intermediate
encoding will usually be Unicode but depends on the specified
codecs.

If output is not given, it defaults to input.

errors may be given to define the error handling. It defaults
to 'strict', which causes ValueError to be raised in case an
encoding error occurs.

Base on that, I wrote the following code at startup:

sys.stdout = codecs.EncodedFile(sys.stdout, 'latin-1', 'cp437')
sys.stdin = codecs.EncodedFile(sys.stdin, 'cp437', 'latin-1')

Now my application never returns from its first call to
sys.stdin.readline.

It turns out to be troublesome for my case because the
EncodedFile object translates calls to readline into calls to
read.

I believe it ought to raise a NotImplemented exception when
readline is called.

As it is it silently causes interactive applications to
apparently hang forever, and breaks the line-buffering
expectation of non-interactive applications.

If raising the exception is too much to ask, then at least it
should be documented better.

 

[Leo Kislov]

It turns out to be troublesome for my case because the
EncodedFile object translates calls to readline into calls to
read.

I believe it ought to raise a NotImplemented exception when
readline is called.

As it is it silently causes interactive applications to
apparently hang forever, and breaks the line-buffering
expectation of non-interactive applications.

Does it work if stdin is a pipe? If it works then raising
NotImplemented doesn't make sense.

If raising the exception is too much to ask, then at least it
should be documented better.

Improving documentation is always a good idea. Meanwhile see my
solution how to make readline method work:
http://groups.google.com/group/comp.lang.python/msg/f1267dc612314657

 

[Neil Cerutti]

Does it work if stdin is a pipe? If it works then raising
NotImplemented doesn't make sense.

You're right, it does work in that case. Excellent! So my second
speculation about it breaking non-interactive applications was
false. I should have peeked at the code before babbling. Sorry
about that. That it might be using an internal buffer never
occurred to me.

Improving documentation is always a good idea. Meanwhile see my
solution how to make readline method work:
http://groups.google.com/group/comp.lang.python/msg/f1267dc612314657

Thanks for that solution. That'll do nicely.

The moral I'm discovering is: Don't try to write an interactive
application in Python that uses stdin and stdout unless it's
strictly a toy program.