UTF-8

[Olivier Verdier]

First off: i thoroughly enjoy python. I use it for scientific
computing with scipy, numpy and matplotlib and it's an amazingly
efficient and elegant language.

About this mailing list: it is very hard to search. I can't find any
search field on the page: http://mail.python.org/mailman/listinfo/
python-list. I would greatly appreciate if you moved that list over
to google, for instance, so that it be both searchable and rss-feedable.

My question is the following: how to set a default encoding in
python? I read an old thread about that and it didn't seem possible
by then.

The default encoding i wish to set is UTF-8 since it encodes unicode
and is nowadays the standard encoding. Having ascii as a default
encoding is a pretty strange choice since ascii encodes only 127
characters (suitable only for English, without fancy symbols like €)
whereas utf-8 encodes virtually infinitely many characters (suitable
for all languages). Besides, utf-8 is a superset of ascii so having
it by default won't harm anything.

At the very least there should be a configuration file that allows to
set a default encoding (although setting anything else than utf-8
should be advised against). Such a feature would be greatly
appreciated by all of us non native English speakers around the world.

Best regards,

== Olivier

 

[Laurent Pointal]

Olivier Verdier wrote:

My question is the following: how to set a default encoding in
python? I read an old thread about that and it didn't seem possible
by then.

You *can* put a sys.setdefaultencoding("utf-8") in your sitecustomize.py
(see Python libs/site-packages/). Note that this function is no longer
available after Python startup.

But, IMHO, this is a *bad idea*, it may break modules which consider default
encoding to be ascii.
You should prefer to put
# -*- coding: utf-8 -*-
at the begining of your sources files. With that you are ok with all Python
installations, whatever be the defautl encoding.
Hope this will become mandatory in a future Python version.

A+

Laurent.

 

[Leif K-Brooks]

You should prefer to put
# -*- coding: utf-8 -*-
at the begining of your sources files. With that you are ok with all Python
installations, whatever be the defautl encoding.
Hope this will become mandatory in a future Python version.

The default encoding specifies how Unicode strings are implicitly
converted into byte strings. The code you gave specifies how Unicode
string literals in a file are encoded, which is something completely
different.

 

[John Machin]

First off: i thoroughly enjoy python. I use it for scientific
computing with scipy, numpy and matplotlib and it's an amazingly
efficient and elegant language.

About this mailing list: it is very hard to search. I can't find any
search field on the page:http://mail.python.org/mailman/listinfo/
python-list. I would greatly appreciate if you moved that list over
to google, for instance, so that it be both searchable and rss-feedable.

The mailing list and the Usenet newsgroup comp.lang.python mirror each
other. Google Groups provides access to Usenet newsgroups; try
http://groups.google.com/group/comp.lang.python

Other possibilities are gmane.org and Yahoo Groups; see
http://www.python.org/community/lists/

HTH,
John

 

[Eric Brunel]

[snip]

The default encoding i wish to set is UTF-8 since it encodes unicode and
is nowadays the standard encoding.

I can't agree with that: there are still many tools completely ignoring
the encoding problem, and just saving sequences of bytes for any text.
This results in letting the current OS deciding what encoding is used, and
this is still not always UTF-8. Just consider how many web pages are still
encoded in a local encoding without even any specification of what it may
be in the HTML header, or how many times you see a [?] or a black square
replacing a character in newsgroups or mailing lists. UTF-8 *should* be
the standard encoding, but certainly isn't...

Having ascii as a default encoding is a pretty strange choice since
ascii encodes only 127 characters (suitable only for English, without
fancy symbols like €) whereas utf-8 encodes virtually infinitely many
characters (suitable for all languages). Besides, utf-8 is a superset of
ascii so having it by default won't harm anything.

On the contrary, it's IMHO a very sensible thing to do: any character with
a code > 127 could be in any encoding. So supposing an encoding would be
guessing, and part of the Python Zen is: "in the face of ambiguity, refuse
the temptation to guess". The problem is here that if the encoding is not
UTF-8, things can seem to work, but give completely wrong results. So the
decision was apparently to use the minimal common set for everything,
which is ASCII; this seems quite logical to me.

(BTW, one could even argue that the default encoding should be
non-existent, as I think there are some non-latin encodings that do not
even include ASCII [I think I saw a Chinese encoding like that one
day...]. So even supposing that ASCII will work is sometimes wrong...)

At the very least there should be a configuration file that allows to
set a default encoding (although setting anything else than utf-8 should
be advised against). Such a feature would be greatly appreciated by all
of us non native English speakers around the world.

Well, I *am* a non-native English speaker, and after the first "why should
I bother about all this encoding stuff?" period, which seems to happen to
everybody, I finally understood the logic behind Python choices, and why I
*should* bother about encodings. Whatever the default encoding is, and
whatever you do, encodings will still be a problem for some time. The
transition phase from local encodings to universal ones is likely to last
a few years more, if not a few decades...

HTH