Py3: Read file with Unicode characters

[Gnarlodious]

Attempting to read a file containing Unicode characters such as ±:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
5007: ordinal not in range(128)

I did succeed by converting all the characters to HTML entities such
as "±", but I want the characters to be the actual font in the
source file. What am I doing wrong? My understanding is that ALL
strings in Py3 are unicode so... confused.

-- Gnarlie

 

[Martin v. Loewis]

Attempting to read a file containing Unicode characters such as ±:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
5007: ordinal not in range(128)

I did succeed by converting all the characters to HTML entities such
as "±", but I want the characters to be the actual font in the
source file. What am I doing wrong? My understanding is that ALL
strings in Py3 are unicode so... confused.

When opening the file, you need to specify the file encoding. If you
don't, it defaults to ASCII (in your situation; the specific default
depends on the environment).

Regards,
Martin

 

[Gnarlodious]

When opening the file, you need to specify the file encoding.

OK, I had tried this:

open(path, 'r').read().encode('utf-8')

however I get error

TypeError: Can't convert 'bytes' object to str implicitly

I had assumed a Unicode string was a Unicode string, so why is it a
bytes string?

Sorry, doing Unicode in Py3 has really been a challenge.

-- Gnarlie

 

[Martin v. Loewis]

OK, I had tried this:

open(path, 'r').read().encode('utf-8')

No, when *opening* the file, you need to specify the encoding:

open(path, 'r', encoding='utf-8').read()

Sorry, doing Unicode in Py3 has really been a challenge.

That's because you need to re-learn some things.

Regards,
Martin

 

[Gnarlodious]

That's because you need to re-learn some things.

Apparently so, every little item is a lesson. Thank you.

-- Gnarlie