Handling text lines from files with some (few) starnge chars

[Paulo da Silva]

I need to read text files and process each line using string
comparisions and regexp.

I have a python2 program that uses <file object>.readline to read each
line as a string. Then, processing it was a trivial job.

With python3 I got error messagew like:
File "./pp1.py", line 93, in RL
line=inf.readline()
File "/usr/lib64/python3.1/codecs.py", line 300, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position
4963-4965: invalid data

How do I handle this?

If I use <file object>.read from an open as binary file I got a <bytes>
object. Then how do I handle it? Reg exps, comparisions with strings, ?...

Thanks for any help.

 

[Chris Rebert]

I need to read text files and process each line using string
comparisions and regexp.

I have a python2 program that uses <file object>.readline to read each
line as a string. Then, processing it was a trivial job.

With python3 I got error messagew like:
File "./pp1.py", line 93, in RL
   line=inf.readline()
 File "/usr/lib64/python3.1/codecs.py", line 300, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position
4963-4965: invalid data

How do I handle this?

Specify the encoding of the text when opening the file using the
`encoding` parameter. For Windows-1252 for example:

your_file = open("path/to/file.ext", 'r', encoding='cp1252')

Cheers,
Chris

 

[python]

Chris,

Specify the encoding of the text when opening the file using the `encoding` parameter. For Windows-1252 for example:

your_file = open("path/to/file.ext", 'r', encoding='cp1252')

This looks similar to the codecs module's functionality. Do you know if
the codecs module is still required in Python 3.x?

Thank you,
Malcolm

 

[Paulo da Silva]

Em 06-06-2010 00:41, Chris Rebert escreveu:

On Sat, Jun 5, 2010 at 4:03 PM, Paulo da Silva


Specify the encoding of the text when opening the file using the
`encoding` parameter. For Windows-1252 for example:

your_file = open("path/to/file.ext", 'r', encoding='cp1252')

OK! This fixes my current problem. I used encoding="iso-8859-15". This
is how my text files are encoded.
But what about a more general case where the encoding of the text file
is unknown? Is there anything like "autodetect"?

 

[MRAB]

Em 06-06-2010 00:41, Chris Rebert escreveu:

OK! This fixes my current problem. I used encoding="iso-8859-15". This
is how my text files are encoded.
But what about a more general case where the encoding of the text file
is unknown? Is there anything like "autodetect"?
>

An encoding like 'cp1252' uses 1 byte/character, but so does 'cp1250'.
How could you tell which was the correct encoding?

Well, if the file contained words in a certain language and some of the
characters were wrong, then you'd know that the encoding was wrong. This
does imply, though, that you'd need to know what the language should
look like!

You could try different encodings, and for each one try to identify what
could be words, then look them up in dictionaries for various languages
to see whether they are real words...

 

[John Machin]

 >
An encoding like 'cp1252' uses 1 byte/character, but so does 'cp1250'.
How could you tell which was the correct encoding?

Well, if the file contained words in a certain language and some of the
characters were wrong, then you'd know that the encoding was wrong. This
does imply, though, that you'd need to know what the language should
look like!

You could try different encodings, and for each one try to identify what
could be words, then look them up in dictionaries for various languages
to see whether they are real words...

This has been automated (semi-successfully, with caveats) by the
chardet package ... see http://chardet.feedparser.org/

 

[Paulo da Silva]

Em 06-06-2010 04:05, John Machin escreveu:

....

....


This has been automated (semi-successfully, with caveats) by the
chardet package ... see http://chardet.feedparser.org/

This seems nice!
Thanks