CSV readers and UTF-8 files

M

mk

Hello everyone,

Is it just me or CSV reader/DictReader and UTF-8 files do not work
correctly in Python 2.6.1 (Windows)?

That is, when I open UTF-8 file in a csv reader (after passing plain
file object), I get fields as plain strings ('str'). Since this has been
mangled, I can't get the non-ascii characters back.

When I do:

csvfo = codecs.open(csvfname, 'rb', 'utf-8')
dl = csv.excel
dl.delimiter=';'
#rd = csv.DictReader(csvfo, dialect=dl)
rd = csv.reader(csvfo, dialect=dl)

...I get plain strings as well (I get <type 'str'> when calling
type(field)), on top of error:

Traceback (most recent call last):
File "C:/Python26/converter3.py", line 99, in <module>
fill_sqla(session,columnlist,rd)
File "C:/Python26/converter3.py", line 73, in fill_sqla
for row in rd:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0144' in
position 74: ordinal not in range(128)

...when doing:

for row in rd:
....

Regards,
mk
 
F

Falcolas

Hello everyone,

Is it just me or CSV reader/DictReader and UTF-8 files do not work
correctly in Python 2.6.1 (Windows)?

I would point out in the CSV module documentation (http://
docs.python.org/library/csv.html) it explicitly mentions that it can't
handle unicode.

You can use their workaround in the examples section for UTF-8, or
with another form of encoding (I used MIME) for UTF-16.

~G
 
C

Chris Rebert

I would point out in the CSV module documentation (http://
docs.python.org/library/csv.html) it explicitly mentions that it can't
handle unicode.

You can use their workaround in the examples section for UTF-8, or
with another form of encoding (I used MIME) for UTF-16.

~G

This really ought to be fixed for 3.0+ (seems to still be ASCII-only
according to the 3.0 docs...)

Cheers,
Chris
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top