UnicodeDecodeError: 'ascii' codec can't decode byte

[Gilles Ganault]

Hello

It seems like I have Unicode data in a CSV file but Python is using
a different code page, so isn't happy when I'm trying to read and put
this data into an SQLite database with APSW:

========
sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"
cursor.executemany(sql, records("test.tsv"))
"""
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position
18: ordinal not in range(128)
"""
========

What should I do so Python doesn't raise this error? Should I convert
data in the CVS file, or is there some function that I should call
before APSW's executemany()?

Thank you.

 

[Peter Otten]

It seems like I have Unicode data in a CSV file but Python is using
a different code page, so isn't happy when I'm trying to read and put
this data into an SQLite database with APSW:

My guess is that you have non-ascii characters in a bytestring.

What should I do so Python doesn't raise this error? Should I convert
data in the CVS file, or is there some function that I should call
before APSW's executemany()?

You cannot have unicode data in a file, only unicode converted to
bytestrings using some encoding. Assuming that encoding is UTF-8 and that
apsw can cope with unicode, try to convert your data to unicode before
feeding it to the database api:

sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"

rows = ([col.decode("utf-8") for col in row] for row in
records("test.tsv"))
cursor.executemany(sql, rows)

Peter

 

[Gilles Ganault]

Assuming that encoding is UTF-8 and that apsw can cope
with unicode, try to convert your data to unicode before
feeding it to the database api:

sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"

rows = ([col.decode("utf-8") for col in row] for row in
records("test.tsv"))
cursor.executemany(sql, rows)

Thanks again.