UnicodeDecodeError: 'ascii' codec can't decode byte

Discussion in 'Python' started by Gilles Ganault, Jun 17, 2008.

  1. Hello

    It seems like I have Unicode data in a CSV file but Python is using
    a different code page, so isn't happy when I'm trying to read and put
    this data into an SQLite database with APSW:

    ========
    sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"
    cursor.executemany(sql, records("test.tsv"))
    """
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position
    18: ordinal not in range(128)
    """
    ========

    What should I do so Python doesn't raise this error? Should I convert
    data in the CVS file, or is there some function that I should call
    before APSW's executemany()?

    Thank you.
     
    Gilles Ganault, Jun 17, 2008
    #1
    1. Advertising

  2. Gilles Ganault

    Peter Otten Guest

    Gilles Ganault wrote:

    > It seems like I have Unicode data in a CSV file but Python is using
    > a different code page, so isn't happy when I'm trying to read and put
    > this data into an SQLite database with APSW:


    My guess is that you have non-ascii characters in a bytestring.

    > What should I do so Python doesn't raise this error? Should I convert
    > data in the CVS file, or is there some function that I should call
    > before APSW's executemany()?


    You cannot have unicode data in a file, only unicode converted to
    bytestrings using some encoding. Assuming that encoding is UTF-8 and that
    apsw can cope with unicode, try to convert your data to unicode before
    feeding it to the database api:

    > sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"


    rows = ([col.decode("utf-8") for col in row] for row in
    records("test.tsv"))
    cursor.executemany(sql, rows)

    Peter
     
    Peter Otten, Jun 17, 2008
    #2
    1. Advertising

  3. On Tue, 17 Jun 2008 09:23:28 +0200, Peter Otten <>
    wrote:
    > Assuming that encoding is UTF-8 and that apsw can cope
    > with unicode, try to convert your data to unicode before
    > feeding it to the database api:
    >
    >> sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"

    >
    > rows = ([col.decode("utf-8") for col in row] for row in
    >records("test.tsv"))
    > cursor.executemany(sql, rows)


    Thanks again.
     
    Gilles Ganault, Jun 17, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robin Siebler
    Replies:
    4
    Views:
    26,701
    Tim Peters
    Oct 8, 2004
  2. Oleg  Parashchenko
    Replies:
    4
    Views:
    1,044
    Paul Boddie
    Mar 31, 2007
  3. Gabriel Genellina
    Replies:
    0
    Views:
    758
    Gabriel Genellina
    Oct 21, 2008
  4. Gilles Ganault
    Replies:
    3
    Views:
    1,953
    Steve Holden
    Oct 29, 2008
  5. Tim Golden
    Replies:
    0
    Views:
    108
    Tim Golden
    Nov 27, 2013
Loading...

Share This Page