Hebrew in idle ans eclipse (Windows)

[iu2]

Hi all,

I'll realy appreciate your help in this:

I read data from a database containg Hebrew words.
When the application is run from IDLE a word looks like this, for
example:
\xe8\xe9\xe5

But when I run the same application from eclipse or the Windows shell
I get the 'e's replaced with '8's:
\x88\x89\x85

The IDLE way is the way I need, since using Hebrew words in the
program text itself, as keys in a dict, for example, yield similar
strings (with 'e's). When running from eclipse I get KeyError for this
dict..

What do I need to do run my app like IDLE does?

Thanks
iu2

 

[Martin v. Löwis]

What do I need to do run my app like IDLE does?

Can you please show the fragment of your program that prints
these strings?

Regards,
Martin

 

[iu2]

Can you please show the fragment of your program that prints
these strings?

Regards,
Martin

Hi,
I use pymssql to get the data from a database, just like this (this is
from the pymssql doc):

import pymssql

con =
pymssql.connect(host='192.168.13.122',user='sa',password='',database='tempdb')
cur = con.cursor()
cur.execute('select firstname, lastname from [users]')
lines = cur.fetchall()

print lines

or

print lines[0]

'lines' is a list containing tuples of 2 values, for firstname and
lastname. The names are Hebrew and their code looks different when I'm
runnig it from IDLE than when running it from Windows shell or
eclipse, as I described in my first post.


Important: This doesn't happer when I read text from a file containing
Hebrew text. In that case both IDLE and eclipse give the same reulst
(the hebrew word itself is printed to the console)

 

[Martin v. Löwis]

import pymssql

con =
pymssql.connect(host='192.168.13.122',user='sa',password='',database='tempdb')
cur = con.cursor()
cur.execute('select firstname, lastname from [users]')
lines = cur.fetchall()

print lines

or

print lines[0]

'lines' is a list containing tuples of 2 values, for firstname and
lastname. The names are Hebrew and their code looks different when I'm
runnig it from IDLE than when running it from Windows shell or
eclipse, as I described in my first post.

Ok. Please understand that there are different ways to represent
characters as bytes; these different ways are called "encodings".

Please also understand that you have to make a choice of encoding
every time you represent characters as bytes: if you read it from a
database, and if you print it to a file or to the terminal.

Please further understand that interpreting bytes in an encoding
different from the one they were meant for results in a phenomenon
called "moji-bake" (from Japanese, "ghost characters"). You get
some text, but it makes no sense (or individual characters are incorrect).

So you need to find out
a) what the encoding is that your data have in MySQL
b) what the encoding is that is used when printing in IDLE
c) what the encoding is that is used when printing into
a terminal window.

b) and c) are different on Windows; the b) encoding is called
the "ANSI code page", and c) is called the "OEM code page".
What the specific choice is depends on your specific Windows
version and local system settings.

As for a: that's a choice somebody made when the database
was created; I don't know how to figure out what encoding
MySQL uses.

In principle, rather than doing

print lines[0]

you should do

print lines[0].decode("<a-encoding>").encode("<c-encoding>")

when printing to the console. Furtenately, you can also write
this as

print lines[0].decode("<a-encoding>")

as Python will figure out the console encoding by itself, but
it can't figure out the MySQL encoding (or atleast doesn't,
the way you use MySQL).

Regards,
Martin

 

[iu2]

import pymssql

con =
pymssql.connect(host='192.168.13.122',user='sa',password='',database='tempd­b')
cur = con.cursor()
cur.execute('select firstname, lastname from [users]')
lines = cur.fetchall()

print lines

or

print lines[0]

'lines' is a list containing tuples of 2 values, for firstname and
lastname. The names areHebrewand their code looks different when I'm
runnig it fromIDLEthan when running it from Windows shell or
eclipse, as I described in my first post.

Ok. Please understand that there are different ways to represent
characters as bytes; these different ways are called "encodings".

Please also understand that you have to make a choice of encoding
every time you represent characters as bytes: if you read it from a
database, and if you print it to a file or to the terminal.

Please further understand that interpreting bytes in an encoding
different from the one they were meant for results in a phenomenon
called "moji-bake" (from Japanese, "ghost characters"). You get
some text, but it makes no sense (or individual characters are incorrect).

So you need to find out
a) what the encoding is that your data have in MySQL
b) what the encoding is that is used when printing inIDLE
c) what the encoding is that is used when printing into
   a terminal window.

b) and c) are different on Windows; the b) encoding is called
the "ANSI code page", and c) is called the "OEM code page".
What the specific choice is depends on your specific Windows
version and local system settings.

As for a: that's a choice somebody made when the database
was created; I don't know how to figure out what encoding
MySQL uses.

In principle, rather than doing

  print lines[0]

you should do

  print lines[0].decode("<a-encoding>").encode("<c-encoding>")

when printing to the console. Furtenately, you can also write
this as

  print lines[0].decode("<a-encoding>")

as Python will figure out the console encoding by itself, but
it can't figure out the MySQL encoding (or atleast doesn't,
the way you use MySQL).

Regards,
Martin- Hide quoted text -

- Show quoted text -

Thanks for the detailed explanation. I'll try that.

 

[iu2]

...
print lines[0].decode("<a-encoding>").encode("<c-encoding>")
...
Regards,
Martin

Ok, I've got the solution, but I still have a question.

Recall:
When I read data using sql I got a sequence like this:
\x88\x89\x85
But when I entered heberw words directly in the print statement (or as
a dictionary key)
I got this:
\xe8\xe9\xe5

Now, scanning the encoding module I discovered that cp1255 maps
'\u05d9' to \xe9
while cp856 maps '\u05d9' to \x89,
so trasforming \x88\x89\x85 to \xe8\xe9\xe5 is done by

s.decode('cp856').encode('cp1255')

ending up with the pattern you suggested.

My qestion is, is there a way I can deduce cp856 and cp1255 from the
string itself? Is there a function doing it? (making the
transformation more robust)

I don't know how IDLE guessed cp856, but it must have done it.
(perhaps because it uses tcl, and maybe tcl guesses the encoding
automatically?)

thanks
iu2

 

[Martin v. Löwis]

Recall:

When I read data using sql I got a sequence like this:
\x88\x89\x85
But when I entered heberw words directly in the print statement (or as
a dictionary key)
I got this:
\xe8\xe9\xe5

Now, scanning the encoding module I discovered that cp1255 maps
'\u05d9' to \xe9
while cp856 maps '\u05d9' to \x89,
so trasforming \x88\x89\x85 to \xe8\xe9\xe5 is done by

Hebrew Windows apparently uses cp1255 (aka windows-1255) as
the "ANSI code page", used in all GUI APIs, and cp856 as the
"OEM code page", used in terminal window - and, for some reason,
in MS SQL.

My qestion is, is there a way I can deduce cp856 and cp1255 from the
string itself?

That's not possible. You have to know where the string comes from.
to know what the encoding is.

In the specific case, if the string comes out of MS SQL, it apparently
has cp856 (but I'm sure you can specify the client encoding somewhere
in SQL server, or in pymssql)

I don't know how IDLE guessed cp856, but it must have done it.

I don't know why you think it did. You said you entered \xe9 directly
into the source code in IDLE, so
a) this is windows-1255, not cp856, and
b) IDLE just *used* windows-1255 (i.e. the ANSI code page), it did
not guess it.

If you are claimaing that the program

import pymssql

con =
pymssql.connect(host='192.168.13.122',user='sa',password='',database='tempdb')
cur = con.cursor()
cur.execute('select firstname, lastname from [users]')
lines = cur.fetchall()
print repr(lines[0])

does different things depending on whether it is run in IDLE or in a
terminal window - I find that hard to believe. IDLE/Tk has nothing to
do with that. It's the *repr* that you are printing, ie. all escaping
has been done before IDLE/Tk even sees the text. So it must have been
pymssql that returns different data in each case.

It could be that the DB-API does such things, see

http://msdn2.microsoft.com/en-us/library/aa937147(SQL.80).aspx

Apparently, they do the OEMtoANSI conversion when you run a console
application (i.e. python.exe), whereas they don't convert when running
a GUI application (pythonw.exe).

I'm not quite sure how they find out whether the program is a console
application or not; the easiest thing to do might be to turn the
autoconversion off on the server.


Regards,
Martin

 

[iu2]

If you are claimaing that the program

Apparently, they do the OEMtoANSI conversion when you run a console
application (i.e. python.exe), whereas they don't convert when running
a GUI application (pythonw.exe).

I'm not quite sure how they find out whether the program is a console
application or not; the easiest thing to do might be to turn the
autoconversion off on the server.

Regards,
Martin

True! It's amazing, I've just written a little code that reads from
the database and writes the data to a file.
Then I ran the code with both python.exe and pythonw.exe and got the
two kinds of results - the IDLE one and the eclipse one!