Help needed with python unicode cgi-bin script

weheh · Dec 10, 2007

Dear web gods:

After much, much, much struggle with unicode, many an hour reading all the
examples online, coding them, testing them, ripping them apart and putting
them back together, I am humbled. Therefore, I humble myself before you to
seek guidance on a simple python unicode cgi-bin scripting problem.

My problem is more complex than this, but how about I boil down one sticking
point for starters. I have a file with a Spanish word in it, "años", which I
wish to read with:

#!C:/Program Files/Python23/python.exe

STARTHTML= u'''Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
</head>
<body>
'''
ENDHTML = u'''
</body>
</html>
'''
print STARTHTML
print open('c:/test/spanish.txt','r').read()
print ENDHTML

Instead of seeing "año" I see "a?o". BAD BAD BAD
Yet, if I open the file with the browser (IE/Mozilla), I see "año." THIS IS
WHAT I WANT

WHAT GIVES?

Next, I'll get into codecs and stuff, but how about starting with this?

The general question is, does anybody have a complete working example of a
cgi-bin script that does the above properly that they'd be willing to share?
I've tried various examples online but haven't been able to get any to work.
I end up seeing hex code for the non-ascii characters u'a\xf1o', and later
on 'a\xc3\xb1o', which are also BAD BAD BAD.

Thanks -- your humble supplicant.

Jack · Dec 10, 2007

You probably need to set stdout mode to binary. They are not by default on
Windows.

Martin v. Löwis · Dec 10, 2007

My problem is more complex than this, but how about I boil down one sticking

point for starters. I have a file with a Spanish word in it, "años", which I
wish to read with:

What is the encoding of that file? Without a correct answer to that
question, you will not be able to achieve what you want.

Possible answers are "iso-8859-1", "utf-8", "windows-1252", and "cp850"
(these all support the word "años")

Instead of seeing "año" I see "a?o". BAD BAD BAD

I don't see anything here. Where do you see the question mark? Did you
perhaps run the CGI script in a web server, and pointed your web browser
to the web page, and saw the question mark in the web browser?

WHAT GIVES?

Sending "Content-type: text/html" is not enough. The web browser needs
to know what the encoding is. So you should send

Content-type: text/html; charset="your-encoding-here"

Use "extras/page information" in Firefox to find out what the web
browser thinks the encoding of the page is.

Regards,
Martin

P.S. Please, stop shouting.

weheh · Dec 10, 2007

Thanks for the reply, Jack. I tried setting mode to binary but it had no
affect.

weheh · Dec 10, 2007

Hi Martin, thanks for your response. My updates are interleaved with your
response below:

What is the encoding of that file? Without a correct answer to that
question, you will not be able to achieve what you want.

I don't know for sure the encoding of the file. I'm assuming it has no
intrinsic encoding since I copied the word "año" into vim and then saved it
as the example text file called, "spanish.txt".

Possible answers are "iso-8859-1", "utf-8", "windows-1252", and "cp850"
(these all support the word "año")

I don't see anything here. Where do you see the question mark? Did you
perhaps run the CGI script in a web server, and pointed your web browser
to the web page, and saw the question mark in the web browser?

The cgi-bin scripts prints to stdout, i.e. to my browser, and when I use
print I see a square box where the ñ should be. When I use print repr(...) I
see 'a\xf1o'. I never see the desired 'ñ' character.

Sending "Content-type: text/html" is not enough. The web browser needs

to know what the encoding is. So you should send

Content-type: text/html; charset="your-encoding-here"

Sorry, somehow my cut and paste job into outlook missed the exact line you
had above that specifies encoding tp be set as "utf8", but it's there in my
program. Not to worry.

Use "extras/page information" in Firefox to find out what the web
browser thinks the encoding of the page is.

Firefox says the page is UTF8.

P.S. Please, stop shouting.

OK, it's just that it hurts when I've been pulling my hair out for days on
end over a single line of code. I don't want to go bald just yet.

John Machin · Dec 10, 2007

Hi Martin, thanks for your response. My updates are interleaved with your
response below:

I don't know for sure the encoding of the file. I'm assuming it has no
intrinsic encoding since I copied the word "año" into vim and then saved it
as the example text file called, "spanish.txt".

Every text file encoded, and very few of them are tagged with the name
of the encoding in any reliable fashion.

The cgi-bin scripts prints to stdout, i.e. to my browser, and when I use
print I see a square box where the ñ should be. When I use print repr(....) I
see 'a\xf1o'. I never see the desired 'ñ' character.

Sending "Content-type: text/html" is not enough. The web browser needs

Sorry, somehow my cut and paste job into outlook missed the exact line you
had above that specifies encoding tp be set as "utf8", but it's there in my
program. Not to worry.

Firefox says the page is UTF8.

OK, it's just that it hurts when I've been pulling my hair out for days on
end over a single line of code. I don't want to go bald just yet.

Forget for the moment what you see in the browser. You need to find
out how your file is encoded.

Look at your file using
print repr(open('c:/test/spanish.txt','rb').read())

If you see 'a\xf1o' then use charset="windows-1252" else if you see
'a\xc3\xb1o' then use charset="utf-8" else ????

Based on your responses to Martin, it appears that your file is
actually windows-1252 but you are telling browsers that it is utf-8.

Another check: if the file is utf-8, then doing
open('c:/test/spanish.txt','rb').read().decode('utf8')
should be OK; if it's not valid utf8, it will complain.

Yet another check: open the file with Notepad. Do File/SaveAs, and
look at the Encoding box -- ANSI or UTF-8?

HTH,
John

Jack · Dec 11, 2007

Just want to make sure, how exactly are you doing that?

weheh · Dec 11, 2007

import sys

if sys.platform == "win32":
import os, msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

weheh · Dec 11, 2007

Hi John:
Thanks for responding.

Look at your file using
> print repr(open('c:/test/spanish.txt','rb').read())

If you see 'a\xf1o' then use charset="windows-1252"

I did this ... no change ... still see 'a\xf1o'

else if you see 'a\xc3\xb1o' then use charset="utf-8" else ????

Based on your responses to Martin, it appears that your file is
actually windows-1252 but you are telling browsers that it is utf-8.

Another check: if the file is utf-8, then doing
> open('c:/test/spanish.txt','rb').read().decode('utf8')
should be OK; if it's not valid utf8, it will complain.

No. this causes decode error:

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-4: invalid
data
args = ('utf8', 'a\, 1, 5, 'invalid data')
encoding = 'utf8'
end = 5
object = 'a\xf1o'
reason = 'invalid data'
start = 1

Yet another check: open the file with Notepad. Do File/SaveAs, and
look at the Encoding box -- ANSI or UTF-8?

Notepad says it's ANSI

Thanks. What now? Also, this is a general problem for me, whether I read
from a file or read from an html text field, or read from an html text area.
So I'm looking for a general solution. If it helps to debug by reading from
textarea or text field, let me know.

John Machin · Dec 11, 2007

Hi John:
Thanks for responding.

I did this ... no change ... still see 'a\xf1o'

So it's not utf-8, it's windows-1252, so stop lying to browsers: like
I said, use charset="windows-1252"

No. this causes decode error:

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-4: invalid
data

No what? YES, the "decode error" is complaining that the data supplied
is NOT valid utf-8 data. So it's not utf-8, it's windows-1252, so stop
lying to browsers: like I said, use charset="windows-1252"

args = ('utf8', 'a\, 1, 5, 'invalid data')
encoding = 'utf8'
end = 5
object = 'a\xf1o'
reason = 'invalid data'
start = 1

Notepad says it's ANSI

That's correct (in Microsoft jargon) -- it's NOT utf-8. It's
windows-1252, so stop lying to browsers: like I said, use
charset="windows-1252"

Thanks. What now?

Listen to the Bellman: "What I tell you three times is true".
Your file is encoded using windows-1252, NOT utf-8.
You need to use charset="windows-1252".

Also, this is a general problem for me, whether I read
from a file or read from an html text field, or read from an html text area.
So I'm looking for a general solution. If it helps to debug by reading from
textarea or text field, let me know.

If you are creating a file, you should know what its encoding is. As I
said earlier, *every* file is encoded -- so-called "Unicode" files on
Windows are encoded using utf16le. If you don't explicitly specify the
encoding, it will typically be the default encoding for your locale
(e.g. cp1252 in Western Europe etc).

If you are reading a file created by others and its encoding is not
known, you will have inspect the file and/or guess (using whatever
knowledge you have about the language/locale of the creator).

"whether I ... read from an html text field, or read from an html text
area": isn't that what "charset" is for?

HTH,
John

Martin v. Löwis · Dec 11, 2007

No what? YES, the "decode error" is complaining that the data supplied

is NOT valid utf-8 data. So it's not utf-8, it's windows-1252, so stop
lying to browsers: like I said, use charset="windows-1252"

I think weheh can manage to resist good advise for a long time.

Regards,
Martin

weheh · Dec 11, 2007

John & Martin,

Thanks for your help and kind words of encouragement. Still, what you have
suggested doesn't seem to work, unless I'm not understanding your directive
to encode as 'windows-1252'. Here's my program in full:

#!C:/Program Files/Python23/python.exe
import cgi, cgitb
import sys, codecs
import os,msvcrt

cgitb.enable()

print u"""Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" lang="en,sp,fr"
xml:lang="en,sp,fr">
<head>
<meta http-equiv="content-type" content="text/html; charset=windows-1252" />
<meta http-equiv="content-language" content="en,fr,sp" />
</head>
<body>
"""
if sys.platform == 'win32':
msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

x = repr(open('c:/test/spanish.txt','rb').read())
print '',x,'# first print'

x = open('c:/test/spanish.txt','rb').read()
print '',x,'# second print'

x = repr((open('c:/test/spanish.txt','rb').read()).decode('windows-1252'))
print '',x,'# third print'

print """
</body>
</html>
"""

The output of the program is this:

'a\xf1o\r\n' # first print

a?o # second print #### Note that there is no ñ between the a and the o,
only a box

u'a\xf1o\r\n' # third print

So what do you advise?

Martin v. Löwis · Dec 11, 2007

Thanks for your help and kind words of encouragement. Still, what you have

suggested doesn't seem to work, unless I'm not understanding your directive
to encode as 'windows-1252'.

Please read John's message again. Nowhere he said you should "encode as
'windows-1252'". Instead, he said 'use charset="windows-1252"'.

print u"""Content-Type: text/html

In one message, you said "somehow my cut and paste job into outlook
missed the exact line you had above that specifies encoding tp be set as
"utf8", but it's there in my program"

What EXACTLY do you mean by that? Where PRECISELY did you set
"encoding tp" (what is tp?) to "utf8"?

I suggested doing that in the Content-type CGI/HTTP header, but
perhaps you do it in the http-equiv HTML element instead:

<meta http-equiv="content-type" content="text/html; charset=windows-1252" />

So this should work fine.

x = repr(open('c:/test/spanish.txt','rb').read())
print '',x,'# first print'

It's no suprise that this prints an escaped text, because that's
what repr() does.

x = open('c:/test/spanish.txt','rb').read()
print '',x,'# second print'

This should work fine.

a?o # second print #### Note that there is no ñ between the a and the o,
only a box

What does the browser say what the encoding of the page is?

What browser are you using, and did you configure it to default to
UTF-8 for all pages? (which you should not have done)

Try "telnet server 80", then type

GET /path HTTP/1.1<enter>
Host: server<enter>
<enter>

and report what response from the server is (the complete one,
not just the character in question)

Regards,
Martin

weheh · Dec 12, 2007

What does the browser say what the encoding of the page is?

What browser are you using, and did you configure it to default to
UTF-8 for all pages? (which you should not have done)

Browser is both IE and Firefox. IE is defaulting to UTF8. If I force it to
"Encoding > Western European (Windows)" it shows the ñ. The browser encoding
"Autoselect" feature is enabled, yet it always seems to default to UTF8. Any
idea how to change that?

Is there something I can put in html that forces it to do that?

I'm using Apache and have the following line in my http.conf file:

AddDefaultCharset utf-8

Is this a problem?

Try "telnet server 80", then type

GET /path HTTP/1.1<enter>
Host: server<enter>
<enter>

and report what response from the server is (the complete one,
not just the character in question)

OK, telnet session yields this:

HTTP/1.1 200 OK
Date: Tue, 11 Dec 2007 23:58:02 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8

1f8
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.or
g/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html
xmlns="http://www.w3.org/1999/xh
tml" lang="en,sp,fr" xml:lang="en,sp,fr">
<head>
<meta
http-equiv="content-type" c
ontent="text/html; charset=utf8" />
<meta http-equiv="content-language"
content="
en,fr,sp" />
</head>
<body>

 a±o
# first print
 'a\xf1o\r\n' # second print
 'a\xf1o\r\n' # third
pri
nt
 'a\xf1o\r\n' # third print

</body>
</html>

0

Connection to host lost.

weheh · Dec 12, 2007

p.s. I modified the code to break things out more explicitly:

#!C:/Program Files/Python23/python.exe
import cgi, cgitb
import sys, codecs
import os,msvcrt

cgitb.enable()

print u"""Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" lang="en,sp,fr"
xml:lang="en,sp,fr">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf8" />
<meta http-equiv="content-language" content="en,fr,sp" />
</head>
<body>
"""
if sys.platform == 'win32':
msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

x = open('c:/test/spanish.txt','rb').read()
print '',x,'# first print'

x = open('c:/test/spanish.txt','rb').read()
x = repr(x)
print '',x,'# second print'

x = open('c:/test/spanish.txt','rb').read()
x = repr(x)
x = x.decode('windows-1252')
print '',x,'# third print'

x = open('c:/test/spanish.txt','rb').read()
x = repr(x)
x = x.decode('windows-1252')
x = x.encode('utf8')
print '',x,'# third print'

print """
</body>
</html>
"""

(The last print should read "fourth print")

John Machin · Dec 12, 2007

p.s. I modified the code to break things out more explicitly:

#!C:/Program Files/Python23/python.exe
import cgi, cgitb
import sys, codecs
import os,msvcrt

cgitb.enable()

print u"""Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" lang="en,sp,fr"
xml:lang="en,sp,fr">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf8"

WTF?

weheh · Dec 12, 2007

John and Martin,

Thanks for your help. However, I have identified the culprit to be with
Apache and the command:
AddDefaultCharset utf-8
which forces my browser to utf-8 encoding.

It looks like your suggestions to change charset were incorrect. My example
works equally well with charset=utf8 as it does with charset=windows-1252.

Incidentally, next time, if you really want to be helpful, might I suggest
you leave out the mocking. I could care less, myself, but someone else might
have gotten their feelings hurt. And in the end, it doesn't make you look
good.

Thanks again. Cheers.

Dennis Lee Bieber · Dec 12, 2007

HTTP/1.1 200 OK
Date: Tue, 11 Dec 2007 23:58:02 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8

If I understand the preceding posts, you do NOT have UTF8 text... So
this header is LYING.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

Duncan Booth · Dec 12, 2007

weheh said:
John and Martin,

Thanks for your help. However, I have identified the culprit to be
with Apache and the command:
AddDefaultCharset utf-8
which forces my browser to utf-8 encoding.

It looks like your suggestions to change charset were incorrect. My
example works equally well with charset=utf8 as it does with
charset=windows-1252.

Incidentally, next time, if you really want to be helpful, might I
suggest you leave out the mocking. I could care less, myself, but
someone else might have gotten their feelings hurt. And in the end, it
doesn't make you look good.

Thanks again. Cheers.

FWIW, the code you posted only ever attempted to set the character set
encoding using an html meta tag which is the wrong place to set it. The
encoding specified in the HTTP headers always takes precedence. This is why
the default charset setting in Apache was the one which applied.

What you should have been doing was setting the encoding in the content-
type header. i.e. in this line of your code:

print u"""Content-Type: text/html

You should have changed it to read:

Content-Type: text/html; charset=windows-1252

but because you didn't Apache was quietly changing it to read:

Content-Type: text/html; charset=utf-8

weheh · Dec 12, 2007

Hi Duncan, thanks for the reply.

FWIW, the code you posted only ever attempted to set the character set
encoding using an html meta tag which is the wrong place to set it. The
encoding specified in the HTTP headers always takes precedence. This is
why
the default charset setting in Apache was the one which applied.

What you should have been doing was setting the encoding in the content-
type header. i.e. in this line of your code:

print u"""Content-Type: text/html

You should have changed it to read:

Content-Type: text/html; charset=windows-1252

but because you didn't Apache was quietly changing it to read:

Content-Type: text/html; charset=utf-8

Will this work under the following situation? Let's say the user is filling
out a text field on a form on my website. The user has their browser
encoding set to utf8. My website has charset=windows-1252 as you indicate
above. Will I run into a conflict somewhere?

Need help with programming in python for class (beginner level)	7	Nov 30, 2013
Help with Visual Lightbox: Scripts	2	May 3, 2023
Help with code	0	Jun 12, 2022
Help with my responsive home page	2	Dec 14, 2022
CGI script parameter error.	3	Dec 20, 2010
Right solution to unicode error?	21	Nov 7, 2012
Python CGI script and CSS style sheet	2	Jan 23, 2008
How to display unicode with the CGI module?	7	Nov 24, 2007

Help needed with python unicode cgi-bin script

weheh

Jack

Martin v. Löwis

weheh

weheh

John Machin

Jack

weheh

weheh

John Machin

Martin v. Löwis

weheh

Martin v. Löwis

weheh

weheh

John Machin

weheh

Dennis Lee Bieber

Duncan Booth

weheh

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads