How to ask sax for the file encoding

[Edward K. Ream]

Following the usual cookbook examples, my app parses an open file as
follows::



parser = xml.sax.make_parser()

parser.setFeature(xml.sax.handler.feature_external_ges,1)

# Hopefully the content handler can figure out the encoding from the <?xml>
element.

handler = saxContentHandler(c,inputFileName,silent)

parser.setContentHandler(handler)

parser.parse(theFile)



Can anyone tell me how the content handler can determine the encoding of the
file? Can sax provide this info?



Thanks!



Edward

 

[Fredrik Lundh]

Can anyone tell me how the content handler can determine the encoding of the file? Can sax
provide this info?

there is no encoding on the "inside" of an XML document; it's all Unicode.

</F>

 

[Edward K. Ream]

Can anyone tell me how the content handler can determine the encoding of

there is no encoding on the "inside" of an XML document; it's all Unicode.

True, but sax is reading the file, so sax is producing the unicode, so it
should (must) be able to determine the encoding. Furthermore, xml files
start with lines like:

<?xml version="1.0" encoding="utf-8"?>

so it would seem reasonable for sax to be able to return 'utf-8' somehow.
Am I missing something?

Edward

 

[Diez B. Roggisch]

True, but sax is reading the file, so sax is producing the unicode, so it
should (must) be able to determine the encoding.

It is, by reading the xml header.

Furthermore, xml files
start with lines like:

<?xml version="1.0" encoding="utf-8"?>

so it would seem reasonable for sax to be able to return 'utf-8' somehow.
Am I missing something?

That sax outputs unicode, which has no encoding associated anymore. And thus
it is a pretty much irrelevant information. It _could_ be retained, but for
what purpose?

Diez

 

[Fredrik Lundh]

<?xml version="1.0" encoding="utf-8"?>

so it would seem reasonable for sax to be able to return 'utf-8' somehow.

why? that's an encoding detail, and should be completely irrelevant for
your application.

Am I missing something?

you're confusing artifacts of an external serialization format with the actual
data model. don't do that, if you can avoid it.

what's your use case ?

</F>

 

[Edward K. Ream]

[The value of the encoding field] _could_ be retained, but for what

purpose?

I'm asking this question because my app needs it Imo, there is *no*
information in any xml file that can be considered irrelvant. My app will
want to know the original encoding when writing the file.

Edward

 

[Diez B. Roggisch]

[The value of the encoding field] _could_ be retained, but for what
purpose?

I'm asking this question because my app needs it
Imo, there is *no*
information in any xml file that can be considered irrelvant.

It sure is! The encoding _is_ irrelevant, in the very moment you get unicode
strings. The order of attributes is irrelevant. There is plenty of
irrelevant whitespace. And so on...

My app will
want to know the original encoding when writing the file.

When your app needs it, whatfor does it need it? If you write out xml again,
use whatever encoding suits you best. If you don't, use the encoding that
the subsequent application or processing step needs.

Diez

 

[Edward K. Ream]

The encoding _is_ irrelevant, in the very moment you get unicode strings.

We shall have to disagree about this. My use case is perfectly reasonable,
imo.

If you write out xml again, use whatever encoding suits you best.

What suits me best is what the *user* specified, and that got put in the
first xml line.
I'm going to have to parse this line myself.

Edward

 

[Rob Wolfe]

Can anyone tell me how the content handler can determine the encoding of the
file? Can sax provide this info?

Try this:

<code>
from xml.parsers import expat

s = """<?xml version='1.0' encoding='iso-8859-1'?>
<book>
<title>Title</title>
<chapter>Chapter 1</chapter>
</book>
"""

class MyParser(object):
def XmlDecl(self, version, encoding, standalone):
print "XmlDecl", version, encoding, standalone

def Parse(self, data):
Parser = expat.ParserCreate()
Parser.XmlDeclHandler = self.XmlDecl
Parser.Parse(data, 1)

parser = MyParser()
parser.Parse(s)
</code>

 

[Irmen de Jong]

What suits me best is what the *user* specified, and that got put in the
first xml line.
I'm going to have to parse this line myself.

Please consider adding some elements to the document itself that
describe the desired output format, such as:

....
<output>
<encoding>utf-8</encoding>
</output>
....

This allows the client to specify the encoding it wants to receive
the document in, even if it's different than the encoding it used
to make the first document. More flexibility. Less fooling around.

--Irmen

 

[Fredrik Lundh]

I'm asking this question because my app needs it Imo, there is *no*
information in any xml file that can be considered irrelvant.

the encoding isn't *in* the XML file, it's an artifact of the
serialization model used for a specific XML infoset. the XML
data is pure Unicode.

</F>

 

[Fredrik Lundh]

What suits me best is what the *user* specified, and that got put in the
first xml line.

are you expecting your users to write XML by hand? ouch.

</F>

 

[Edward K. Ream]

are you expecting your users to write XML by hand?

Of course not. Leo has the following option:

@string new_leo_file_encoding = utf-8

Edward

 

[Edward K. Ream]

Please consider adding some elements to the document itself that
describe the desired output format,

Well, that's what the encoding field in the xml line was supposed to do.
Not a bad idea though, except it changes the file format, and I would really
rather not do that.

Edward

 

[Edward K. Ream]

the encoding isn't *in* the XML file, it's an artifact of the

serialization model used for a specific XML infoset. the XML
data is pure Unicode.

Sorry, but no. The *file* is what I am talking about, and the way it is
encoded does, in fact, really make a difference to some users. They have a
right, I think, to expect that the original encoding gets preserved when the
file is rewritten.

Edward

 

[Edward K. Ream]

Try this:
[snip]
Parser.XmlDeclHandler = self.XmlDecl
[snip]

Excellent! Thanks so much.

Edward

 

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

Can anyone tell me how the content handler can determine the encoding of the
file? Can sax provide this info?

That's not supported in SAX. If you use Expat directly (module pyexpat),
you can set the XmlDeclHandler, which is called when the XML declaration
is received (with the parameters version, encoding, and standalone).
However, as the XML declaration is optional, this callback might
not get invoked.

Regards,
Martin

 

[Irmen de Jong]

describe the desired output format,

Well, that's what the encoding field in the xml line was supposed to do.

As others have tried to explain, the encoding in the xml header is
not part of the document data itself, it says something about the data.
It would be a bad design decision imo to rely on this meta information
if you really meant that information to be part of the data document.

Not a bad idea though, except it changes the file format, and I would really
rather not do that.

XML allows you to easily skip any elements that you think you don't need.

--Irmen

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

As others have tried to explain, the encoding in the xml header is
not part of the document data itself, it says something about the data.
It would be a bad design decision imo to rely on this meta information
if you really meant that information to be part of the data document.

A common problem is to save the data in the same encoding that they
original had; this is what an editor typically does (you may know
Edward Ream for writing editors). XML parsers are notoriously bad
in supporting editors. There are too many lexical details that may
need to be preserved (such as the order of the attributes, and the
spaces inside the opening tag) to make it impractical to report all
that to the application.

IMO, the only way to edit XML on a level that does preserving
of the tiniest lexical details is to edit it as plain text
(i.e. without using an XML parser).

Regards,
Martin

 

[Fredrik Lundh]

A common problem is to save the data in the same encoding that they
original had; this is what an editor typically does (you may know
Edward Ream for writing editors). XML parsers are notoriously bad
in supporting editors. There are too many lexical details that may
need to be preserved (such as the order of the attributes, and the
spaces inside the opening tag) to make it impractical to report all
that to the application.

an editor designed to work on the XML serialization level shouldn't use
a traditional XML parser at all, of course. definitely not SAX or DOM,
or any other infoset-or-higher-level API.

on the other hand, an editor that just happens to use XML as a
serialization format might as well decide on a model representation
and an encoding and stick to it. being tolerant in what it accepts
is a good idea, of course, but being consistent in what it generates
is an even better idea.

</F>