xmlrpclib and binary data as normal parameter strings

[Rune Froysa]

Trying something like::
import xmlrpclib
svr = xmlrpclib.Server("http://127.0.0.1:8000")
svr.test("\x1btest")

Failes on the server with::
xml.parsers.expat.ExpatError: not well-formed (invalid token)

(Smaller test-case: xmlrpclib.loads(xmlrpclib.dumps(('\x1btest',))))

Shouldn't this be allowed?

From http://www.xmlrpc.com/spec ::
Any characters are allowed in a string except < and &, which are
encoded as &lt; and &amp;. A string can be used to encode binary
data.

From http://www.w3.org/TR/2004/REC-xml11-20040204/#dt-character ::
Consequently, XML processors MUST accept any character in the range
specified for Char
...
Char ::= [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

(I'm aware that xmlrpclib.Binary can be used as an ugly work-around.)

 

[Fredrik Lundh]

From http://www.w3.org/TR/2004/REC-xml11-20040204/#dt-character ::

Consequently, XML processors MUST accept any character in the range
specified for Char
...
Char ::= [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

you're looking at the XML 1.1 specification. don't do that. nobody uses 1.1.

here's the 1.0 version:

http://www.w3.org/TR/REC-xml/#dt-character

Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

</F>

 

[Richard Brodie]

From http://www.xmlrpc.com/spec ::
Any characters are allowed in a string except < and &, which are
encoded as &lt; and &amp;. A string can be used to encode binary
data.

the XMLRPC specification is worded pretty loosely. Obviously characters
forbidden in XML will be problematic.

From http://www.w3.org/TR/2004/REC-xml11-20040204/#dt-character ::

That's XML 1.1; it's very rarely used in practice. Can XMLRPC use XML 1.1?
In principle, undefined, as far as I know. In practice, not in Python anyway.

 

[Rune Froysa]

the XMLRPC specification is worded pretty loosely. Obviously characters
forbidden in XML will be problematic.

IMHO that is not obvious. The API-user wants to call a function on a
remote server with an arbitrary argument. What the transport-layer
does to achieve this should be completly transparent to the user.

We already have code that takes care of "&<>". Ideally we should do
that for other data as well. Unfortunately the XML-spec doesn't allow
us to use character references "&%#x1b;".

With the patch below, setting BINARY_AS_STRING=True will cause all
strings to be transparently base64-encoded during transport.
Unfortunately it will do this for all incoming binary data, thus
breaking aplications that explicitly uses the Binary class for this
purpose.

IMHO the xmlrpc spec should be updated to allow something like this.

--- /local/lib/python2.3/xmlrpclib.py 2005-01-10 10:30:43.000000000 +0100
+++ xmlrpclib.py 2005-04-20 09:56:11.000000000 +0200
@@ -177,6 +177,8 @@

__version__ = "1.0.1"

+BINARY_AS_STRING = False
+
# xmlrpc integer limits
MAXINT = 2L**31-1
MININT = -2L**31
@@ -652,6 +654,11 @@
dispatch[FloatType] = dump_double

def dump_string(self, value, write, escape=escape):
+ if BINARY_AS_STRING:
+ self.write = write
+ Binary(value).encode(self)
+ del self.write
+ return
write("<value><string>")
write(escape(value))
write("</string></value>\n")
@@ -843,7 +850,10 @@
def end_base64(self, data):
value = Binary()
value.decode(data)
- self.append(value)
+ if BINARY_AS_STRING:
+ self.append(value.data)
+ else:
+ self.append(value)
self._value = 0
dispatch["base64"] = end_base64