Bug in base64 decoder of xmlrpc?

[Lars Hoss]

Hi,

I am currently implementing support for the MetaWeblog/MovableType API
in my weblog engine. The API is simply a bunch of XML-RPC methods. One
method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64 encoded.
The xmlrpc library automatically decodes the data. Writing the file to
disk looks something like this:

def newMediaObject(blogid, username, password, data)
fileData = data["bits"] # Decoding is handled by xmlrpc automatically
File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes are
missing. Uploading plain text files seems to be ok. But uploading arger
files such as JPEG files will result in corrupted data. The client I am
using is MarsEdit.

Any suggestions?

Regards,
Lars

 

[Daniel Berger]

Hi,

I am currently implementing support for the MetaWeblog/MovableType API
in my weblog engine. The API is simply a bunch of XML-RPC methods. One
method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64 encoded.
The xmlrpc library automatically decodes the data. Writing the file to
disk looks something like this:

def newMediaObject(blogid, username, password, data)
fileData = data["bits"] # Decoding is handled by xmlrpc automatically
File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes are
missing. Uploading plain text files seems to be ok. But uploading arger
files such as JPEG files will result in corrupted data. The client I am
using is MarsEdit.

Any suggestions?

Regards,
Lars

umgekehrt." -- Cocteau, Jean

I remember bringing something like this up with the author last year
after reading Randy Ray's use.perl journal entry at
http://use.perl.org/~rjray/journal/9899 and
http://use.perl.org/~rjray/journal/9922. Randy Ray is the author of
the Perl module RPC::XML, btw.

This was Michael's response at the time (please correct me if things
have changed Michael):

"Implementing [Base64 streaming support] in xmlrpc4r is possible, but
it would require some parts to be redesigned, especially where strings
are returned (e.g. methodResponse).

One problem is that the XML-RPC spec. requires the size of the
XML-message to
be known in the HTTP header (which is send first). Without this
requirement, the whole
XML-RPC message could be streamed out a socket."

Whether anything has been done since then (about a year ago), I don't
know.
If this is unrelated, then I apologize for the noise.

Regards,

Dan

 

[Michael Neumann]

Hi,

I am currently implementing support for the MetaWeblog/MovableType
API

in my weblog engine. The API is simply a bunch of XML-RPC methods.
One

method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64
encoded.

The xmlrpc library automatically decodes the data. Writing the file
to

disk looks something like this:

def newMediaObject(blogid, username, password, data)
fileData = data["bits"] # Decoding is handled by xmlrpc
automatically

File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes
are

missing. Uploading plain text files seems to be ok. But uploading
arger

files such as JPEG files will result in corrupted data. The client I
am

using is MarsEdit.

Any suggestions?

Regards,
Lars

I remember bringing something like this up with the author last year
after reading Randy Ray's use.perl journal entry at
http://use.perl.org/~rjray/journal/9899 and
http://use.perl.org/~rjray/journal/9922. Randy Ray is the author of
the Perl module RPC::XML, btw.

This was Michael's response at the time (please correct me if things
have changed Michael):

"Implementing [Base64 streaming support] in xmlrpc4r is possible, but
it would require some parts to be redesigned, especially where strings
are returned (e.g. methodResponse).

One problem is that the XML-RPC spec. requires the size of the
XML-message to
be known in the HTTP header (which is send first). Without this
requirement, the whole
XML-RPC message could be streamed out a socket."

Whether anything has been done since then (about a year ago), I don't
know.
If this is unrelated, then I apologize for the noise.

Hm, it should work if you pass a XMLRPC::Base64 object as argument, and
not a String. Of course streaming does not work, but that's not related
to the corruption. Maybe the best would be to encode every string as
base64, but that would probably break many clients or servers (other
than xmlrpc4r).

Regards,

Michael

 

[Lars Hoss]

Hi Michael, hi Daniel,

first thanks for your help. After digging in the sources I think I
discovered the problem. The problem is that somehow all "+" characters
in the base64 encoded string get converted into a " " character. I am
still not sure at which point the conversion happens. Probably Rails
thinks that the XML-RPC data is a request parameter and therefore
automatically converts all "+" characters back into " " characters
which is of course ok for regular request parameters. But as I said
before, I am still searching ...

Yours,
Lars

Am 17.01.2005 um 22:17 schrieb Michael Neumann:

Hi,

I am currently implementing support for the MetaWeblog/MovableType API
in my weblog engine. The API is simply a bunch of XML-RPC methods. One
method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64 encoded.
The xmlrpc library automatically decodes the data. Writing the file to
disk looks something like this:

def newMediaObject(blogid, username, password, data)
fileData = data["bits"] # Decoding is handled by xmlrpc automatically
File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes are
missing. Uploading plain text files seems to be ok. But uploading arger
files such as JPEG files will result in corrupted data. The client I am
using is MarsEdit.

Any suggestions?

Regards,
Lars

I remember bringing something like this up with the author last year
after reading Randy Ray's use.perl journal entry at
http://use.perl.org/~rjray/journal/9899 and
http://use.perl.org/~rjray/journal/9922. Randy Ray is the author of
the Perl module RPC::XML, btw.
This was Michael's response at the time (please correct me if things
have changed Michael):
"Implementing [Base64 streaming support] in xmlrpc4r is possible, but
it would require some parts to be redesigned, especially where strings
are returned (e.g. methodResponse).
One problem is that the XML-RPC spec. requires the size of the
XML-message to
be known in the HTTP header (which is send first). Without this
requirement, the whole
XML-RPC message could be streamed out a socket."
Whether anything has been done since then (about a year ago), I don't
know.
If this is unrelated, then I apologize for the noise.

Hm, it should work if you pass a XMLRPC::Base64 object as argument,
and not a String. Of course streaming does not work, but that's not
related to the corruption. Maybe the best would be to encode every
string as base64, but that would probably break many clients or
servers (other than xmlrpc4r).

Regards,

Michael