End of line problem with xmlrpc

[Hubert Martin-deidier]

Hello all,

I try to transfer binary files between two (Redmine) servers. By the
way, I use the ruby language to do it and all informations are exchanged
through the XML-RPC protocol.

Unfortunatly everything success except that transfered binary files
doesn't work. For instance a word file can not be open.
If I compare the original binary file with the transfered binary file, I
can see that everything is similar except that the 'CR' character is
always replaced with a 'LF' character and 'CR LF' characters are always
replaced with 'LF' characters.

I open the file on local server in this way :
binary_content_to_transfer = File.open(binary_file_path_path,"rb")
{|io| io.read}

I transfer the binary_content_to_transfer with either :string (I also
try with [:base64] but same problem occurs)

On remote server, I save content int this way :
binary_remote_file = File.new(binary_remote_file_path, "wb")
binary_remote_file.write(binary_content_to_transfer)
attachment_file.close

I also try many other adjustments to understand where is the "mistake".
All the tests that I made let me think that the XML-RPC protocol is
responsible of the problem :
- I try to execute directly the same read and write code (without using
the XMLRPC procedure) => The created file is perfectly similar with the
original one.
- I try to use different types for transfering data string and
:base64) => Result is the same.
- I try different way of reading and writing files (wb, w, rb, r) =>
Result is worst.

Is anyone has an idea or an alternative solution to transfer binary
files ?

Thanks in advance.

 

[Robert Klemme]

Hello all,

I try to transfer binary files between two (Redmine) servers. By the
way, I use the ruby language to do it and all informations are exchanged
through the XML-RPC protocol.

Unfortunatly everything success except that transfered binary files
doesn't work. For instance a word file can not be open.
If I compare the original binary file with the transfered binary file, I
can see that everything is similar except that the 'CR' character is
always replaced with a 'LF' character and 'CR LF' characters are always
replaced with 'LF' characters.

I open the file on local server in this way :
binary_content_to_transfer = File.open(binary_file_path_path,"rb")
{|io| io.read}

I transfer the binary_content_to_transfer with either :string (I also
try with [:base64] but same problem occurs)

On remote server, I save content int this way :
binary_remote_file = File.new(binary_remote_file_path, "wb")
binary_remote_file.write(binary_content_to_transfer)
attachment_file.close

I also try many other adjustments to understand where is the "mistake".
All the tests that I made let me think that the XML-RPC protocol is
responsible of the problem :
- I try to execute directly the same read and write code (without using
the XMLRPC procedure) => The created file is perfectly similar with the
original one.
- I try to use different types for transfering data string and
:base64) => Result is the same.

This one irritates me: by using base64 encoding you should be fine,
shouldn't you?

- I try different way of reading and writing files (wb, w, rb, r) =>
Result is worst.

Is anyone has an idea or an alternative solution to transfer binary
files ?

I'd say that encoding it in some format like base64 would be the right
way to go. That way also text editors don't choke if you use them to
look at the XML message.

Kind regards

robert

 

[Hubert Martin-deidier]

This one irritates me: by using base64 encoding you should be fine,
shouldn't you?

I also agree with you, I can even say that I completly agree with you.
I quicky look at source code of XMLRPC protol but I'm afraid that for
base64 there is a stirng conversion that may explain this problem. I'm
not sure of what I'm saying.

Someone know If there is a way to look at the XML message sent. In this
way I can determine if the problem is from the client (emission of the
file) or from the server (reception of the file)

 

[Robert Klemme]

I also agree with you, I can even say that I completly agree with you.
I quicky look at source code of XMLRPC protol but I'm afraid that for
base64 there is a stirng conversion that may explain this problem. I'm
not sure of what I'm saying.

As far as I can see this works ok on 1.9.1

irb(main):011:0> raw =3D "".encode! "BINARY"
=3D> ""
irb(main):012:0> raw.encoding
=3D> #<Encoding:ASCII-8BIT>
irb(main):013:0> 255.times {|i| raw << i}
=3D> 255
irb(main):014:0> raw.length
=3D> 255
irb(main):015:0> raw.bytesize
=3D> 255
irb(main):016:0> enc =3D Base64.encode64 raw
=3D> "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKiss\nLS4vMDE=
yMzQ1Njc4OTo7PD0+P0BBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZ\nWltcXV5fYGFiY2RlZmdo=
aWprbG1ub3BxcnN0dXZ3eHl6e3x9fn+AgYKDhIWG\nh4iJiouMjY6PkJGSk5SVlpeYmZqbnJ2en=
6ChoqOkpaanqKmqq6ytrq+wsbKz\ntLW2t7i5uru8vb6/wMHCw8TFxsfIycrLzM3Oz9DR0tPU1d=
bX2Nna29zd3t/g\n4eLj5OXm5+jp6uvs7e7v8PHy8/T19vf4+fr7/P3+\n"
irb(main):017:0> enc.encoding
=3D> #<Encoding:US-ASCII>
irb(main):018:0> dec =3D Base64.decode64 enc
=3D> "\x00\x01\x02\x03\x04\x05\x06\a\b\t\n\v\f\r\x0E\x0F\x10\x11\x12\x13\x1=
4\x15\x16\x17\x18\x19\x1A\e\x1C\x1D\x1E\x1F
!\"\#$%&'()*+,-./0123456789:;<=3D>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdef=
ghijklmnopqrstuvwxyz{|}~\x7F\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8A\x8=
B\x8C\x8D\x8E\x8F\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9A\x9B\x9C\x9D\x=
9E\x9F\xA0\xA1\xA2\xA3\xA4\xA5\xA6\xA7\xA8\xA9\xAA\xAB\xAC\xAD\xAE\xAF\xB0\=
xB1\xB2\xB3\xB4\xB5\xB6\xB7\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF\xC0\xC1\xC2\xC3=
\xC4\xC5\xC6\xC7\xC8\xC9\xCA\xCB\xCC\xCD\xCE\xCF\xD0\xD1\xD2\xD3\xD4\xD5\xD=
6\xD7\xD8\xD9\xDA\xDB\xDC\xDD\xDE\xDF\xE0\xE1\xE2\xE3\xE4\xE5\xE6\xE7\xE8\x=
E9\xEA\xEB\xEC\xED\xEE\xEF\xF0\xF1\xF2\xF3\xF4\xF5\xF6\xF7\xF8\xF9\xFA\xFB\=
xFC\xFD\xFE"
irb(main):019:0> dec.length
=3D> 255
irb(main):020:0> dec.bytesize
=3D> 255
irb(main):021:0> dec.encoding
=3D> #<Encoding:ASCII-8BIT>
irb(main):022:0> raw =3D=3D dec
=3D> true
irb(main):023:0>

Someone know If there is a way to look at the XML message sent. In this
way I can determine if the problem is from the client (emission of the
file) or from the server (reception of the file)

Dunno that library. Someone else might be able to help. If
everything fails you can use a packet sniffer and look at the network
traffic.

Kind regards

robert


--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Phillip Gawlowski]

I also agree with you, I can even say that I completly agree with you.
I quicky look at source code of XMLRPC protol but I'm afraid that for
base64 there is a stirng conversion that may explain this problem. I'm
not sure of what I'm saying.

That's probably the issue. CR and LF are line ending characters. And
Windows, Linux, and UNIX deal with them differently. If your string
conversion happens on a *NIX, Windows applications won't be able to
handle the files gracefully, like Word. (Windows expects CR LF, while
Linux produces LF, and Mac OS CR; or something like that).

Someone know If there is a way to look at the XML message sent. In this
way I can determine if the problem is from the client (emission of the
file) or from the server (reception of the file)

You could hook up a logger.

 

[Robert Klemme]

That's probably the issue. CR and LF are line ending characters. And
Windows, Linux, and UNIX deal with them differently. If your string
conversion happens on a *NIX, Windows applications won't be able to handle
the files gracefully, like Word. (Windows expects CR LF, while Linux
produces LF, and Mac OS CR; or something like that).

That should not be the case since a) he is doing reading and writing
in binary and b) base64 encoding does not care whether you have CR or
CRLF as line ending. It does not influence the decoding.

Admittedly we haven't seen the sample code so there might be a quirk
somewhere we haven't thought of yet.

Kind regards

robert