Problem with Net/HTTP

[Peer Allan]

Hi all,

I am running into a serious problem with a custom library I am building
in Ruby. Basically I am writing a Ruby interface to a custom web
service that we have here at our company. The problem I am running into
is that I am posting the XML document to the service and it is
responding, but net/http is telling me there is no body
(response.body.empty? => true). If I post to the service from a web
form or using PHP I get a response. This only seems to happen when
there is an error with the request, for example the request is missing
an postal code. The service returns an XML doc with the errors but
net/http won't let me see it. A 200 status code is returned and I can
not find no obvious difference between the headers of a successful and
failed request. I can't even find a way to see the raw response the
net/http received so I can check to see if it is just parsing it
incorrectly.

I am open to any suggestions at this point as this is a big problem if I
can't get this working. Thanks

Peer

 

[Brian Candler]

I am running into a serious problem with a custom library I am building
in Ruby. Basically I am writing a Ruby interface to a custom web
service that we have here at our company. The problem I am running into
is that I am posting the XML document to the service and it is
responding, but net/http is telling me there is no body
(response.body.empty? => true).

Can you prove the response is there? The best way to do this is

tcpdump -i eth0 -n -s0 -A tcp port 80

and look at the raw IP packets being sent by Ruby, and the response coming
back.

If you are on Windoze, then a sniffer like wireshark will do the trick.

I can't even find a way to see the raw response the
net/http received so I can check to see if it is just parsing it
incorrectly.

There are two possible cases: Ruby is sending the request differently to
PHP, or Ruby is parsing the response differently. Using tcpdump you should
be able to find out (and compare what PHP is sending with what Ruby is
sending)

HTH,

Brian.

 

[Peer Allan]

Thanks Brian,

That gave me somewhere to do, but I failed to mention that the web
service requires an SSL connection so everything in the dump is
encrypted. Nuts! Not sure what to do now. I may try to generate the
request on my own.

Thanks

Peer

 

[Brian Candler]

That gave me somewhere to do, but I failed to mention that the web
service requires an SSL connection so everything in the dump is
encrypted. Nuts! Not sure what to do now. I may try to generate the
request on my own.

Are you sure you used net/https and enabled ssl? This question seems to have
arisen several times in the last day or two.

 

[Peer Allan]

Unfortunately, I am sure. Most of the time there is no problem with the
request. Just the occasional error response is not showing a body.

However, after playing with cURL to get a good look at the whole request
I think I may have found the problem. The failing requests are
returning headers that look like this:

=start
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Date: Fri, 25 May 2007 11:11:28 GMT
Server: Apache
WWW-Authenticate: Basic realm="none"
Content-Length: 692
Cache-Control: private
Content-Type: text/xml

=end

I think that the HTTP 100 response code is confusing the response
somehow. I can't find anyone else who has had to deal with this as it
isn't exactly a redirection. I will keep looking and let you know what
I find.

Thanks again for your help

Peer

 

[Brian Candler]

However, after playing with cURL to get a good look at the whole request
I think I may have found the problem. The failing requests are
returning headers that look like this:

=start
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Date: Fri, 25 May 2007 11:11:28 GMT
Server: Apache
WWW-Authenticate: Basic realm="none"
Content-Length: 692
Cache-Control: private
Content-Type: text/xml

=end

See RFC 2616 section 8.2.3. The server will only send a 100 Continue if the
client has explicitly asked for it, and therefore the client should be able
to handle it.

Another approach you can do is to build a plain TCP to SSL proxy
(e.g. stunnel), and point your client at stunnel with plain HTTP. This will
allow you to tcpdump the plain connection.

Alternatively, start modifying net/http to show you the raw data sent and
received into openssl, or make an IO proxy object which does that.

Regards,

Brian.