HTTP POST request --> Ruby server

[Chananya Freiman]

I am making a tiny web server, and I am having problems with HTTP POST
requests.

I made a button which sends a POST request to my server containing 3
characters (just to test), say "abc", using Javascript (XMLHttpRequest).
My server gets all the HTTP headers just fine, even the content length,
but when it gets to the actual content, after the finishing \r\n\r\n of
the headers, it simply gets stuck.

I got the HttpFox add-on for FireFox to see if FF is actually sending
everything, and it confirmed that something odd is going around here.

If I only read the headers, or even read nothing - FireFox sends
everything and I can respond from my server (of course, without knowing
the POST data...).
If I try to get past the headers - FireFox simply stops sending data,
which reflects why my Ruby server doesn't get it.

Is this somehow a bug in Ruby? Has anyone encountered something similar
and knows how to solve it?

Thanks.

 

[Alex Stahl]

[Note: parts of this message were removed to make it a legal post.]

Can't say I know a solution for you... but you might want to try using
Wireshark to check what's being sent. FF dev tools, in my experience,
tend to be somewhat buggy and don't always accurately report what's
going on. Wireshark will give you a much more exacting view as to
what's in the content-body of the POST request.

Especially because this part sounds really odd and not at all reflective
of what's happening - "FireFox simply stops sending data" - FF doesn't
send a header, wait for a response, and then send the body, which is
what this implies. Lest you somehow configured it to use HEAD instead
of POST.


________________________________________________________________________

Alex Stahl | Sr. Quality Engineer | hi5 Networks, Inc. | (e-mail address removed)
|

 

[Cody Scharfe]

[Note: parts of this message were removed to make it a legal post.]

Unsubscribe
Cancel
No more emails!

 

[John W Higgins]

[Note: parts of this message were removed to make it a legal post.]

Good Morning

I am making a tiny web server, and I am having problems with HTTP POST
requests.

In continuing with my trend of stupid questions for the week. Why on earth
are you not simply using Rack or any of the 55,000 variants that use Rack as
the base?

John

 

[Chananya Freiman]

Alex Stahl wrote in post #966971:

Can't say I know a solution for you... but you might want to try using
Wireshark to check what's being sent. FF dev tools, in my experience,
tend to be somewhat buggy and don't always accurately report what's
going on. Wireshark will give you a much more exacting view as to
what's in the content-body of the POST request.

Especially because this part sounds really odd and not at all reflective
of what's happening - "FireFox simply stops sending data" - FF doesn't
send a header, wait for a response, and then send the body, which is
what this implies. Lest you somehow configured it to use HEAD instead
of POST.


________________________________________________________________________

Well, it seems like everything is ok by what Wireshark sees, yet it
doesn't work.

I thought that maybe Ruby doesn't like printing whatever data it
received, for some reason, but just iterating over gets the thread
stuck.

This is just weird...

 

[Jeremy Bopp]

Alex Stahl wrote in post #966971:

Well, it seems like everything is ok by what Wireshark sees, yet it
doesn't work.

I thought that maybe Ruby doesn't like printing whatever data it
received, for some reason, but just iterating over gets the thread
stuck.

This is just weird...

How are you iterating over the input? Are you using each or each_line?
This is just a complete guess, but the problem could be that your
iteration logic is waiting to see an end of line from the client.

-Jeremy

 

[Alex Stahl]

[Note: parts of this message were removed to make it a legal post.]

Hard to say then what else might be the problem... are you using a
framework or writing "a tiny web server" from the ground up? If it's
the latter, you're definitely reinventing the wheel and, IMHO, on your
own, since that problem's already been solved in perfectly adequate
ways. (If the former, look for forums specific to that framework).

If you don't need to write your own server, but instead just need to
serve content/handle requests, why not use Sinatra?
(http://www.sinatrarb.com/). If you must write your own server, then
perhaps run a packet sniffer on the server-side to see what it's
receiving, then work your way up the stack to see what ruby actually
receives.



________________________________________________________________________

Alex Stahl | Sr. Quality Engineer | hi5 Networks, Inc. | (e-mail address removed)
|

 

[Chananya Freiman]

Damn, that sounded promising.

Tried each, each_lines, lines.each, each_byte.
Neither worked

 

[Chananya Freiman]

No, wait, you were right!
I tried readchar and it worked!

Thanks a lot, I've been stuck on this for two days

 

[Jeremy Bopp]

No, wait, you were right!
I tried readchar and it worked!

Thanks a lot, I've been stuck on this for two days

Great to hear. For the record, the other methods you tried will all run
until the end of the stream is seen, even each_byte. FireFox is
probably not closing the stream it opens for the POST message after
sending the payload, in case you want use it to make another request
most likely. Beware that readchar will also block if you try to use it
to read when there is no more data but the stream is still open.

In any case, you really should try out one of the existing web
frameworks unless your goal is actually to learn how to build your own
HTTP server. ;-)

-Jeremy

 

[Chananya Freiman]

After more debugging it looks like Ruby gets stuck when it tries to add
the very last character, AFTER the POST data (why is there a character
there anyway?) to a string.
So, in this example the whole packet is 559 characters long.
This code prints it just fine:

(1..560).each { |a| print peer.readchar }

Yet this one gets stuck (inside the loop, it doesn't even get to the
print):

str = ""
(1..560).each { |a| str += peer.readchar }
print str

Can't figure why

 

[Nathan Clark]

After more debugging it looks like Ruby gets stuck when it tries to add
the very last character, AFTER the POST data (why is there a character
there anyway?) to a string.
So, in this example the whole packet is 559 characters long.
This code prints it just fine:

(1..560).each { |a| print peer.readchar }

Put a print statement after this line, to make sure it isn't actually
blocking on that last character (since it's printing a character at a
time). If the socket is still open but has no more data to read,
readchar may be waiting for more data.

Usually clients will include a "Content-Length" header field,
specifying the length of the content body (in bytes). Since clients
will often leave the connection open rather than closing/reopening
whenever more communication is needed, you will probably need to check
the amount of data read against the value in this header to see if
you've reached the end of the request.

 

[Jeremy Bopp]

After more debugging it looks like Ruby gets stuck when it tries to add
the very last character, AFTER the POST data (why is there a character
there anyway?) to a string.
So, in this example the whole packet is 559 characters long.
This code prints it just fine:

(1..560).each { |a| print peer.readchar }

Yet this one gets stuck (inside the loop, it doesn't even get to the
print):

str = ""
(1..560).each { |a| str += peer.readchar }
print str

peer.readchar is blocking because there is no further data available but
the stream is still open.

Are you certain that your two examples both accurately reflect the code
you're actually using? In both cases you are creating an inclusive
range over which to iterate, and both ranges go from 1 to 560. As a
result, your loops will run 560 times, so if your data is only 559 bytes
long, you will run peer.readchar one more time than you have bytes
available. In that case, both loops should block at the peer.readchar
call in the 560th iteration.

-Jeremy

 

[Chananya Freiman]

Jeremy Bopp wrote in post #967039:

peer.readchar is blocking because there is no further data available but
the stream is still open.

Are you certain that your two examples both accurately reflect the code
you're actually using? In both cases you are creating an inclusive
range over which to iterate, and both ranges go from 1 to 560. As a
result, your loops will run 560 times, so if your data is only 559 bytes
long, you will run peer.readchar one more time than you have bytes
available. In that case, both loops should block at the peer.readchar
call in the 560th iteration.

-Jeremy

Yes that's what should happen, but the first one didn't block for some
reason.

Anyway, the problem is solved now.

I did try using readbytes(length) where length is the Content-Length
value, but it blocks wheres

(length.to_i).times do
data += peer.readchar
end

doesn't.

But, as long as I get the POST data, I am happy.

 

[Jeremy Bopp]

Jeremy Bopp wrote in post #967039:

Yes that's what should happen, but the first one didn't block for some
reason.

Anyway, the problem is solved now.

As long as there is a question about why there is a difference in
behavior in your code samples, there is a good chance that you've simply
masked a bug that will bite you later. You should probably flag this
code as needing attention at the very least unless this is a throwaway
project.

I did try using readbytes(length) where length is the Content-Length
value, but it blocks wheres

(length.to_i).times do
data += peer.readchar
end

doesn't.

Try using read(length) instead; although, readbytes(length) should work
the same in this case. What you're doing is highly inefficient since
you're creating a new string for every byte of data you read from your
POST. If you start POSTing nontrivial amounts of data to your server,
your solution will cause you problems.

-Jeremy

 

[Chananya Freiman]

That worked!
Is there a reason read would react different then readbytes? after all
this is all ASCII.

 

[Jeremy Bopp]

That worked!
Is there a reason read would react different then readbytes? after all
this is all ASCII.

Go take a look at the source for readbytes. You'll see that it actually
uses read under the covers. It's really just a convenience wrapper, so
if read works for you, readbytes should as well:

http://rdoc.info/docs/ruby-stdlib/1.8.7/IO#readbytes-instance_method

Try your tests again and take care to only change the lines you want to
test in the exact way you need. I've learned that if I see something
mysterious happening, the problem is usually my code rather than
anything else. ;-)

-Jeremy