Another TCPSocket Question

[Greg Chambers]

Sorry to bother you guys about this again. So my main problem right now
is probably due to my lack of knowledge of the TCPSocket, even with the
documentation. I mean, all the methods I have read about either require
that I know the length I am reading ahead of time or that the only thing
the socket needs to be doing is reading. I need to be able to just get
all the data there is out of the socket, display it for the user, and
then accept a string from the user to send back to the server. The main
problem is the size of the strings sent back and forth are dependent on
the commands being sent by the user. If the user types in "help", then
the user will get back the whole help text file while if the user types
in "connections", the user will just get an integer of connections there
are.

So far, the best read method I have found for this is #readpartial but
there seems to be a catch. So I do the read right after the command is
sent. The problem here is often the data doesn't have enough time to
get returned before #readpartial determines there is no data to be read
and then prompts the user for the next command while showing no data.
However, after that command is sent, it then successfully shows the data
from two commands ago. Does anyone have any idea as how to fix my use
of #readpartial or maybe another method that will suit my needs?

 

[Gary Wright]

Does anyone have any idea as how to fix my use
of #readpartial or maybe another method that will suit my needs?

I suspect that you are thinking about this problem
as if TCP keeps track of message boundaries, which
is not the case.

If one side of a TCP connection sends 20 bytes of
data via a single call to write, there is no way
to ensure that the other side reads the same exact
20 bytes of data via a single read. The reading
side may see one read of 20 bytes or perhaps 2
reads of 10 bytes or even 20 reads of a byte each.
In theory there can be arbitrarily long delays
between the successful reads. TCP connections
are streams of bytes with no message boundaries.

In practice, this means that you need to impose
your own record boundaries on the data and track
them yourself. The easiest example would be to
insert newlines while sending and look for them
while receiving. Another common approach is to
have a header that indicates the length of the
record so that the reading size knows how many
bytes to read.

There is no right or wrong answer here it just
depends on how you design your protocol on top
of the TCP byte stream.

If your data has newlines ending each message
(or even some other message separator)
you should be able to use gets or readline. If
the message size is built-in to the protocol then
you should be read the correct number of bytes
based on your protocol definition (or header).

If there is no way to determine the end of the
message other then some sort of ambiguous timing
constraint (data hasn't arrived for a couple of
seconds) then I think you'll need to rethink
your protocol.

Gary Wright

 

[Greg Chambers]

Gary Wright

Okay, not too bad. Wait, if I send the fixnum ahead of time, is the
number of bytes I need to receive dependent on how long the message is
still? Or is there a way to send an integer always using 4 bytes or
something like that?

 

[Gary Wright]

Okay, not too bad. Wait, if I send the fixnum ahead of time, is the
number of bytes I need to receive dependent on how long the message is
still? Or is there a way to send an integer always using 4 bytes or
something like that?

Look at Array#pack and String#unpack if you want to send binary
numeric values (i.e., 8-bit or 32-bit integers).

I'd recommend going the text/newline approach if at all possible,
otherwise you are dealing with a binary protocol which can be more
difficult to work with.

Even better would be to used a pre-existing protocol instead of
re-inventing the wheel. I'd have to know more about your situation
though to recommend anything in particular.

Gary Wright

 

[Eleanor McHugh]

Okay, not too bad. Wait, if I send the fixnum ahead of time, is the
number of bytes I need to receive dependent on how long the message is
still? Or is there a way to send an integer always using 4 bytes or
something like that?

Post us some code so that we can see what you're doing and make
relevant suggestions.

Also take a look at some of the presentations linked to in my sig, in
particular the Semantic DNS and Shoes presentations, as there are
simple examples of writing UDP and and TCP client-server systems in
both. The code will designed to be instructive rather than
prescriptive though so you'll have to play with it if you need to use
it in a production environment.


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

 

[Greg Chambers]

Post us some code so that we can see what you're doing and make
relevant suggestions.

Note: I acknowledge some code was copied and then modified. The server
script was kind of made last second just to test this out.

# server.rb
#
require 'socket'
require 'bin_protocol'

dts = TCPServer.new('localhost', 20000 )
puts "Entering loop"
loop do
Thread.start(dts.accept) do |s|
puts "In loop!"
sock = BinProtocol.new( s )
print(s, " is accepted\n")
sock.send_data("Welcome to the Swat Server!")
print(s, "says '#{sock.get_data}")
sock.send_data("Can I get some help?")
print sock.get_data
sock.send_data("Goodbye!")
print(s, " is gone\n")
s.close
end
end

# client.rb
#
require 'socket'
require 'bin_protocol'

s = TCPSocket.new( 'localhost', 20000 )
sock = BinProtocol.new( s )
print sock.get_data
sock.send_data("Why thank you Server!")
print sock.get_data
sock.send_data( File.new("help.txt", "r").readlines.to_s )
print sock.get_data
s.close

# bin_protocol.rb
#
require 'socket'
require 'bit-struct'

class BinProtocol

class SizeStruct < BitStruct
unsigned :message_length, 4*8, "Message Length", :endian => :network
end # SizeStruct

class DataStruct < BitStruct
rest :message, "String containing data"
end

def initialize( tcp_sock )
@tcp_sock = tcp_sock
end

def send_data( data_str )
data_struct = DataStruct.new
size_struct = SizeStruct.new
data_struct.message = data_str
size_struct.message_length = DataStruct.round_byte_length
@tcp_sock.write( size_struct.to_s )
@tcp_sock.write( data_struct.to_s )
end

def get_data
size_struct = SizeStruct.new( @tcp_sock.recv(4).to_s )
data_struct = DataStruct.new( @tcp_sock.recv(
size_struct.message_length ) )
return data_struct.to_s
end

end

and help.txt is just a simple, multiple line, text file. Just note that
I am using the Bit-Struct gem and I am getting a NoMemoryError from
get_data when my computer says I should have plenty of memory to run
this. Also I took the binary route because another part of the program
not shown here is required to deal with binary networking streams
anyways, so I figured I could just apply one to the other. So... help?

 

[Joel VanderWerf]

def send_data( data_str )
data_struct = DataStruct.new
size_struct = SizeStruct.new
data_struct.message = data_str
size_struct.message_length = DataStruct.round_byte_length

size_struct.message_length = data_struct.length

@tcp_sock.write( size_struct.to_s )
@tcp_sock.write( data_struct.to_s )
end

That was the main problem. You were getting the minimum length of any
instance of DataStruct, which is 0, instead of the length of the packet
you had assembled.

A minor point: you generally don't need all those to_s calls, since
BitStruct < String. I would replace

return data_struct.to_s

with

return data_struct.message

even though it's the same in this case (just in case you add a field
before message).

Otherwise, this is pretty much the right approach to managing discrete
messages using a length field. Your code works for me with a few edits.

 

[Greg Chambers]

size_struct.message_length = data_struct.length

In the words of Homer Simpson, "D'Oh!" Thanks for all your help guys!

 

[Joel VanderWerf]

Otherwise, this is pretty much the right approach to managing discrete
messages using a length field. Your code works for me with a few edits.

My own impl.:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/177704

(doesn't even depend on BitStruct, and includes a C version and tests!)

 

[Brian Candler]

Look at Array#pack and String#unpack if you want to send binary
numeric values (i.e., 8-bit or 32-bit integers).

I'd recommend going the text/newline approach if at all possible,
otherwise you are dealing with a binary protocol which can be more
difficult to work with.

Such as: http://en.wikipedia.org/wiki/Netstrings