Reading binary socket data and copying to another socket ?

[swebb99]

Hi,

I need to write a java process that listens on a socket for a client
sending binary data (actually compressed messages). This socket will
always be open.

This data will then be written to another socket the other end of
which understands the content of the data. The problem I have is that
the data can be sent at any time and there is nothing in the data (no
characters) I can use to indicate the end of a raw message. So how can
I read the original data and decide that it should now be copied to
the out bound socket ?

If I call read(buff[]) then that blocks until the required number of
bytes is received which doesn't help me as I don't know how many bytes
there are, it could be 1 it could be 1000+. I noticed the socket class
has a setSOTimeout() method which might be useful but could I rely on
a timeout for detecting the end of the binary data associated with a
message !

Any idea's how I can cope with this ?

Thanks for any help

Steve

 

[Gordon Beaton]

I need to write a java process that listens on a socket for a client
sending binary data (actually compressed messages). This socket will
always be open.

This data will then be written to another socket the other end of
which understands the content of the data. The problem I have is
that the data can be sent at any time and there is nothing in the
data (no characters) I can use to indicate the end of a raw message.
So how can I read the original data and decide that it should now be
copied to the out bound socket ?

If there is truly *nothing* in the data that tells you whether it
should be copied to the outbound socket, then nobody here can help
you. But from your description here is sounds like *all* data should
be copied to the outbound socket. Is that the case?

If it's important for you to handle "messages" individually, but the
protocol does not provide information about message boundaries, then
you are out of luck. Complain to the protocol designer, and mention
that TCP has no concept of message boundaries.

If I call read(buff[]) then that blocks until the required number of
bytes is received which doesn't help me as I don't know how many
bytes there are, it could be 1 it could be 1000+. I noticed the
socket class has a setSOTimeout() method which might be useful but
could I rely on a timeout for detecting the end of the binary data
associated with a message !

You cannot rely on timing as a way to separate messages.

If you call read(buf) you will only block until *some* data arrives
(often less than the requested amount), or until the socket is closed
at the other end, causing EOF at your end.

If you don't want to block at all, then you should have a look at
java.nio.channels.Selector for non-blocking mechanisms. A Selector can
tell you when the socket can be read without blocking.

/gordon

--

 

[Owen Jacobson]

Hi,

I need to write a java process that listens on a socket for a client
sending binary data (actually compressed messages). This socket will
always be open.

This data will then be written to another socket the other end of
which understands the content of the data. The problem I have is that
the data can be sent at any time and there is nothing in the data (no
characters) I can use to indicate the end of a raw message. So how can
I read the original data and decide that it should now be copied to
the out bound socket ?

If I call read(buff[]) then that blocks until the required number of
bytes is received which doesn't help me as I don't know how many bytes
there are, it could be 1 it could be 1000+. I noticed the socket class
has a setSOTimeout() method which might be useful but could I rely on
a timeout for detecting the end of the binary data associated with a
message !

Any idea's how I can cope with this ?

It sounds like you have an app that proxies communication between two
endpoints but does nothing with the communication directly. If that's
the case, then the intermediate app doesn't need to care about message
boundaries: it just needs to copy every byte from the input stream to
the output stream. This will not fragment or damage messages any more
than TCP itself does, and will be transparent to correctly-written
endpoints (that make no assumptions about message boundaries with
respect to read calls).

At a high level, then, all you need is this:

prepare a buffer
while there is data:
read input into the buffer
write buffer to output

Hope that helps,
Owen

 

[swebb99]

Hi,

I need to write a java process that listens on a socket for a client
sending binary data (actually compressed messages). This socket will
always be open.

This data will then be written to another socket the other end of
which understands the content of the data. The problem I have is that
the data can be sent at any time and there is nothing in the data (no
characters) I can use to indicate the end of a raw message. So how can
I read the original data and decide that it should now be copied to
the out bound socket ?

If I call read(buff[]) then that blocks until the required number of
bytes is received which doesn't help me as I don't know how many bytes
there are, it could be 1 it could be 1000+. I noticed the socket class
has a setSOTimeout() method which might be useful but could I rely on
a timeout for detecting the end of the binary data associated with a
message !

Any idea's how I can cope with this ?

It sounds like you have an app that proxies communication between two
endpoints but does nothing with the communication directly. If that's
the case, then the intermediate app doesn't need to care about message
boundaries: it just needs to copy every byte from the input stream to
the output stream. This will not fragment or damage messages any more
than TCP itself does, and will be transparent to correctly-written
endpoints (that make no assumptions about message boundaries with
respect to read calls).

At a high level, then, all you need is this:

prepare a buffer
while there is data:
read input into the buffer
write buffer to output

Hope that helps,
Owen

Yes thats exactly my problem Basically all I am doing is taking a
binary stream of data that is actually meaningful messages to other
clients/servers and wrapping the binary in another protocol for one
hop of the connection so it can get through a firewall in a known
protocol. Then once through I have another process that strips the
binary back out and sends it on to the final destination. So basically
I don't know the contents of the binary data. I just know that I have
to read it, wrap it, unwrap it again and then pass it on again as
binary. The socket from the source of the message to my first process
should stay open all the time. The socket however from my 2nd process
to the destination is supposed to disconnect after 60secs of idle (may
use setSOTimeout for this).

I thought that if I used read(data[x]) the read would block until the
socket goes down or x bytes were received, is this not the case ? If
it is how cn I write the while loop without potentially blocking on
the last bit of data ?

Thanks for the help

Steve

 

[Roedy Green]

I can use to indicate the end of a raw message.

either you put a count on a message, or you reserve some end marker
that can't exist naturally in the message.

One scheme for short messages reserves the byte 0 as an end marker.
Any accidental 0 is replaced by the offset to the next accidental 0.
The byte you reserve is not actually 0, but a rarely used one.

With lots of RAM a buffering up the message before you send it makes
the count method easiest.

 

[Esmond Pitt]

I thought that if I used read(data[x]) the read would block until the
socket goes down or x bytes were received, is this not the case ?

No. It blocks until at least 1 byte is received.

 

[Gordon Beaton]

Basically all I am doing is taking a binary stream of data that is
actually meaningful messages to other clients/servers and wrapping
the binary in another protocol for one hop of the connection so it
can get through a firewall in a known protocol. Then once through I
have another process that strips the binary back out and sends it on
to the final destination.

If you can use SSH through your firewall, then this can be solved
without writing any code, using the port redirection feature (of
OpenSSL for example).

/gordon

--

 

[swebb99]

If you can use SSH through your firewall, then this can be solved
without writing any code, using the port redirection feature (of
OpenSSL for example).

/gordon

--

Thanks for that but the stuff I'm working on has its own protocol that
is allow through the firewall (via various registration routines) so
the data can only be packaged in the payload of messages within this
protocol.

I'm working on the code at the moment and will be testing in the next
few days so will report back when I get somewhere with it.