Suggestions for reading binary data from a connected socket.

M

My Name

I am new to Java coming from Win32 C/CPP. After a lot of searching, I have
yet
to find an example explaining the best solution for reading and writing a
binary
message from a socket. All the devices I work with write raw memory
structures to the stream. They are usually proceeded with integer byte
lengths
to help determine message boundaries. In some cases, the message is
variable length with structures inside of structure etc.

1) I have never seen the output for a serialized object in Java.
If I was working with a fixed message length could I try to load
an object by creating a serializable one and loading into it from the stream
?
Or does serialized objects include overhead bytes/data preventing this ?

2) The only solution I came up with is to use class ObjectInputStream
and use the methods to extract each field type one after another and
save values into a respective object / storage.

This seems like a lot method calls() especially when lots of fields
aren't always needed.!
Should I use the buffered stream class in this scenario ?

3) Finally, it seems like I could read the entire message into a java Byte
array BUT
is there any way to convert parts of byte array into long, short,
floats, strings etc ?
This would be my first choice.

Thanks all,
 
T

Thomas Schodt

My said:
I could read the entire message into a java Byte
array BUT
is there any way to convert parts of byte array into long, short,
floats, strings etc ?
This would be my first choice.

java.nio.ByteBuffer
 
N

Nigel Wade

I am new to Java coming from Win32 C/CPP. After a lot of searching, I have
yet
to find an example explaining the best solution for reading and writing a
binary
message from a socket. All the devices I work with write raw memory
structures to the stream. They are usually proceeded with integer byte
lengths
to help determine message boundaries. In some cases, the message is
variable length with structures inside of structure etc.

1) I have never seen the output for a serialized object in Java.
If I was working with a fixed message length could I try to load
an object by creating a serializable one and loading into it from the stream

Almost impossible if it wasn't Java creating the serialized object.
?
Or does serialized objects include overhead bytes/data preventing this ?

Very much so.
2) The only solution I came up with is to use class ObjectInputStream
and use the methods to extract each field type one after another and
save values into a respective object / storage.

Are you sure you mean ObjectInputStream and not DataInputStream?
ObjectInputStream implies you are reading serialized objects rather than a
byte stream.
This seems like a lot method calls() especially when lots of fields
aren't always needed.!

Unfortunately, that's the way it has to be done.
Should I use the buffered stream class in this scenario ?

I don't think buffering would help much.
3) Finally, it seems like I could read the entire message into a java
Byte array BUT
is there any way to convert parts of byte array into long, short,
floats, strings etc ?
This would be my first choice.


You can certainly do this. Create a DataInputStream, and read the first
integer containing the byte length. Create a byte[] array and use
readFully() method of the DataInputStream to read the bytes of the message.

One think you need to be aware of in any network protocol is the
endian-ness of the data coming into your program. Java always works with
big-endian data, but your data may be little-endian. It's up to you to
provide the conversion somewhere.

The method I use to do this is to create a ByteBuffer from the data I've
read in and then set the endian-ness of the ByteBuffer. You can then read
data from the ByteBuffer with it's methods getShort() getInt() etc.
 
S

Steve Horsley

My said:
I am new to Java coming from Win32 C/CPP. After a lot of searching, I have
yet
to find an example explaining the best solution for reading and writing a
binary
message from a socket. All the devices I work with write raw memory
structures to the stream. They are usually proceeded with integer byte
lengths
to help determine message boundaries. In some cases, the message is
variable length with structures inside of structure etc.
Then everything you have used until now is broken. Writing memory images
over a socket is inherently non-portable since you do not necessarily
know the byte order or the alignment padding policy that the compiler uses.

If you are trying to talk to one of them with java then you first need to
find out exactly what the C/CPP program sends on the wire. This may require
a protocol analyser to figure out what padding has been inserted into the
structures, and what byte ordering is used.

Note that if you ever read an RFC, it always specifies the bytes on the wire,
not how information will be stored in memory.
1) I have never seen the output for a serialized object in Java.
If I was working with a fixed message length could I try to load
an object by creating a serializable one and loading into it from the stream
?
Or does serialized objects include overhead bytes/data preventing this ?
Java serialisation is clearly specified, but it's really only suitable
for java to java communication. I would not recommend it for java to C
for instance.
2) The only solution I came up with is to use class ObjectInputStream
and use the methods to extract each field type one after another and
save values into a respective object / storage.

This seems like a lot method calls() especially when lots of fields
aren't always needed.!
Should I use the buffered stream class in this scenario ?
Not sure what you mean here. ObjectInputStream is for serialized Objects.
If you're serializing objects, why not just write the whole object - don't
bother with doing the individual fields yourself.

Using buffering will gnerally help performance, but don't forget to flush()
at the end of sending. Particularly, use a BufferedInputStream round your
InputStream to prevent a low-level I/O call for every byte you read.
3) Finally, it seems like I could read the entire message into a java Byte
array BUT
is there any way to convert parts of byte array into long, short,
floats, strings etc ?
This would be my first choice.

There is a DataInputStream that has readLong() etc, but it assumes big-endian
byte ordering on the wire, which may not be what a C/CPP prog sends. I believe
that Roedy Green has an LEDataInputStream class that uses little-endian
ordering on his web site (www.mindprod.com).

Beyond (LE)DataInutStream, you are looking at doing your own decoding, whether
you choose to read piece by piece or grab one bug byte[] and decompose it later.

I would suggest that your classes shold have a constructor that takes an
InputStream as an argument, so you can confine the encoding details to the
class concerned.

HTH
Steve
 
X

xarax

Thomas Schodt said:
java.nio.ByteBuffer

In addition to everything else said on this topic, I'll
weigh in with my 2 cents. I use the ByteBuffer approach
with a program generator that has a GUI front-end. I
specify the "message packet" and its field names and types
(byte, boolean, short, int, long). The program generator
spits out Java source code and C source code that can
manage the various message packets. The code has Java
methods and C functions for getting/setting the various
fields by name for their data type.

Thus, it's very simple to read/write TCP/IP message
packets containing binary data, and to communicate
between the Java and C applications. The program
generator generates the source code for inserting
the bytes into the ByteBuffer and later extracting
bytes from the ByteBuffer (in the appropriate data
type for the field, like byte, short, long).

Find an automated way of handling the data conversion
and layout of a ByteBuffer. If you only need a single
message type and two int data fields, then you could
probably hand-write the code. Otherwise, as your
application grows to require more message packet types
and more data fields in each message, you'll quickly
exhaust all of your time making the code changes by hand.
 
M

My Name

Thomas Schodt said:
java.nio.ByteBuffer

Thanks, that answers one big issue! Question is would it be faster to use this
or the stream f() to build and extract ?
 
M

My Name

Thanks to everyone for the suggestions.

I should have stated the following.
1) I know the binary format of the messages, coming and going.
2) I am aware of endian issues and where it is of concern.

Does anyone know which of these methods would be prefered.

1) Use something like DataInputStream build a class which
takes the stream and reads through the message each field
and filling members of the class as it goes. i.e.
x = obj.readInt(socket)
y = obj.readShotr(socket)
....

2) Read and extract length, then readFull message into a ByteBuffer
then use ByteBuffer methods similar to above to extract
each field ?

Best wishes,
 
R

Roedy Green

I am new to Java coming from Win32 C/CPP. After a lot of searching, I have
yet
to find an example explaining the best solution for reading and writing a
binary
message from a socket.

If you are starting from scratch and need to communicate with C, you
might use big-endian DataOutputStreams. If you are working with
legacy, you might use LEDataStream to read the little-endian data.

On the Java end, there is also nio which can handle both big and
little endian formats.

see http://mindprod.com/jgloss/endian.html
 
N

Nigel Wade

Thanks to everyone for the suggestions.

I should have stated the following.
1) I know the binary format of the messages, coming and going.
2) I am aware of endian issues and where it is of concern.

Does anyone know which of these methods would be prefered.

1) Use something like DataInputStream build a class which
takes the stream and reads through the message each field
and filling members of the class as it goes. i.e.
x = obj.readInt(socket)
y = obj.readShotr(socket)
....

2) Read and extract length, then readFull message into a ByteBuffer
then use ByteBuffer methods similar to above to extract each field ?

1) is slightly simpler, but will only work if the data coming over
the network is in network byte order (big-endian).

2) will work for either big- or little-endian.
 
R

Roedy Green

1) is slightly simpler, but will only work if the data coming over
the network is in network byte order (big-endian).

If you have littleendian data, you can use LEDataStream. What you
can't easily do is flip back and forth between big and little endian.

That would be easier with a byte buffer approach.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,479
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top