read binary data from C file???

J

John Adams

Hi,

I am sorry if this has been asked before but I couldn't find any hint
scheming through the list. So here is my problem:

I have to write a client end in java reading data sent over UDP from
a server (the server is written in C). The data format is a C struct
with all the fields including long, int, float, etc. I got the data in
the java client end in the format of a DatagramPacket, then I convert it
to a byte array. Now the question is, how do I extract each field from
the byte array? I know this is silly but I have tried using classes like
ByteBuffer, Array, etc, without good result (The print out of the
recieiving end differs than that from the sending end).

Thanks for all the advice.
 
M

Michael Borgwardt

John said:
I am sorry if this has been asked before but I couldn't find any hint
scheming through the list. So here is my problem:

I have to write a client end in java reading data sent over UDP from a
server (the server is written in C). The data format is a C struct with
all the fields including long, int, float, etc. I got the data in the
java client end in the format of a DatagramPacket, then I convert it to
a byte array. Now the question is, how do I extract each field from the
byte array? I know this is silly but I have tried using classes like
ByteBuffer, Array, etc, without good result (The print out of the
recieiving end differs than that from the sending end).

What do you expect? That we can see your code and find the bugs in it by
telepathy?

Show us what you tried, then maybe we can tell you what seems fishy.

The most obvious thing that can go wrong is endianness and alignment
in the C struct.

The most obvious way to debug it is to put some easily identifiable
values into the struct on the server side and then dump the entire
byte array on the client to see how your data actually arrives.
 
W

Will Hartung

John Adams said:
Hi,

I am sorry if this has been asked before but I couldn't find any hint
scheming through the list. So here is my problem:

I have to write a client end in java reading data sent over UDP from
a server (the server is written in C). The data format is a C struct
with all the fields including long, int, float, etc. I got the data in
the java client end in the format of a DatagramPacket, then I convert it
to a byte array. Now the question is, how do I extract each field from
the byte array? I know this is silly but I have tried using classes like
ByteBuffer, Array, etc, without good result (The print out of the
recieiving end differs than that from the sending end).

Thanks for all the advice.

Take a look at java.io.DataInputStream. It may do what you want, but you may
have issues with endian-ness.

Regards,

Will Hartung
([email protected])
 
B

Boudewijn Dijkstra

John Adams said:
Hi,

I am sorry if this has been asked before but I couldn't find any hint
scheming through the list. So here is my problem:

I have to write a client end in java reading data sent over UDP from
a server (the server is written in C). The data format is a C struct
with all the fields including long, int, float, etc. I got the data in
the java client end in the format of a DatagramPacket, then I convert it
to a byte array. Now the question is, how do I extract each field from
the byte array? I know this is silly but I have tried using classes like
ByteBuffer, Array, etc, without good result (The print out of the
recieiving end differs than that from the sending end).

Experiment with a few struct values to find out where each byte ends up in the
array. Then use << and | to fill ints and longs. If you want floats and
doubles, use Float.intBitsToFloat() and Double.longBitsToDouble().
 
G

Gordon Beaton

I have to write a client end in java reading data sent over UDP
from a server (the server is written in C). The data format is a C
struct with all the fields including long, int, float, etc. I got
the data in the java client end in the format of a DatagramPacket,
then I convert it to a byte array. Now the question is, how do I
extract each field from the byte array?

Others have already pointed out how you can read the data, however
it's worth mentioning that if the server gets recompiled with a
different compiler, or moved to a different platform (with no changes
to the data types) it could break your client.

The real solution is to change the server so that it doesn't write the
data in a format that is quite so compiler and platform dependent.

Writing raw binary data directly from a C struct is lazy, fragile and
non-portable. And as you've already discovered, it makes it difficult
and error prone to write a client in a different language, that
doesn't use the same compiler, or that will run on a different
hardware platform.

/gordon
 
J

John Adams

Gordon said:
Others have already pointed out how you can read the data, however
it's worth mentioning that if the server gets recompiled with a
different compiler, or moved to a different platform (with no changes
to the data types) it could break your client.

The real solution is to change the server so that it doesn't write the
data in a format that is quite so compiler and platform dependent.

Writing raw binary data directly from a C struct is lazy, fragile and
non-portable. And as you've already discovered, it makes it difficult
and error prone to write a client in a different language, that
doesn't use the same compiler, or that will run on a different
hardware platform.

/gordon


Hi,

Thank you all for your invaluable advice. To date, I have seemed to
try all the methods mentioned: Array, ByteBuffer, DataInputStream,
ByteArrayInputStream, intBitsToFloat(), etc. I had, ofcourse as Michael
Borgwardt so "brilliantly" pointed out, tried putting some values on the
server side and dumping them out on the client side (That is HOW I know
they are not the same!!!).

Flatform and hardware also are not the concern since I had tried
running the server on the very same intel machine as the client (just to
debug). Endianess is also out of question in this scenario.

This is how I do it:

// The data format from a C server.
// To make thing simple, I make all fiels float
typede struct {
float something;
float some_other_thing; ...
}

// class to convert data on the java end
public class StructureInputStream extends ByteArrayInputStream
{
static int curr = 0;
private DataInputStream dis = null;

public StructureInputStream(byte[] b)
{
super(b);
curr = 1;
dis = new DataInputStream((InputStream)this);
}

public final long readLong() throws IOException, EOFException
{
return dis.readLong();
}

public final double readDouble() throws IOException, EOFException
{
return dis.readDouble();
}

public final int readInt() throws IOException, EOFException
{
return dis.readInt();
}

public final int readUnsignedShort() throws IOException, EOFException
{
return dis.readUnsignedShort();
}

public final float readFloat() throws IOException, EOFException
{
return dis.readFloat();
}

public final short readShort() throws IOException, EOFException
{
return dis.readShort();
}

public final char readChar() throws IOException, EOFException
{
return (char)dis.readByte();
}

public void skipBytes(int n) throws IOException
{
int num_bytes_to_skip = n;
int x = 0;

do {
x = dis.skipBytes(num_bytes_to_skip);
num_bytes_to_skip -= x;
} while (num_bytes_to_skip>0);
}
}


// And how the UDP client gets data
// try - catch blocks are omitted for clarity
DatagramPacket pkt = new DatagramPacket(buffer, buffer.length);
socket.receive(pkt);
byte[] data = pkt.getData();
StructureInputStream t = new StructureInputStream(data);
float a1 = t.readFloat(); t.skipBytes(4);
float a2 = t.readFloat(); t.skipBytes(4);
float a3 = t.readFloat(); t.skipBytes(4);
float a4 = t.readFloat(); t.skipBytes(4);

// print out ...


Again, thank you all for your advice.
Regards,
 
P

Paul Lutus

John Adams wrote:

/ ...
Thank you all for your invaluable advice. To date, I have seemed to
try all the methods mentioned: Array, ByteBuffer, DataInputStream,
ByteArrayInputStream, intBitsToFloat(), etc. I had, ofcourse as Michael
Borgwardt so "brilliantly" pointed out, tried putting some values on the
server side and dumping them out on the client side (That is HOW I know
they are not the same!!!).

Okay, to restate a point made by someone else, why are you trying to
transfer the data in binary form? Why not make the server emit the data in
plain-text form? This is a much more robust solution, one that is immune to
nearly all the issues that are occupying your thinking right now.

In other words, you may spend months debugging this binary transfer
solution, only to see your work entirely unraveled by a revision in a
future compiler, or you can use plain text and be done with it.
 
W

Will Hartung

Paul Lutus said:
Okay, to restate a point made by someone else, why are you trying to
transfer the data in binary form? Why not make the server emit the data in
plain-text form? This is a much more robust solution, one that is immune to
nearly all the issues that are occupying your thinking right now.

In other words, you may spend months debugging this binary transfer
solution, only to see your work entirely unraveled by a revision in a
future compiler, or you can use plain text and be done with it.

The choice of format may well be out of the hands of the guy with the
problem. Could be a legacy or 3rd party application.

The fact that there may be other, more appropriate ways to transfer data
doesn't necessarily lessen the need for someone to do exactly what he's
asking.

Regards,

Will Hartung
([email protected])
 
P

Paul Lutus

Will Hartung wrote:

/ ...
The choice of format may well be out of the hands of the guy with the
problem. Could be a legacy or 3rd party application.

The fact that there may be other, more appropriate ways to transfer data
doesn't necessarily lessen the need for someone to do exactly what he's
asking.

Yes, but we don't know that, and the OP may not know the advantages of plain
text. Therefore someone needed to mention it.
 
?

=?ISO-8859-1?Q?Daniel_Sj=F6blom?=

John said:
Thank you all for your invaluable advice. To date, I have seemed to
try all the methods mentioned: Array, ByteBuffer, DataInputStream,
ByteArrayInputStream, intBitsToFloat(), etc. I had, ofcourse as Michael
Borgwardt so "brilliantly" pointed out, tried putting some values on the
server side and dumping them out on the client side (That is HOW I know
they are not the same!!!).

Flatform and hardware also are not the concern since I had tried
running the server on the very same intel machine as the client (just to
debug). Endianess is also out of question in this scenario.

Sigh. Have you even read the documentation for DataInputStream? It reads
big endian data, while your x86 is little endian. The other problem is
that your structs probably contain random padding bytes.

Your problem is that you use C structs as a communication format. This
is inherently unportable (since it is in no way defined what a struct
looks like in memory), even on the same platform if you happen to
compile with a different compiler or even another version of the same
compiler. You will need to actually design and specify the transmission
format. XML or binary data with endianess/padding issues removed, do
whatever you wish, but do not dump raw C structs.
 
S

Steve Horsley

John said:
Gordon Beaton wrote:

// And how the UDP client gets data
// try - catch blocks are omitted for clarity
DatagramPacket pkt = new DatagramPacket(buffer, buffer.length);
socket.receive(pkt);
byte[] data = pkt.getData();
StructureInputStream t = new StructureInputStream(data);
float a1 = t.readFloat(); t.skipBytes(4);
float a2 = t.readFloat(); t.skipBytes(4);
float a3 = t.readFloat(); t.skipBytes(4);
float a4 = t.readFloat(); t.skipBytes(4);

So you are ignoring the endian-ness issue completely, although you
seem to be addressing the possible alignment issue (skipping 4
bytes after every float).

Look in your compiler documentation and see exactly how it stores
a float. Compare this with the java documentation on how a
DataInputStream reads a float. I bet they're different. Endian-ness
may possibly be the only difference.

Only you know how the compiler stores floats in memory. Only you
can work out how to decode that back into float values.

If the server is outside your control then you have no option
but to ask the author exactly what format it sends on the network,
or work it out yourself.

If the server is in your control then it's your responsibility
to document what the server sends. And change the format to
something you CAN document (parhaps text as Paul suggests) if
you cannot work out what your current implementation sends.

Either way, you cannot work on decoding the data until you know
how it's been encoded.

Steve
 
J

John Adams

Steve said:
John said:
Gordon Beaton wrote:

// And how the UDP client gets data
// try - catch blocks are omitted for clarity
DatagramPacket pkt = new DatagramPacket(buffer, buffer.length);
socket.receive(pkt);
byte[] data = pkt.getData();
StructureInputStream t = new StructureInputStream(data);
float a1 = t.readFloat(); t.skipBytes(4);
float a2 = t.readFloat(); t.skipBytes(4);
float a3 = t.readFloat(); t.skipBytes(4);
float a4 = t.readFloat(); t.skipBytes(4);

So you are ignoring the endian-ness issue completely, although you
seem to be addressing the possible alignment issue (skipping 4
bytes after every float).
Look in your compiler documentation and see exactly how it stores a
float. Compare this with the java documentation on how a DataInputStream
reads a float. I bet they're different. Endian-ness
may possibly be the only difference.
Only you know how the compiler stores floats in memory. Only you
can work out how to decode that back into float values.

If the server is outside your control then you have no option
but to ask the author exactly what format it sends on the network,
or work it out yourself.

If the server is in your control then it's your responsibility
to document what the server sends. And change the format to something
you CAN document (parhaps text as Paul suggests) if
you cannot work out what your current implementation sends.

Either way, you cannot work on decoding the data until you know
how it's been encoded.

Steve

Hi all,

I am deeply grateful for all your advice. Thanks to you all I have
found the problem; and it is actually a legacy problem. The code has
worked all along (I mean the Java end). The reason we saw a discrepancy
in value display is that the C code did not print out the correct value!
You can rest assure that if I could beat the guy who wrote that I would
already have done so! Anyway, since I have no control over that, all I
can do is just smile and be thankful that I have hit the right newsgroup
to ask for help.

It is really nice of you to pour in such a wonderful help.

Regards,
 
M

Michael Borgwardt

John said:
I had, ofcourse as Michael
Borgwardt so "brilliantly" pointed out, tried putting some values on the
server side and dumping them out on the client side (That is HOW I know
they are not the same!!!).

You obviously didn't understand my suggestion. My point was to NOT try
to extract the individual values, but instead to dump the entire byte
array to see how the values (which should be chosen to be easily
recognizable even if scrambled a bit by endian issues) end up in the
output. That can give you far more information than "it's not the same".
 
G

Gordon Beaton

Just to clarify, I didn't write *any* of the text (a short section of
code) that you quoted in your reply to John.

/gordon
 
J

John Adams

John said:
Hi,

I am sorry if this has been asked before but I couldn't find any hint
scheming through the list. So here is my problem:

I have to write a client end in java reading data sent over UDP from a
server (the server is written in C). The data format is a C struct with
all the fields including long, int, float, etc. I got the data in the
java client end in the format of a DatagramPacket, then I convert it to
a byte array. Now the question is, how do I extract each field from the
byte array? I know this is silly but I have tried using classes like
ByteBuffer, Array, etc, without good result (The print out of the
recieiving end differs than that from the sending end).

Thanks for all the advice.


Hi Guys,

Just to let you know in case you missed my post, I have found the
problem thanks to all your advice. If you'd like to know where the
problem came from, check my reply I mistakenly posted under Steve
Horsley's post.
 
S

Steve Horsley

Gordon said:
Just to clarify, I didn't write *any* of the text (a short section of
code) that you quoted in your reply to John.

/gordon

Whoops, sorry. Sloppy snipping.
I'll try to be more careful in future.

Steve
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,773
Messages
2,569,594
Members
45,114
Latest member
GlucoPremiumReview
Top