read binary data from C file???

Discussion in 'Java' started by John Adams, Sep 22, 2004.

  1. John Adams

    John Adams Guest

    Hi,

    I am sorry if this has been asked before but I couldn't find any hint
    scheming through the list. So here is my problem:

    I have to write a client end in java reading data sent over UDP from
    a server (the server is written in C). The data format is a C struct
    with all the fields including long, int, float, etc. I got the data in
    the java client end in the format of a DatagramPacket, then I convert it
    to a byte array. Now the question is, how do I extract each field from
    the byte array? I know this is silly but I have tried using classes like
    ByteBuffer, Array, etc, without good result (The print out of the
    recieiving end differs than that from the sending end).

    Thanks for all the advice.
    John Adams, Sep 22, 2004
    #1
    1. Advertising

  2. John Adams wrote:
    > I am sorry if this has been asked before but I couldn't find any hint
    > scheming through the list. So here is my problem:
    >
    > I have to write a client end in java reading data sent over UDP from a
    > server (the server is written in C). The data format is a C struct with
    > all the fields including long, int, float, etc. I got the data in the
    > java client end in the format of a DatagramPacket, then I convert it to
    > a byte array. Now the question is, how do I extract each field from the
    > byte array? I know this is silly but I have tried using classes like
    > ByteBuffer, Array, etc, without good result (The print out of the
    > recieiving end differs than that from the sending end).


    What do you expect? That we can see your code and find the bugs in it by
    telepathy?

    Show us what you tried, then maybe we can tell you what seems fishy.

    The most obvious thing that can go wrong is endianness and alignment
    in the C struct.

    The most obvious way to debug it is to put some easily identifiable
    values into the struct on the server side and then dump the entire
    byte array on the client to see how your data actually arrives.
    Michael Borgwardt, Sep 22, 2004
    #2
    1. Advertising

  3. John Adams

    Will Hartung Guest

    "John Adams" <> wrote in message
    news:415183b6$...
    > Hi,
    >
    > I am sorry if this has been asked before but I couldn't find any hint
    > scheming through the list. So here is my problem:
    >
    > I have to write a client end in java reading data sent over UDP from
    > a server (the server is written in C). The data format is a C struct
    > with all the fields including long, int, float, etc. I got the data in
    > the java client end in the format of a DatagramPacket, then I convert it
    > to a byte array. Now the question is, how do I extract each field from
    > the byte array? I know this is silly but I have tried using classes like
    > ByteBuffer, Array, etc, without good result (The print out of the
    > recieiving end differs than that from the sending end).
    >
    > Thanks for all the advice.


    Take a look at java.io.DataInputStream. It may do what you want, but you may
    have issues with endian-ness.

    Regards,

    Will Hartung
    ()
    Will Hartung, Sep 22, 2004
    #3
  4. "John Adams" <> schreef in bericht
    news:415183b6$...
    > Hi,
    >
    > I am sorry if this has been asked before but I couldn't find any hint
    > scheming through the list. So here is my problem:
    >
    > I have to write a client end in java reading data sent over UDP from
    > a server (the server is written in C). The data format is a C struct
    > with all the fields including long, int, float, etc. I got the data in
    > the java client end in the format of a DatagramPacket, then I convert it
    > to a byte array. Now the question is, how do I extract each field from
    > the byte array? I know this is silly but I have tried using classes like
    > ByteBuffer, Array, etc, without good result (The print out of the
    > recieiving end differs than that from the sending end).


    Experiment with a few struct values to find out where each byte ends up in the
    array. Then use << and | to fill ints and longs. If you want floats and
    doubles, use Float.intBitsToFloat() and Double.longBitsToDouble().
    Boudewijn Dijkstra, Sep 23, 2004
    #4
  5. On Wed, 22 Sep 2004 16:53:23 -0400, John Adams wrote:
    > I have to write a client end in java reading data sent over UDP
    > from a server (the server is written in C). The data format is a C
    > struct with all the fields including long, int, float, etc. I got
    > the data in the java client end in the format of a DatagramPacket,
    > then I convert it to a byte array. Now the question is, how do I
    > extract each field from the byte array?


    Others have already pointed out how you can read the data, however
    it's worth mentioning that if the server gets recompiled with a
    different compiler, or moved to a different platform (with no changes
    to the data types) it could break your client.

    The real solution is to change the server so that it doesn't write the
    data in a format that is quite so compiler and platform dependent.

    Writing raw binary data directly from a C struct is lazy, fragile and
    non-portable. And as you've already discovered, it makes it difficult
    and error prone to write a client in a different language, that
    doesn't use the same compiler, or that will run on a different
    hardware platform.

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
    Gordon Beaton, Sep 23, 2004
    #5
  6. John Adams

    John Adams Guest

    Gordon Beaton wrote:
    > On Wed, 22 Sep 2004 16:53:23 -0400, John Adams wrote:
    >
    >> I have to write a client end in java reading data sent over UDP
    >>from a server (the server is written in C). The data format is a C
    >>struct with all the fields including long, int, float, etc. I got
    >>the data in the java client end in the format of a DatagramPacket,
    >>then I convert it to a byte array. Now the question is, how do I
    >>extract each field from the byte array?

    >
    >
    > Others have already pointed out how you can read the data, however
    > it's worth mentioning that if the server gets recompiled with a
    > different compiler, or moved to a different platform (with no changes
    > to the data types) it could break your client.
    >
    > The real solution is to change the server so that it doesn't write the
    > data in a format that is quite so compiler and platform dependent.
    >
    > Writing raw binary data directly from a C struct is lazy, fragile and
    > non-portable. And as you've already discovered, it makes it difficult
    > and error prone to write a client in a different language, that
    > doesn't use the same compiler, or that will run on a different
    > hardware platform.
    >
    > /gordon
    >



    Hi,

    Thank you all for your invaluable advice. To date, I have seemed to
    try all the methods mentioned: Array, ByteBuffer, DataInputStream,
    ByteArrayInputStream, intBitsToFloat(), etc. I had, ofcourse as Michael
    Borgwardt so "brilliantly" pointed out, tried putting some values on the
    server side and dumping them out on the client side (That is HOW I know
    they are not the same!!!).

    Flatform and hardware also are not the concern since I had tried
    running the server on the very same intel machine as the client (just to
    debug). Endianess is also out of question in this scenario.

    This is how I do it:

    // The data format from a C server.
    // To make thing simple, I make all fiels float
    typede struct {
    float something;
    float some_other_thing; ...
    }

    // class to convert data on the java end
    public class StructureInputStream extends ByteArrayInputStream
    {
    static int curr = 0;
    private DataInputStream dis = null;

    public StructureInputStream(byte[] b)
    {
    super(b);
    curr = 1;
    dis = new DataInputStream((InputStream)this);
    }

    public final long readLong() throws IOException, EOFException
    {
    return dis.readLong();
    }

    public final double readDouble() throws IOException, EOFException
    {
    return dis.readDouble();
    }

    public final int readInt() throws IOException, EOFException
    {
    return dis.readInt();
    }

    public final int readUnsignedShort() throws IOException, EOFException
    {
    return dis.readUnsignedShort();
    }

    public final float readFloat() throws IOException, EOFException
    {
    return dis.readFloat();
    }

    public final short readShort() throws IOException, EOFException
    {
    return dis.readShort();
    }

    public final char readChar() throws IOException, EOFException
    {
    return (char)dis.readByte();
    }

    public void skipBytes(int n) throws IOException
    {
    int num_bytes_to_skip = n;
    int x = 0;

    do {
    x = dis.skipBytes(num_bytes_to_skip);
    num_bytes_to_skip -= x;
    } while (num_bytes_to_skip>0);
    }
    }


    // And how the UDP client gets data
    // try - catch blocks are omitted for clarity
    DatagramPacket pkt = new DatagramPacket(buffer, buffer.length);
    socket.receive(pkt);
    byte[] data = pkt.getData();
    StructureInputStream t = new StructureInputStream(data);
    float a1 = t.readFloat(); t.skipBytes(4);
    float a2 = t.readFloat(); t.skipBytes(4);
    float a3 = t.readFloat(); t.skipBytes(4);
    float a4 = t.readFloat(); t.skipBytes(4);

    // print out ...


    Again, thank you all for your advice.
    Regards,
    John Adams, Sep 23, 2004
    #6
  7. John Adams

    Paul Lutus Guest

    John Adams wrote:

    / ...

    > Thank you all for your invaluable advice. To date, I have seemed to
    > try all the methods mentioned: Array, ByteBuffer, DataInputStream,
    > ByteArrayInputStream, intBitsToFloat(), etc. I had, ofcourse as Michael
    > Borgwardt so "brilliantly" pointed out, tried putting some values on the
    > server side and dumping them out on the client side (That is HOW I know
    > they are not the same!!!).


    Okay, to restate a point made by someone else, why are you trying to
    transfer the data in binary form? Why not make the server emit the data in
    plain-text form? This is a much more robust solution, one that is immune to
    nearly all the issues that are occupying your thinking right now.

    In other words, you may spend months debugging this binary transfer
    solution, only to see your work entirely unraveled by a revision in a
    future compiler, or you can use plain text and be done with it.

    --
    Paul Lutus
    http://www.arachnoid.com
    Paul Lutus, Sep 23, 2004
    #7
  8. John Adams

    Will Hartung Guest

    "Paul Lutus" <> wrote in message
    news:...
    > Okay, to restate a point made by someone else, why are you trying to
    > transfer the data in binary form? Why not make the server emit the data in
    > plain-text form? This is a much more robust solution, one that is immune

    to
    > nearly all the issues that are occupying your thinking right now.
    >
    > In other words, you may spend months debugging this binary transfer
    > solution, only to see your work entirely unraveled by a revision in a
    > future compiler, or you can use plain text and be done with it.


    The choice of format may well be out of the hands of the guy with the
    problem. Could be a legacy or 3rd party application.

    The fact that there may be other, more appropriate ways to transfer data
    doesn't necessarily lessen the need for someone to do exactly what he's
    asking.

    Regards,

    Will Hartung
    ()
    Will Hartung, Sep 23, 2004
    #8
  9. John Adams

    Paul Lutus Guest

    Will Hartung wrote:

    / ...

    > The choice of format may well be out of the hands of the guy with the
    > problem. Could be a legacy or 3rd party application.
    >
    > The fact that there may be other, more appropriate ways to transfer data
    > doesn't necessarily lessen the need for someone to do exactly what he's
    > asking.


    Yes, but we don't know that, and the OP may not know the advantages of plain
    text. Therefore someone needed to mention it.

    --
    Paul Lutus
    http://www.arachnoid.com
    Paul Lutus, Sep 23, 2004
    #9
  10. John Adams wrote:
    > Thank you all for your invaluable advice. To date, I have seemed to
    > try all the methods mentioned: Array, ByteBuffer, DataInputStream,
    > ByteArrayInputStream, intBitsToFloat(), etc. I had, ofcourse as Michael
    > Borgwardt so "brilliantly" pointed out, tried putting some values on the
    > server side and dumping them out on the client side (That is HOW I know
    > they are not the same!!!).
    >
    > Flatform and hardware also are not the concern since I had tried
    > running the server on the very same intel machine as the client (just to
    > debug). Endianess is also out of question in this scenario.


    Sigh. Have you even read the documentation for DataInputStream? It reads
    big endian data, while your x86 is little endian. The other problem is
    that your structs probably contain random padding bytes.

    Your problem is that you use C structs as a communication format. This
    is inherently unportable (since it is in no way defined what a struct
    looks like in memory), even on the same platform if you happen to
    compile with a different compiler or even another version of the same
    compiler. You will need to actually design and specify the transmission
    format. XML or binary data with endianess/padding issues removed, do
    whatever you wish, but do not dump raw C structs.

    --
    Daniel Sjöblom
    Remove _NOSPAM to reply by mail
    =?ISO-8859-1?Q?Daniel_Sj=F6blom?=, Sep 23, 2004
    #10
  11. John Adams wrote:
    > Gordon Beaton wrote:
    >
    > // And how the UDP client gets data
    > // try - catch blocks are omitted for clarity
    > DatagramPacket pkt = new DatagramPacket(buffer, buffer.length);
    > socket.receive(pkt);
    > byte[] data = pkt.getData();
    > StructureInputStream t = new StructureInputStream(data);
    > float a1 = t.readFloat(); t.skipBytes(4);
    > float a2 = t.readFloat(); t.skipBytes(4);
    > float a3 = t.readFloat(); t.skipBytes(4);
    > float a4 = t.readFloat(); t.skipBytes(4);
    >
    >


    So you are ignoring the endian-ness issue completely, although you
    seem to be addressing the possible alignment issue (skipping 4
    bytes after every float).

    Look in your compiler documentation and see exactly how it stores
    a float. Compare this with the java documentation on how a
    DataInputStream reads a float. I bet they're different. Endian-ness
    may possibly be the only difference.

    Only you know how the compiler stores floats in memory. Only you
    can work out how to decode that back into float values.

    If the server is outside your control then you have no option
    but to ask the author exactly what format it sends on the network,
    or work it out yourself.

    If the server is in your control then it's your responsibility
    to document what the server sends. And change the format to
    something you CAN document (parhaps text as Paul suggests) if
    you cannot work out what your current implementation sends.

    Either way, you cannot work on decoding the data until you know
    how it's been encoded.

    Steve
    Steve Horsley, Sep 23, 2004
    #11
  12. John Adams

    John Adams Guest

    Steve Horsley wrote:
    > John Adams wrote:
    >
    >> Gordon Beaton wrote:
    >>
    >> // And how the UDP client gets data
    >> // try - catch blocks are omitted for clarity
    >> DatagramPacket pkt = new DatagramPacket(buffer, buffer.length);
    >> socket.receive(pkt);
    >> byte[] data = pkt.getData();
    >> StructureInputStream t = new StructureInputStream(data);
    >> float a1 = t.readFloat(); t.skipBytes(4);
    >> float a2 = t.readFloat(); t.skipBytes(4);
    >> float a3 = t.readFloat(); t.skipBytes(4);
    >> float a4 = t.readFloat(); t.skipBytes(4);
    >>
    >>

    >
    > So you are ignoring the endian-ness issue completely, although you
    > seem to be addressing the possible alignment issue (skipping 4
    > bytes after every float).
    > Look in your compiler documentation and see exactly how it stores a
    > float. Compare this with the java documentation on how a DataInputStream
    > reads a float. I bet they're different. Endian-ness
    > may possibly be the only difference.
    > Only you know how the compiler stores floats in memory. Only you
    > can work out how to decode that back into float values.
    >
    > If the server is outside your control then you have no option
    > but to ask the author exactly what format it sends on the network,
    > or work it out yourself.
    >
    > If the server is in your control then it's your responsibility
    > to document what the server sends. And change the format to something
    > you CAN document (parhaps text as Paul suggests) if
    > you cannot work out what your current implementation sends.
    >
    > Either way, you cannot work on decoding the data until you know
    > how it's been encoded.
    >
    > Steve


    Hi all,

    I am deeply grateful for all your advice. Thanks to you all I have
    found the problem; and it is actually a legacy problem. The code has
    worked all along (I mean the Java end). The reason we saw a discrepancy
    in value display is that the C code did not print out the correct value!
    You can rest assure that if I could beat the guy who wrote that I would
    already have done so! Anyway, since I have no control over that, all I
    can do is just smile and be thankful that I have hit the right newsgroup
    to ask for help.

    It is really nice of you to pour in such a wonderful help.

    Regards,
    John Adams, Sep 23, 2004
    #12
  13. John Adams wrote:
    > I had, ofcourse as Michael
    > Borgwardt so "brilliantly" pointed out, tried putting some values on the
    > server side and dumping them out on the client side (That is HOW I know
    > they are not the same!!!).


    You obviously didn't understand my suggestion. My point was to NOT try
    to extract the individual values, but instead to dump the entire byte
    array to see how the values (which should be chosen to be easily
    recognizable even if scrambled a bit by endian issues) end up in the
    output. That can give you far more information than "it's not the same".
    Michael Borgwardt, Sep 23, 2004
    #13
  14. On Thu, 23 Sep 2004 20:35:19 +0100, Steve Horsley wrote:
    > John Adams wrote:
    >> Gordon Beaton wrote:


    Just to clarify, I didn't write *any* of the text (a short section of
    code) that you quoted in your reply to John.

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
    Gordon Beaton, Sep 24, 2004
    #14
  15. John Adams

    John Adams Guest

    John Adams wrote:
    > Hi,
    >
    > I am sorry if this has been asked before but I couldn't find any hint
    > scheming through the list. So here is my problem:
    >
    > I have to write a client end in java reading data sent over UDP from a
    > server (the server is written in C). The data format is a C struct with
    > all the fields including long, int, float, etc. I got the data in the
    > java client end in the format of a DatagramPacket, then I convert it to
    > a byte array. Now the question is, how do I extract each field from the
    > byte array? I know this is silly but I have tried using classes like
    > ByteBuffer, Array, etc, without good result (The print out of the
    > recieiving end differs than that from the sending end).
    >
    > Thanks for all the advice.



    Hi Guys,

    Just to let you know in case you missed my post, I have found the
    problem thanks to all your advice. If you'd like to know where the
    problem came from, check my reply I mistakenly posted under Steve
    Horsley's post.
    John Adams, Sep 24, 2004
    #15
  16. Gordon Beaton wrote:
    > On Thu, 23 Sep 2004 20:35:19 +0100, Steve Horsley wrote:
    >
    >>John Adams wrote:
    >>
    >>>Gordon Beaton wrote:

    >
    >
    > Just to clarify, I didn't write *any* of the text (a short section of
    > code) that you quoted in your reply to John.
    >
    > /gordon
    >


    Whoops, sorry. Sloppy snipping.
    I'll try to be more careful in future.

    Steve
    Steve Horsley, Sep 24, 2004
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Pete Fraser
    Replies:
    4
    Views:
    6,801
    Mike Treseler
    Nov 4, 2004
  2. Doug
    Replies:
    3
    Views:
    4,362
    Bruce Barker
    Nov 4, 2005
  3. Replies:
    37
    Views:
    3,216
    Nigel Wade
    Sep 3, 2007
  4. Santiago Romero

    Read binary file and dump data in

    Santiago Romero, Jan 13, 2009, in forum: Python
    Replies:
    3
    Views:
    2,233
    Santiago Romero
    Jan 14, 2009
  5. izumi
    Replies:
    2
    Views:
    193
    izumi
    Aug 3, 2004
Loading...

Share This Page