Implementing Legacy Byte Packed Message Interfaces

Discussion in 'Java' started by Chris, Nov 5, 2004.

  1. Chris

    Chris Guest

    We are currently working on a project where new Java based sub-systems
    are being created to replace legacy ADA implementation. The Java
    inter-component communication is being realized by Java JMS. The
    architecture provides for "boundary" components that will be
    responsible for communicating with legacy systems that implement
    messages as C Structs many of which use C bit notation ( message
    fields may represent bit masks that are not on byte boundaries). We
    have been looking at using Java JNI to perform the packing/unpacking
    to/from Java Objects and C Structs. What other alternatives exist? It
    was suggested that we create Java implementation to do all of the
    pack/unpack. This would work for primitive types but would break down
    when dealing with legacy fields that are bit masks. I have briefly
    looked at the Java NIO implementation of ByteBuffer but in appears
    that it is limited to primitive types. This problem space must be
    common in systems that must interface with legacy devices whose
    message interfaces are embodied in tightly packed byte based messages.
    Are there any open source solutions in the Java space that attempts to
    solve this problem so that we do not have to re-invent the wheel ???
    Chris, Nov 5, 2004
    #1
    1. Advertising

  2. Chris

    Chris Uppal Guest

    Chris wrote:

    > [...] legacy systems that implement
    > messages as C Structs many of which use C bit notation ( message
    > fields may represent bit masks that are not on byte boundaries).


    The C bit notation should never be, and should never /have/ been, used to
    specify in-memory layout.

    The layout of bitfields is almost totally non-portable. IIRC, the only defined
    requirement is that they appear in memory in the same order as they do in the
    source, how much padding -- if any -- is between them is totally implementation
    specific. There might be problems with signedness too, but my memory of that
    aspect of the standard has faded completely :-(

    (All the above, of course, is true for C structs in general, but bitfields are
    even worse than usual because there seems to be a nearly irresistible
    temptation for C programmers to abuse them.)

    In essence you will have to do exactly what you would do if you were
    interfacing with those legacy systems using C. Find out how the compiler for
    the legacy code lays out the data in memory, read the data in as some primitive
    type (byte or int seems natural), and define appropriate bitmasks that overlay
    the layout to extract the data you want.

    I would avoid JNI for this sort of thing, if I were you -- all you are doing is
    putting off the problem (which is really caused by whoever it was who thought
    you could use C structs and bitfields to define memory layout portably), and
    the cost of crossing the JNI boundary is quite high (I put it at in the order
    of magnitude of 1 usec on the machines I use).

    -- chris
    Chris Uppal, Nov 5, 2004
    #2
    1. Advertising

  3. Chris

    Chris Guest

    "Chris Uppal" <-THIS.org> wrote in message news:<>...
    > Chris wrote:
    >
    > > [...] legacy systems that implement
    > > messages as C Structs many of which use C bit notation ( message
    > > fields may represent bit masks that are not on byte boundaries).

    >
    > The C bit notation should never be, and should never /have/ been, used to
    > specify in-memory layout.
    >
    > The layout of bitfields is almost totally non-portable. IIRC, the only defined
    > requirement is that they appear in memory in the same order as they do in the
    > source, how much padding -- if any -- is between them is totally implementation
    > specific. There might be problems with signedness too, but my memory of that
    > aspect of the standard has faded completely :-(
    >
    > (All the above, of course, is true for C structs in general, but bitfields are
    > even worse than usual because there seems to be a nearly irresistible
    > temptation for C programmers to abuse them.)
    >
    > In essence you will have to do exactly what you would do if you were
    > interfacing with those legacy systems using C. Find out how the compiler for
    > the legacy code lays out the data in memory, read the data in as some primitive
    > type (byte or int seems natural), and define appropriate bitmasks that overlay
    > the layout to extract the data you want.
    >
    > I would avoid JNI for this sort of thing, if I were you -- all you are doing is
    > putting off the problem (which is really caused by whoever it was who thought
    > you could use C structs and bitfields to define memory layout portably), and
    > the cost of crossing the JNI boundary is quite high (I put it at in the order
    > of magnitude of 1 usec on the machines I use).
    >
    > -- chris



    Chris,

    Thanks for feedback. It is unfortunate that the C Structs do not just
    support primitive types. That's a constraint placed upon us by the
    legacy device message interface. My initial thought was to leverage
    the C based JNI layer to overlay the input bag of bytes using the C
    Struct, extract the fields (leveraging the underlying C code to
    minipulate the bits), and then retain in the target Java Object via
    the JNI calls. Is the run-time cost of this approach less/more
    efficient then doing all the bit minipulation in Java via home-grown
    conversion utilities ???

    Thanks
    Chris, Nov 6, 2004
    #3
  4. Chris

    xarax Guest

    "Chris" <> wrote in message
    news:...
    > "Chris Uppal" <-THIS.org> wrote in message

    news:<>...
    > > Chris wrote:
    > >
    > > > [...] legacy systems that implement
    > > > messages as C Structs many of which use C bit notation ( message
    > > > fields may represent bit masks that are not on byte boundaries).

    > >
    > > The C bit notation should never be, and should never /have/ been, used to
    > > specify in-memory layout.
    > >
    > > The layout of bitfields is almost totally non-portable. IIRC, the only

    defined
    > > requirement is that they appear in memory in the same order as they do in

    the
    > > source, how much padding -- if any -- is between them is totally

    implementation
    > > specific. There might be problems with signedness too, but my memory of

    that
    > > aspect of the standard has faded completely :-(
    > >
    > > (All the above, of course, is true for C structs in general, but bitfields

    are
    > > even worse than usual because there seems to be a nearly irresistible
    > > temptation for C programmers to abuse them.)
    > >
    > > In essence you will have to do exactly what you would do if you were
    > > interfacing with those legacy systems using C. Find out how the compiler

    for
    > > the legacy code lays out the data in memory, read the data in as some

    primitive
    > > type (byte or int seems natural), and define appropriate bitmasks that

    overlay
    > > the layout to extract the data you want.
    > >
    > > I would avoid JNI for this sort of thing, if I were you -- all you are doing

    is
    > > putting off the problem (which is really caused by whoever it was who

    thought
    > > you could use C structs and bitfields to define memory layout portably), and
    > > the cost of crossing the JNI boundary is quite high (I put it at in the

    order
    > > of magnitude of 1 usec on the machines I use).
    > >
    > > -- chris

    >
    >
    > Chris,
    >
    > Thanks for feedback. It is unfortunate that the C Structs do not just
    > support primitive types. That's a constraint placed upon us by the
    > legacy device message interface. My initial thought was to leverage
    > the C based JNI layer to overlay the input bag of bytes using the C
    > Struct, extract the fields (leveraging the underlying C code to
    > minipulate the bits), and then retain in the target Java Object via
    > the JNI calls. Is the run-time cost of this approach less/more
    > efficient then doing all the bit minipulation in Java via home-grown
    > conversion utilities ???


    I had this problem for implementing a TCP/IP layer that
    could speak both Java and C. I used the java.nio packages
    for the SocketChannel and ByteBuffer things. I defined
    each message packet as a collection of fields with the
    various primitive types. The "definition" was an XML
    specification file that is processed by a Java language
    program generator. The program generator spits out the
    Java and C source files that can manage the message
    packets.

    The Java side has a class for each message packet type.
    The class has getter/setter methods for each field that
    can read/write the value from/to the ByteBuffer at the
    appropriate offset.

    The C side has a set of functions (similar to a Java
    class) for each message packet type. The functions support
    reading/writing the field values to a byte array buffer.
    The functions are named similarly to the Java getter/setter
    methods and referenced through function pointers in a
    generated struct to avoid name clashes between different
    message packet types. For example, there are several
    types that each have a "threadId" field. Each "class" has
    a "getThreadId" and a "setThreadId" function, which are
    actually function pointer fields in a "class" struct.

    The whole thing is managed by a Java GUI that displays/edits
    the contents of the XML specification file using JTables.
    Adding/changing field names and types is very simple with
    the GUI. A button click generates the Java and C source
    code, which I then compile in the usual way.

    The "client" code that uses the generated code simply calls
    the getter/setter methods to read/write the various fields.
    "Endianness" is also handled transparently by specifying
    whether the C code is expecting Big versus Little Endian.
    (The Java side is always Big Endian.) The ByteBuffer has an
    Endianness flag.

    btw: This mechanism doesn't support C bit fields. Instead I
    use the usual bitwise AND/OR masking to process the individual
    bits of a byte field. Using bit masks, instead of bit
    fields, insulates my code from the weird ordering and
    boundaries of bits within a bit field.


    --
    ----------------------------
    Jeffrey D. Smith
    Farsight Systems Corporation
    24 BURLINGTON DRIVE
    LONGMONT, CO 80501-6906
    http://www.farsight-systems.com
    z/Debug debugs your Systems/C programs running on IBM z/OS for FREE!
    xarax, Nov 6, 2004
    #4
  5. Chris

    Ken Guest

    "Chris Uppal" <-THIS.org> wrote in message news:<>...

    >
    > In essence you will have to do exactly what you would do if you were
    > interfacing with those legacy systems using C. Find out how the compiler for
    > the legacy code lays out the data in memory, read the data in as some primitive
    > type (byte or int seems natural), and define appropriate bitmasks that overlay
    > the layout to extract the data you want.


    How do you make bitmasks in Java, especially ones that will handle
    these non-byte boundary bit fields?

    Thanks,

    Ken
    Ken, Nov 6, 2004
    #5
  6. Chris

    xarax Guest

    "Ken" <> wrote in message
    news:...
    > "Chris Uppal" <-THIS.org> wrote in message

    news:<>...
    >
    > >
    > > In essence you will have to do exactly what you would do if you were
    > > interfacing with those legacy systems using C. Find out how the compiler

    for
    > > the legacy code lays out the data in memory, read the data in as some

    primitive
    > > type (byte or int seems natural), and define appropriate bitmasks that

    overlay
    > > the layout to extract the data you want.

    >
    > How do you make bitmasks in Java, especially ones that will handle
    > these non-byte boundary bit fields?


    Use the bitwise operators just like in C for
    AND'ing and OR'ing, and use shifting.
    xarax, Nov 6, 2004
    #6
  7. Chris

    Chris Uppal Guest

    Chris wrote:

    > Thanks for feedback. It is unfortunate that the C Structs do not just
    > support primitive types. That's a constraint placed upon us by the
    > legacy device message interface. My initial thought was to leverage
    > the C based JNI layer to overlay the input bag of bytes using the C
    > Struct, extract the fields (leveraging the underlying C code to
    > minipulate the bits), and then retain in the target Java Object via
    > the JNI calls. Is the run-time cost of this approach less/more
    > efficient then doing all the bit minipulation in Java via home-grown
    > conversion utilities ???


    Without knowing the details of you application, I can only make guesses from
    a distance. However...

    If you think about it, accessing these fields will require bit-twiddling
    (masking , shifting, etc) to decode them however you do it, the only difference
    is whether the relevant instructions are generated automatically by the C
    compiler, or semi-automatically by the JVM's JITer following instructions
    provided by you in Java. There's not likely to be a huge amount of difference
    between the two. So the additional overhead of JNI is only justified if either
    (a) you are already using JNI to get the data and the only question is whether
    to translate it to a more portable format before or after passing it across the
    JNI boundary, or (b) ....

    Well, actually, I can't think of a (b). JNI is complicated and difficult to
    maintain (if only because it has a different build-path, and requires skills
    that not all Java programmers have). So it's hard to imagine a situation where
    the (comparative) slowness and complexity of JNI would be better than doing a
    little bit-twiddling in Java.

    I suppose if you had vast numbers of these structures to decode, and they kept
    changing (i.e. the "legacy" code isn't really legacy, but is under active
    development and you can't persuade the developers that changing externally
    defined structures is a /bad/ idea....). But even then, I think there would be
    better ways of handing it (hitting the other developers of their collective
    heads with a brick would be a good start ;-) as would be sending them on a good
    course for a refresher in the dos-and-don'ts of C programming).

    Note that the layout of structures and bitfields in memory is dependent on the
    compiler, and particularly on the /settings/ of the compiler -- and there's
    always the risk that the settings required for compiling JNI code would
    conflict the settings that your legacy code assumes. (Whether that would
    /actually/ be a problem is impossible for me to guess.)

    -- chris
    Chris Uppal, Nov 7, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. James D Carroll

    Implementing "package" interfaces

    James D Carroll, Jun 28, 2004, in forum: Java
    Replies:
    4
    Views:
    453
    Chris Smith
    Jun 28, 2004
  2. Oliver Wong
    Replies:
    16
    Views:
    596
    Oliver Wong
    Jun 28, 2006
  3. josh
    Replies:
    6
    Views:
    423
    Ed Kirwan
    Dec 19, 2006
  4. Replies:
    13
    Views:
    721
    Dave Rahardja
    Feb 2, 2007
  5. Nicolas Vincent
    Replies:
    4
    Views:
    188
    Nicolas Vincent
    Dec 18, 2009
Loading...

Share This Page