Implementing Legacy Byte Packed Message Interfaces

[Chris]

We are currently working on a project where new Java based sub-systems
are being created to replace legacy ADA implementation. The Java
inter-component communication is being realized by Java JMS. The
architecture provides for "boundary" components that will be
responsible for communicating with legacy systems that implement
messages as C Structs many of which use C bit notation ( message
fields may represent bit masks that are not on byte boundaries). We
have been looking at using Java JNI to perform the packing/unpacking
to/from Java Objects and C Structs. What other alternatives exist? It
was suggested that we create Java implementation to do all of the
pack/unpack. This would work for primitive types but would break down
when dealing with legacy fields that are bit masks. I have briefly
looked at the Java NIO implementation of ByteBuffer but in appears
that it is limited to primitive types. This problem space must be
common in systems that must interface with legacy devices whose
message interfaces are embodied in tightly packed byte based messages.
Are there any open source solutions in the Java space that attempts to
solve this problem so that we do not have to re-invent the wheel ???

 

[Chris Uppal]

[...] legacy systems that implement
messages as C Structs many of which use C bit notation ( message
fields may represent bit masks that are not on byte boundaries).

The C bit notation should never be, and should never /have/ been, used to
specify in-memory layout.

The layout of bitfields is almost totally non-portable. IIRC, the only defined
requirement is that they appear in memory in the same order as they do in the
source, how much padding -- if any -- is between them is totally implementation
specific. There might be problems with signedness too, but my memory of that
aspect of the standard has faded completely :-(

(All the above, of course, is true for C structs in general, but bitfields are
even worse than usual because there seems to be a nearly irresistible
temptation for C programmers to abuse them.)

In essence you will have to do exactly what you would do if you were
interfacing with those legacy systems using C. Find out how the compiler for
the legacy code lays out the data in memory, read the data in as some primitive
type (byte or int seems natural), and define appropriate bitmasks that overlay
the layout to extract the data you want.

I would avoid JNI for this sort of thing, if I were you -- all you are doing is
putting off the problem (which is really caused by whoever it was who thought
you could use C structs and bitfields to define memory layout portably), and
the cost of crossing the JNI boundary is quite high (I put it at in the order
of magnitude of 1 usec on the machines I use).

-- chris

 

[Chris]

[...] legacy systems that implement
messages as C Structs many of which use C bit notation ( message
fields may represent bit masks that are not on byte boundaries).

The C bit notation should never be, and should never /have/ been, used to
specify in-memory layout.

The layout of bitfields is almost totally non-portable. IIRC, the only defined
requirement is that they appear in memory in the same order as they do in the
source, how much padding -- if any -- is between them is totally implementation
specific. There might be problems with signedness too, but my memory of that
aspect of the standard has faded completely :-(

(All the above, of course, is true for C structs in general, but bitfields are
even worse than usual because there seems to be a nearly irresistible
temptation for C programmers to abuse them.)

In essence you will have to do exactly what you would do if you were
interfacing with those legacy systems using C. Find out how the compiler for
the legacy code lays out the data in memory, read the data in as some primitive
type (byte or int seems natural), and define appropriate bitmasks that overlay
the layout to extract the data you want.

I would avoid JNI for this sort of thing, if I were you -- all you are doing is
putting off the problem (which is really caused by whoever it was who thought
you could use C structs and bitfields to define memory layout portably), and
the cost of crossing the JNI boundary is quite high (I put it at in the order
of magnitude of 1 usec on the machines I use).

-- chris

Chris,

Thanks for feedback. It is unfortunate that the C Structs do not just
support primitive types. That's a constraint placed upon us by the
legacy device message interface. My initial thought was to leverage
the C based JNI layer to overlay the input bag of bytes using the C
Struct, extract the fields (leveraging the underlying C code to
minipulate the bits), and then retain in the target Java Object via
the JNI calls. Is the run-time cost of this approach less/more
efficient then doing all the bit minipulation in Java via home-grown
conversion utilities ???

Thanks

 

[xarax]

"Chris Uppal" <[email protected]> wrote in message

[...] legacy systems that implement
messages as C Structs many of which use C bit notation ( message
fields may represent bit masks that are not on byte boundaries).

The C bit notation should never be, and should never /have/ been, used to
specify in-memory layout.

The layout of bitfields is almost totally non-portable. IIRC, the only defined
requirement is that they appear in memory in the same order as they do in the
source, how much padding -- if any -- is between them is totally implementation
specific. There might be problems with signedness too, but my memory of that
aspect of the standard has faded completely :-(

(All the above, of course, is true for C structs in general, but bitfields are
even worse than usual because there seems to be a nearly irresistible
temptation for C programmers to abuse them.)

In essence you will have to do exactly what you would do if you were
interfacing with those legacy systems using C. Find out how the compiler for
the legacy code lays out the data in memory, read the data in as some primitive
type (byte or int seems natural), and define appropriate bitmasks that overlay
the layout to extract the data you want.

I would avoid JNI for this sort of thing, if I were you -- all you are doing is
putting off the problem (which is really caused by whoever it was who thought
you could use C structs and bitfields to define memory layout portably), and
the cost of crossing the JNI boundary is quite high (I put it at in the order
of magnitude of 1 usec on the machines I use).

-- chris

Chris,

Thanks for feedback. It is unfortunate that the C Structs do not just
support primitive types. That's a constraint placed upon us by the
legacy device message interface. My initial thought was to leverage
the C based JNI layer to overlay the input bag of bytes using the C
Struct, extract the fields (leveraging the underlying C code to
minipulate the bits), and then retain in the target Java Object via
the JNI calls. Is the run-time cost of this approach less/more
efficient then doing all the bit minipulation in Java via home-grown
conversion utilities ???

I had this problem for implementing a TCP/IP layer that
could speak both Java and C. I used the java.nio packages
for the SocketChannel and ByteBuffer things. I defined
each message packet as a collection of fields with the
various primitive types. The "definition" was an XML
specification file that is processed by a Java language
program generator. The program generator spits out the
Java and C source files that can manage the message
packets.

The Java side has a class for each message packet type.
The class has getter/setter methods for each field that
can read/write the value from/to the ByteBuffer at the
appropriate offset.

The C side has a set of functions (similar to a Java
class) for each message packet type. The functions support
reading/writing the field values to a byte array buffer.
The functions are named similarly to the Java getter/setter
methods and referenced through function pointers in a
generated struct to avoid name clashes between different
message packet types. For example, there are several
types that each have a "threadId" field. Each "class" has
a "getThreadId" and a "setThreadId" function, which are
actually function pointer fields in a "class" struct.

The whole thing is managed by a Java GUI that displays/edits
the contents of the XML specification file using JTables.
Adding/changing field names and types is very simple with
the GUI. A button click generates the Java and C source
code, which I then compile in the usual way.

The "client" code that uses the generated code simply calls
the getter/setter methods to read/write the various fields.
"Endianness" is also handled transparently by specifying
whether the C code is expecting Big versus Little Endian.
(The Java side is always Big Endian.) The ByteBuffer has an
Endianness flag.

btw: This mechanism doesn't support C bit fields. Instead I
use the usual bitwise AND/OR masking to process the individual
bits of a byte field. Using bit masks, instead of bit
fields, insulates my code from the weird ordering and
boundaries of bits within a bit field.


--
----------------------------
Jeffrey D. Smith
Farsight Systems Corporation
24 BURLINGTON DRIVE
LONGMONT, CO 80501-6906
http://www.farsight-systems.com
z/Debug debugs your Systems/C programs running on IBM z/OS for FREE!

 

[Ken]

In essence you will have to do exactly what you would do if you were
interfacing with those legacy systems using C. Find out how the compiler for
the legacy code lays out the data in memory, read the data in as some primitive
type (byte or int seems natural), and define appropriate bitmasks that overlay
the layout to extract the data you want.

How do you make bitmasks in Java, especially ones that will handle
these non-byte boundary bit fields?

Thanks,

Ken

 

[xarax]

"Chris Uppal" <[email protected]> wrote in message

How do you make bitmasks in Java, especially ones that will handle
these non-byte boundary bit fields?

Use the bitwise operators just like in C for
AND'ing and OR'ing, and use shifting.

 

[Chris Uppal]

Thanks for feedback. It is unfortunate that the C Structs do not just
support primitive types. That's a constraint placed upon us by the
legacy device message interface. My initial thought was to leverage
the C based JNI layer to overlay the input bag of bytes using the C
Struct, extract the fields (leveraging the underlying C code to
minipulate the bits), and then retain in the target Java Object via
the JNI calls. Is the run-time cost of this approach less/more
efficient then doing all the bit minipulation in Java via home-grown
conversion utilities ???

Without knowing the details of you application, I can only make guesses from
a distance. However...

If you think about it, accessing these fields will require bit-twiddling
(masking , shifting, etc) to decode them however you do it, the only difference
is whether the relevant instructions are generated automatically by the C
compiler, or semi-automatically by the JVM's JITer following instructions
provided by you in Java. There's not likely to be a huge amount of difference
between the two. So the additional overhead of JNI is only justified if either
(a) you are already using JNI to get the data and the only question is whether
to translate it to a more portable format before or after passing it across the
JNI boundary, or (b) ....

Well, actually, I can't think of a (b). JNI is complicated and difficult to
maintain (if only because it has a different build-path, and requires skills
that not all Java programmers have). So it's hard to imagine a situation where
the (comparative) slowness and complexity of JNI would be better than doing a
little bit-twiddling in Java.

I suppose if you had vast numbers of these structures to decode, and they kept
changing (i.e. the "legacy" code isn't really legacy, but is under active
development and you can't persuade the developers that changing externally
defined structures is a /bad/ idea....). But even then, I think there would be
better ways of handing it (hitting the other developers of their collective
heads with a brick would be a good start ;-) as would be sending them on a good
course for a refresher in the dos-and-don'ts of C programming).

Note that the layout of structures and bitfields in memory is dependent on the
compiler, and particularly on the /settings/ of the compiler -- and there's
always the risk that the settings required for compiling JNI code would
conflict the settings that your legacy code assumes. (Whether that would
/actually/ be a problem is impossible for me to guess.)

-- chris