Best way to store structured data

[achalk]

I am reading configuration information from a device using a C API.
There are hundreds of settings (data items) so the logical way to
store the data in my app. would appear to be a struct. However, this
data is passed around across modules and different modules can have
different structure alignments.

What is the best way to store this type of data that is not subject to
data alignment problems?

Thanks!

 

[Ian Collins]

I am reading configuration information from a device using a C API.
There are hundreds of settings (data items) so the logical way to
store the data in my app. would appear to be a struct. However, this
data is passed around across modules and different modules can have
different structure alignments.

What is the best way to store this type of data that is not subject to
data alignment problems?

If you want complete portability, as text.

Declare your struct and add methods to stream it to and from text.

 

[achalk]

If you want complete portability, as text.

Declare your struct and add methods to stream it to and from text.

Thanks, Ian. I was afraid of that.

 

[Ebenezer]

Thanks, Ian. I was afraid of that.- Hide quoted text -

You can get pretty good portability with binary. What are
your requirements? If you were to use the C++ Middleware
Writer, writing methods to stream the data isn't needed.
Instead you add prototypes to your class and the
implementations are created for you.

Brian Wood
Ebenezer Enterprises
http://webEbenezer.net

 

[Goran]

I am reading configuration information from a device using a C API.
There are hundreds of settings (data items) so the logical way to
store the data in my app. would appear to be a struct. However, this
data is passed around across modules and different modules can have
different structure alignments.

What is the best way to store this type of data that is not subject to
data alignment problems?

For exchange between computers, a binary stream with defined endiannes
and/or alignment (alignment is normally not a question when data is in
such stream). Once your data is on the target host, use alignment
appropriate for that computer. You should be able to find a library
that helps you turn your stream into data (rolling your own ain't
hard, but is busywork).

I wouldn't use text as first choice, no thanks, Ian ;-).

Goran

 

[Goran]

I am reading configuration information from a device using a C API.
There are hundreds of settings (data items) so the logical way to
store the data in my app. would appear to be a struct. However, this
data is passed around across modules and different modules can have
different structure alignments.

Wait, I missed this. "Around modules", as in "separately compiled
*.lib/*.obj/*.so modules"?

If so, you are in the land module binary compatibility land (inside
one toolchain, on one platform, most likely). There, your best bet are
alignment-related compiler directives. On x86, for example, you either
just "pack" everything, or you use the default 32-bit alignment. If
so, forget streams altogether.

Goran.

 

[Ian Collins]

For exchange between computers, a binary stream with defined endiannes
and/or alignment (alignment is normally not a question when data is in
such stream). Once your data is on the target host, use alignment
appropriate for that computer. You should be able to find a library
that helps you turn your stream into data (rolling your own ain't
hard, but is busywork).

I wouldn't use text as first choice, no thanks, Ian ;-).

I use JSON for just about everything these days, portable between
machines and languages!

 

[achalk]

Wait, I missed this. "Around modules", as in "separately compiled
*.lib/*.obj/*.so modules"?

If so, you are in the land module binary compatibility land (inside
one toolchain, on one platform, most likely). There, your best bet are
alignment-related compiler directives. On x86, for example, you either
just "pack" everything, or you use the default 32-bit alignment. If
so, forget streams altogether.

Goran.

Yes. For now, this is Windows 32/64 but it is across languages C++
(native) and C#.
I assumed Ian meant binary when he said text.

 

[Ebenezer]

I use JSON for just about everything these days, portable between
machines and languages!

The OP says nothing about needing to support multiple languages.
There are more efficient alternatives if you know you can use C++.


Brian Wood

 

[Ebenezer]

Yes. For now, this is Windows 32/64 but it is across languages C++
(native) and C#.

In this case your best option may be an XML or JSON type
approach. And posting more of your requirements up front
would be helpful.

Brian Wood

 

[achalk]

In this case your best option may be an XML or JSON type
approach.  And posting more of your requirements up front
would be helpful.

Brian Wood

I was deliberately a bit vague Brian in order to encourage a broad
range of answers. JSON may be the way to go here.

 

[Goran]

Yes. For now, this is Windows 32/64 but it is across languages C++
(native) and C#.
I assumed Ian meant binary when he said text.

If you are mixing with C# or other stuff from .NET, you have interop
marshaling, which makes alignment a marshaling problem, not a binary
layout problem (although binary layout is a factor to consider when
writing your p/invoke stuff).

If you are mixing with e.g. FreePascal/native code Delphi, you need to
get alignment right for them. That's not hard, but requires a bit of
care when doing. Any way you look at it, it's not hard, and any
language/toolchain worth it's salt will allow you to get there.

If you ever move to a hardware architecture with different endiannes,
and you stay on binary for data exchange, you are gonna have to take
care about endiannes conversion for data created on one architecture
and consumed on the other. Or use some serialization library,
conversion to text notwithstanding.

Goran.

 

[achalk]

If you are mixing with C# or other stuff from .NET, you have interop
marshaling, which makes alignment a marshaling problem, not a binary
layout problem (although binary layout is a factor to consider when
writing your p/invoke stuff).

If you are mixing with e.g. FreePascal/native code Delphi, you need to
get alignment right for them. That's not hard, but requires a bit of
care when doing. Any way you look at it, it's not hard, and any
language/toolchain worth it's salt will allow you to get there.

If you ever move to a hardware architecture with different endiannes,
and you stay on binary for data exchange, you are gonna have to take
care about endiannes conversion for data created on one architecture
and consumed on the other. Or use some serialization library,
conversion to text notwithstanding.

Goran.

That's a good synopsis

 

[Nick Keighley]

I am reading configuration information from a device using a C API.
There are hundreds of settings (data items) so the logical way to
store the data in my app. would appear to be a struct. However, this
data is passed around across modules and different modules can have
different structure alignments.

why?!

Can you hide the data behind a class and have getters to retrieve the
data? Or does your bizzare module system preclude this?

What is the best way to store this type of data that is not subject to
data alignment problems?

text? ASN.1? XDR?

strike whoever used pragma pack repeatedly over the head with a copy
of <any heavy textbook>?