Best way to store structured data

Discussion in 'C++' started by achalk, Dec 20, 2010.

  1. achalk

    achalk Guest

    I am reading configuration information from a device using a C API.
    There are hundreds of settings (data items) so the logical way to
    store the data in my app. would appear to be a struct. However, this
    data is passed around across modules and different modules can have
    different structure alignments.

    What is the best way to store this type of data that is not subject to
    data alignment problems?

    Thanks!
    achalk, Dec 20, 2010
    #1
    1. Advertising

  2. achalk

    Ian Collins Guest

    On 12/21/10 11:59 AM, achalk wrote:
    > I am reading configuration information from a device using a C API.
    > There are hundreds of settings (data items) so the logical way to
    > store the data in my app. would appear to be a struct. However, this
    > data is passed around across modules and different modules can have
    > different structure alignments.
    >
    > What is the best way to store this type of data that is not subject to
    > data alignment problems?


    If you want complete portability, as text.

    Declare your struct and add methods to stream it to and from text.

    --
    Ian Collins
    Ian Collins, Dec 20, 2010
    #2
    1. Advertising

  3. achalk

    achalk Guest

    On Dec 20, 5:10 pm, Ian Collins <> wrote:
    > On 12/21/10 11:59 AM, achalk wrote:
    >
    > > I am reading configuration information from a device using a C API.
    > > There are hundreds of settings (data items) so the logical way to
    > > store the data in my app. would appear to be a struct. However, this
    > > data is passed around across modules and different modules can have
    > > different structure alignments.

    >
    > > What is the best way to store this type of data that is not subject to
    > > data alignment problems?

    >
    > If you want complete portability, as text.
    >
    > Declare your struct and add methods to stream it to and from text.
    >
    > --
    > Ian Collins


    Thanks, Ian. I was afraid of that.
    achalk, Dec 20, 2010
    #3
  4. achalk

    Ebenezer Guest

    On Dec 20, 5:33 pm, achalk <> wrote:
    > On Dec 20, 5:10 pm, Ian Collins <> wrote:
    >
    >
    >
    >
    >
    > > On 12/21/10 11:59 AM, achalk wrote:

    >
    > > > I am reading configuration information from a device using a C API.
    > > > There are hundreds of settings (data items) so the logical way to
    > > > store the data in my app. would appear to be a struct. However, this
    > > > data is passed around across modules and different modules can have
    > > > different structure alignments.

    >
    > > > What is the best way to store this type of data that is not subject to
    > > > data alignment problems?

    >
    > > If you want complete portability, as text.

    >
    > > Declare your struct and add methods to stream it to and from text.

    >
    > Thanks, Ian. I was afraid of that.- Hide quoted text -
    >


    You can get pretty good portability with binary. What are
    your requirements? If you were to use the C++ Middleware
    Writer, writing methods to stream the data isn't needed.
    Instead you add prototypes to your class and the
    implementations are created for you.

    Brian Wood
    Ebenezer Enterprises
    http://webEbenezer.net
    Ebenezer, Dec 21, 2010
    #4
  5. achalk

    Goran Guest

    On Dec 20, 11:59 pm, achalk <> wrote:
    > I am reading configuration information from a device using a C API.
    > There are hundreds of settings (data items) so the logical way to
    > store the data in my app. would appear to be a struct. However, this
    > data is passed around across modules and different modules can have
    > different structure alignments.
    >
    > What is the best way to store this type of data that is not subject to
    > data alignment problems?


    For exchange between computers, a binary stream with defined endiannes
    and/or alignment (alignment is normally not a question when data is in
    such stream). Once your data is on the target host, use alignment
    appropriate for that computer. You should be able to find a library
    that helps you turn your stream into data (rolling your own ain't
    hard, but is busywork).

    I wouldn't use text as first choice, no thanks, Ian ;-).

    Goran
    Goran, Dec 21, 2010
    #5
  6. achalk

    Goran Guest

    On Dec 20, 11:59 pm, achalk <> wrote:
    > I am reading configuration information from a device using a C API.
    > There are hundreds of settings (data items) so the logical way to
    > store the data in my app. would appear to be a struct. However, this
    > data is passed around across modules and different modules can have
    > different structure alignments.


    Wait, I missed this. "Around modules", as in "separately compiled
    *.lib/*.obj/*.so modules"?

    If so, you are in the land module binary compatibility land (inside
    one toolchain, on one platform, most likely). There, your best bet are
    alignment-related compiler directives. On x86, for example, you either
    just "pack" everything, or you use the default 32-bit alignment. If
    so, forget streams altogether.

    Goran.
    Goran, Dec 21, 2010
    #6
  7. achalk

    Ian Collins Guest

    On 12/21/10 06:44 PM, Goran wrote:
    > On Dec 20, 11:59 pm, achalk<> wrote:
    >> I am reading configuration information from a device using a C API.
    >> There are hundreds of settings (data items) so the logical way to
    >> store the data in my app. would appear to be a struct. However, this
    >> data is passed around across modules and different modules can have
    >> different structure alignments.
    >>
    >> What is the best way to store this type of data that is not subject to
    >> data alignment problems?

    >
    > For exchange between computers, a binary stream with defined endiannes
    > and/or alignment (alignment is normally not a question when data is in
    > such stream). Once your data is on the target host, use alignment
    > appropriate for that computer. You should be able to find a library
    > that helps you turn your stream into data (rolling your own ain't
    > hard, but is busywork).
    >
    > I wouldn't use text as first choice, no thanks, Ian ;-).


    I use JSON for just about everything these days, portable between
    machines and languages!

    --
    Ian Collins
    Ian Collins, Dec 21, 2010
    #7
  8. achalk

    achalk Guest

    On Dec 21, 1:50 am, Goran <> wrote:
    > On Dec 20, 11:59 pm, achalk <> wrote:
    >
    > > I am reading configuration information from a device using a C API.
    > > There are hundreds of settings (data items) so the logical way to
    > > store the data in my app. would appear to be a struct. However, this
    > > data is passed around across modules and different modules can have
    > > different structure alignments.

    >
    > Wait, I missed this. "Around modules", as in "separately compiled
    > *.lib/*.obj/*.so modules"?
    >
    > If so, you are in the land module binary compatibility land (inside
    > one toolchain, on one platform, most likely). There, your best bet are
    > alignment-related compiler directives. On x86, for example, you either
    > just "pack" everything, or you use the default 32-bit alignment. If
    > so, forget streams altogether.
    >
    > Goran.


    Yes. For now, this is Windows 32/64 but it is across languages C++
    (native) and C#.
    I assumed Ian meant binary when he said text.
    achalk, Dec 21, 2010
    #8
  9. achalk

    Ebenezer Guest

    On Dec 21, 1:52 am, Ian Collins <> wrote:
    > On 12/21/10 06:44 PM, Goran wrote:
    >
    >
    >
    >
    >
    > > On Dec 20, 11:59 pm, achalk<>  wrote:
    > >> I am reading configuration information from a device using a C API.
    > >> There are hundreds of settings (data items) so the logical way to
    > >> store the data in my app. would appear to be a struct. However, this
    > >> data is passed around across modules and different modules can have
    > >> different structure alignments.

    >
    > >> What is the best way to store this type of data that is not subject to
    > >> data alignment problems?

    >
    > > For exchange between computers, a binary stream with defined endiannes
    > > and/or alignment (alignment is normally not a question when data is in
    > > such stream). Once your data is on the target host, use alignment
    > > appropriate for that computer. You should be able to find a library
    > > that helps you turn your stream into data (rolling your own ain't
    > > hard, but is busywork).

    >
    > > I wouldn't use text as first choice, no thanks, Ian ;-).

    >
    > I use JSON for just about everything these days, portable between
    > machines and languages!
    >


    The OP says nothing about needing to support multiple languages.
    There are more efficient alternatives if you know you can use C++.


    Brian Wood
    Ebenezer, Dec 21, 2010
    #9
  10. achalk

    Ebenezer Guest

    On Dec 21, 10:12 am, achalk <> wrote:
    > On Dec 21, 1:50 am, Goran <> wrote:
    >
    >
    >
    >
    >
    > > On Dec 20, 11:59 pm, achalk <> wrote:

    >
    > > > I am reading configuration information from a device using a C API.
    > > > There are hundreds of settings (data items) so the logical way to
    > > > store the data in my app. would appear to be a struct. However, this
    > > > data is passed around across modules and different modules can have
    > > > different structure alignments.

    >
    > > Wait, I missed this. "Around modules", as in "separately compiled
    > > *.lib/*.obj/*.so modules"?

    >
    > > If so, you are in the land module binary compatibility land (inside
    > > one toolchain, on one platform, most likely). There, your best bet are
    > > alignment-related compiler directives. On x86, for example, you either
    > > just "pack" everything, or you use the default 32-bit alignment. If
    > > so, forget streams altogether.

    >
    > > Goran.

    >
    > Yes. For now, this is Windows 32/64 but it is across languages C++
    > (native) and C#.


    In this case your best option may be an XML or JSON type
    approach. And posting more of your requirements up front
    would be helpful.

    Brian Wood
    Ebenezer, Dec 21, 2010
    #10
  11. achalk

    achalk Guest

    On Dec 21, 11:28 am, Ebenezer <> wrote:
    > On Dec 21, 10:12 am, achalk <> wrote:
    >
    >
    >
    >
    >
    > > On Dec 21, 1:50 am, Goran <> wrote:

    >
    > > > On Dec 20, 11:59 pm, achalk <> wrote:

    >
    > > > > I am reading configuration information from a device using a C API.
    > > > > There are hundreds of settings (data items) so the logical way to
    > > > > store the data in my app. would appear to be a struct. However, this
    > > > > data is passed around across modules and different modules can have
    > > > > different structure alignments.

    >
    > > > Wait, I missed this. "Around modules", as in "separately compiled
    > > > *.lib/*.obj/*.so modules"?

    >
    > > > If so, you are in the land module binary compatibility land (inside
    > > > one toolchain, on one platform, most likely). There, your best bet are
    > > > alignment-related compiler directives. On x86, for example, you either
    > > > just "pack" everything, or you use the default 32-bit alignment. If
    > > > so, forget streams altogether.

    >
    > > > Goran.

    >
    > > Yes. For now, this is Windows 32/64 but it is across languages C++
    > > (native) and C#.

    >
    > In this case your best option may be an XML or JSON type
    > approach.  And posting more of your requirements up front
    > would be helpful.
    >
    > Brian Wood


    I was deliberately a bit vague Brian in order to encourage a broad
    range of answers. JSON may be the way to go here.
    achalk, Dec 21, 2010
    #11
  12. achalk

    Goran Guest

    On Dec 21, 5:12 pm, achalk <> wrote:
    > On Dec 21, 1:50 am, Goran <> wrote:
    >
    >
    >
    > > On Dec 20, 11:59 pm, achalk <> wrote:

    >
    > > > I am reading configuration information from a device using a C API.
    > > > There are hundreds of settings (data items) so the logical way to
    > > > store the data in my app. would appear to be a struct. However, this
    > > > data is passed around across modules and different modules can have
    > > > different structure alignments.

    >
    > > Wait, I missed this. "Around modules", as in "separately compiled
    > > *.lib/*.obj/*.so modules"?

    >
    > > If so, you are in the land module binary compatibility land (inside
    > > one toolchain, on one platform, most likely). There, your best bet are
    > > alignment-related compiler directives. On x86, for example, you either
    > > just "pack" everything, or you use the default 32-bit alignment. If
    > > so, forget streams altogether.

    >
    > > Goran.

    >
    > Yes. For now, this is Windows 32/64 but it is across languages C++
    > (native) and C#.
    > I assumed Ian meant binary when he said text.


    If you are mixing with C# or other stuff from .NET, you have interop
    marshaling, which makes alignment a marshaling problem, not a binary
    layout problem (although binary layout is a factor to consider when
    writing your p/invoke stuff).

    If you are mixing with e.g. FreePascal/native code Delphi, you need to
    get alignment right for them. That's not hard, but requires a bit of
    care when doing. Any way you look at it, it's not hard, and any
    language/toolchain worth it's salt will allow you to get there.

    If you ever move to a hardware architecture with different endiannes,
    and you stay on binary for data exchange, you are gonna have to take
    care about endiannes conversion for data created on one architecture
    and consumed on the other. Or use some serialization library,
    conversion to text notwithstanding.

    Goran.
    Goran, Dec 22, 2010
    #12
  13. achalk

    achalk Guest

    On Dec 22, 1:30 am, Goran <> wrote:
    > On Dec 21, 5:12 pm, achalk <> wrote:
    >
    >
    >
    >
    >
    > > On Dec 21, 1:50 am, Goran <> wrote:

    >
    > > > On Dec 20, 11:59 pm, achalk <> wrote:

    >
    > > > > I am reading configuration information from a device using a C API.
    > > > > There are hundreds of settings (data items) so the logical way to
    > > > > store the data in my app. would appear to be a struct. However, this
    > > > > data is passed around across modules and different modules can have
    > > > > different structure alignments.

    >
    > > > Wait, I missed this. "Around modules", as in "separately compiled
    > > > *.lib/*.obj/*.so modules"?

    >
    > > > If so, you are in the land module binary compatibility land (inside
    > > > one toolchain, on one platform, most likely). There, your best bet are
    > > > alignment-related compiler directives. On x86, for example, you either
    > > > just "pack" everything, or you use the default 32-bit alignment. If
    > > > so, forget streams altogether.

    >
    > > > Goran.

    >
    > > Yes. For now, this is Windows 32/64 but it is across languages C++
    > > (native) and C#.
    > > I assumed Ian meant binary when he said text.

    >
    > If you are mixing with C# or other stuff from .NET, you have interop
    > marshaling, which makes alignment a marshaling problem, not a binary
    > layout problem (although binary layout is a factor to consider when
    > writing your p/invoke stuff).
    >
    > If you are mixing with e.g. FreePascal/native code Delphi, you need to
    > get alignment right for them. That's not hard, but requires a bit of
    > care when doing. Any way you look at it, it's not hard, and any
    > language/toolchain worth it's salt will allow you to get there.
    >
    > If you ever move to a hardware architecture with different endiannes,
    > and you stay on binary for data exchange, you are gonna have to take
    > care about endiannes conversion for data created on one architecture
    > and consumed on the other. Or use some serialization library,
    > conversion to text notwithstanding.
    >
    > Goran.


    That's a good synopsis
    achalk, Dec 22, 2010
    #13
  14. On Dec 20, 10:59 pm, achalk <> wrote:
    > I am reading configuration information from a device using a C API.
    > There are hundreds of settings (data items) so the logical way to
    > store the data in my app. would appear to be a struct. However, this
    > data is passed around across modules and different modules can have
    > different structure alignments.


    why?!

    Can you hide the data behind a class and have getters to retrieve the
    data? Or does your bizzare module system preclude this?

    > What is the best way to store this type of data that is not subject to
    > data alignment problems?


    text? ASN.1? XDR?

    strike whoever used pragma pack repeatedly over the head with a copy
    of <any heavy textbook>?
    Nick Keighley, Dec 30, 2010
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. KatB
    Replies:
    3
    Views:
    1,862
    Guest
    Oct 29, 2003
  2. Vince
    Replies:
    6
    Views:
    293
    Alipha
    Sep 1, 2005
  3. single threaded

    Best way to display and administer semi-structured data.

    single threaded, Jun 11, 2005, in forum: ASP .Net Web Controls
    Replies:
    1
    Views:
    106
    single threaded
    Jun 17, 2005
  4. Luis Esteban Valencia

    Best Way to store simple data with web service?

    Luis Esteban Valencia, Jan 17, 2005, in forum: ASP .Net Web Services
    Replies:
    0
    Views:
    126
    Luis Esteban Valencia
    Jan 17, 2005
  5. Jack
    Replies:
    2
    Views:
    197
    John W. Krahn
    Jan 25, 2008
Loading...

Share This Page