Re: What is the right way to do this simple thing

Discussion in 'C Programming' started by Tim Rentsch, Mar 10, 2012.

  1. Tim Rentsch

    Tim Rentsch Guest

    (Richard Harter) writes:

    > Suppose that I have some data that is pointer free. It isn't
    > necessarily serialized in some format such as JSON but I would like to
    > be able to copy it from one from one machine to another compatible
    > machine and use it without trouble. Also I don't want to pass around
    > a collection of items and I want everything to be properly aligned.
    >
    > Naturally I think of using the struct hack and a structure something
    > like this:
    >
    > struct package {
    > message_type type; /* The type has access info */
    > size_t total_len; /* Total length of the package */
    > size_t data_len; /* Length of the data */
    > char[1] data; /* The actual data */
    > };
    >
    > Presumably at some point I will copy said data into an instance of the
    > package which can then be passed around. Also I really would like to
    > be able to use the data upon receipt without copying it out, e.g.,
    >
    > local_data = (data_type *)(pkg->data);
    >
    > It occurs to me that this might not work there is no guarantee that
    > pkg->data is correctly aligned. There are a whole bunch of other
    > potential gotchas that I can foresee. This is particularly troubling
    > to me because my PC has four cores, each with a different DS9000 chip.
    > What's worse my boss is on a "compiler of the week" binge. Every week
    > we get to test a new compiler. I really want to get this code
    > bullet-proof.
    >
    > Help! What is the right way to do this simple thing?


    I was in the middle of a somewhat longer reply when I changed
    my mind and wrote this instead.

    It's hard to give an answer without more information. Some
    of the key questions are:

    1. are the package struct's being passed through shared
    memory or through some sort of serial interface (such
    as file writing/reading or a network socket)?

    2. is it guaranteed that the raw data bytes will work on
    all the different machines used provided the base is
    suitably aligned?

    3. is 'message_type' a scalar type, or is it something
    more complicated that could be affected by changing
    between processors or compilers?

    4. I assume that all the different machines used will
    work identically on integer data types as long as
    the types have the same length and the data values
    have the same alignment from one machine to the next.
    Is this assumption correct, or might it be incorrect?
    If incorrect, then please say more.

    5. if shared memory is going to be used - is malloc()
    guaranteed to return a memory block that is suitably
    aligned in, and for, all the different execution
    environments used? If not, please say more.
    Tim Rentsch, Mar 10, 2012
    #1
    1. Advertising

  2. Tim Rentsch

    Tim Rentsch Guest

    (Richard Harter) writes:

    > On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
    > <> wrote:
    >
    >> (Richard Harter) writes:
    >>
    >>> Suppose that I have some data that is pointer free. It isn't
    >>> necessarily serialized in some format such as JSON but I would like to
    >>> be able to copy it from one from one machine to another compatible
    >>> machine and use it without trouble. Also I don't want to pass around
    >>> a collection of items and I want everything to be properly aligned.
    >>>
    >>> Naturally I think of using the struct hack and a structure something
    >>> like this:
    >>>
    >>> struct package {
    >>> message_type type; /* The type has access info */
    >>> size_t total_len; /* Total length of the package */
    >>> size_t data_len; /* Length of the data */
    >>> char[1] data; /* The actual data */
    >>> };
    >>>
    >>> Presumably at some point I will copy said data into an instance of the
    >>> package which can then be passed around. Also I really would like to
    >>> be able to use the data upon receipt without copying it out, e.g.,
    >>>
    >>> local_data = (data_type *)(pkg->data);
    >>>
    >>> It occurs to me that this might not work there is no guarantee that
    >>> pkg->data is correctly aligned. There are a whole bunch of other
    >>> potential gotchas that I can foresee. This is particularly troubling
    >>> to me because my PC has four cores, each with a different DS9000 chip.
    >>> What's worse my boss is on a "compiler of the week" binge. Every week
    >>> we get to test a new compiler. I really want to get this code
    >>> bullet-proof.
    >>>
    >>> Help! What is the right way to do this simple thing?

    >>
    >>I was in the middle of a somewhat longer reply when I changed
    >>my mind and wrote this instead.
    >>
    >>It's hard to give an answer without more information. Some
    >>of the key questions are:

    >
    > All good questions. It is quite unfair of you to ask astute
    > questions.


    I try.

    > [snip q's and a's]


    Let me try a very pragmatic suggestion, first motivating it, and
    afterwards talk about how to mitigate a potential objection.

    It seems likely that all the processors you'll be running on
    will share two characteristics (please tell me the first one
    might NOT be true of all your target platforms):

    CHAR_BIT == 8
    sizes of all "elemental" data types are powers of two

    If we assume for a moment that these two conditions are true, then
    we might try something like this:

    struct meta_stuff {
    ... whatever you want in here as long as you can get
    it out reliably, and the size isn't "too big" ...
    };

    #define DATA_OFFSET 64 /* or 128, 256, etc, as appropriate */

    union package {
    struct meta_stuff meta;
    unsigned char bytes[ DATA_OFFSET * 2 ];
    };

    ... make sure 'sizeof (struct meta_data) <= DATA_OFFSET' ...

    local_data = (data_type *) &package->bytes[ DATA_OFFSET ];

    Obviously we might want to wrap the RHS of that expression in
    a macro call or something similar. That's just a style issue,
    no need to delve into it further.

    Now, as long as we use malloc() to reserve space for packages,
    and all the elemental types have sizes that are powers of two no
    larger than 64, this approach will work. (Note: I am also
    assuming that stuff in the data area isn't some weird mapped
    memory or something similar that has some exotic (and probably
    extra-linguistic) alignment requirement. If there is then
    Heaven help you.) As long as the sizes are all powers of two,
    we don't have to care about their alignments.

    Practically speaking I expect this approach will work for all
    your current or future target platforms. But now let's suppose
    that it doesn't; suppose it works for some but not all, or
    may fail on some future platform. This possibility is easy
    to protect against:

    local_data =
    CHAR_BIT != 8 ||
    DATA_OFFSET % offsetof( struct { char c; data_type t;}, t )
    ? (data_type *) weird_case_fallback( package, ...etc... )
    : (data_type *) &package->bytes[ DATA_OFFSET ];

    Again we probably want to wrap this in a macro or some similar
    scheme. I wrote it out in full so the details would be evident.
    The 'weird_case_fallback()' function would be responsible for
    extracting out the relevant bytes appropriately, eg, copying into
    some suitably aligned structure. That shouldn't be too hard to do,
    especially since the weird_case_fallback() function can be made
    platform specific (perhaps by doing a platform check and setting a
    pointer-to-function variable during initialization).

    Obviously there needs to be some similar accommodatation in the
    code that "creates" packages. I leave the details of that to you.

    If it were me, I would propose a scheme like this one and suggest
    running with it until there was some evidence that a different
    scheme would be better. That lets us get something running now
    that probably will work for every target we're actually going to
    run on but still leaves an out for "difficult" targets. And if
    later we need to switch to a different scheme, we will then have a
    better idea of why that is, and what a better scheme might be.

    Also, one other minor point - the alignment check shown in the last
    code fragment can also be done during initialization of each
    process (or processor, perhaps, depending) to give an early warning
    of potential problems. Doing these checks, for all potential
    payload types if possible, is probably a good idea no matter what
    scheme is used to align the payload data.
    Tim Rentsch, Mar 11, 2012
    #2
    1. Advertising

  3. Tim Rentsch

    Tim Rentsch Guest

    (Richard Harter) writes:

    > On Sun, 11 Mar 2012 14:11:50 -0700, Tim Rentsch
    > <> wrote:
    >
    >> (Richard Harter) writes:
    >>
    >>> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
    >>> <> wrote:
    >>>
    >>>> (Richard Harter) writes:

    >
    > [snip lots of good stuff]
    >
    > Basically what you seem to be saying is to use an appropriate power of
    > two to force alignment. Sounds good to me, but it's not quite squeaky
    > clean bullet proof. Just a crochet.


    In that case I would want to ask for a more complete description
    of what constitutes, using your phrase, a squeaky clean bullet
    proof solution. An arbitrarily perverse implementation can
    screw up basically any scheme imaginable for this problem. What
    simplifying assumptions are you willing to grant about the
    various implementations involved, or what parts of the language
    can be excluded (one of those is not having pointers in the data
    area; what others)? If there aren't any then I think the
    problem is pretty much intractable.

    Related to that, unless I am misunderstanding something, there
    is a significant flaw in the approach you outlined in another
    message, where a struct is used to place and align the payload
    data within a larger package structure, and an offset for
    locating the data is put in the package header. The problem is,
    the alignment guarantees for the processor (and implementation)
    that _produces_ the package may not be sufficient to meet the
    alignment requirements for the processor (and implementation)
    that _receives_ the package. Whatever alignment guarantees are
    made must suffice to meet the requirements for all processors
    and all implementations simultaneously. This stipulation is
    what takes the problem out of the realm of just writing
    "portable code". Getting the alignment exactly right just in
    the package-creating process doesn't solve the problem.
    Tim Rentsch, Mar 14, 2012
    #3
  4. Tim Rentsch

    Tim Rentsch Guest

    (Richard Harter) writes:

    > On Wed, 14 Mar 2012 14:55:01 -0700, Tim Rentsch
    > <> wrote:
    >
    >> (Richard Harter) writes:
    >>
    >>> On Sun, 11 Mar 2012 14:11:50 -0700, Tim Rentsch
    >>> <> wrote:
    >>>
    >>>> (Richard Harter) writes:
    >>>>
    >>>>> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
    >>>>> <> wrote:
    >>>>>
    >>>>>> (Richard Harter) writes:
    >>>
    >>> [snip lots of good stuff]
    >>>
    >>> Basically what you seem to be saying is to use an appropriate power of
    >>> two to force alignment. Sounds good to me, but it's not quite squeaky
    >>> clean bullet proof. Just a crochet.

    >>
    >>In that case I would want to ask for a more complete description
    >>of what constitutes, using your phrase, a squeaky clean bullet
    >>proof solution. An arbitrarily perverse implementation can
    >>screw up basically any scheme imaginable for this problem. What
    >>simplifying assumptions are you willing to grant about the
    >>various implementations involved, or what parts of the language
    >>can be excluded (one of those is not having pointers in the data
    >>area; what others)? If there aren't any then I think the
    >>problem is pretty much intractable.

    >
    > Agreed that the general problem is intractable. Here is some
    > background.
    >
    > The relevant context is a message passing system written in C.
    > Messages are passed between "mini-processes" which behave as though
    > they were processes (suspend/resume logic, input ports and output
    > ports, etc). The architecture is
    >
    > Mini-processes run within process managers
    > Multiple process managers can run within a single OS thread.
    > Process managers can run in different OS threads.
    > Process managers can run on multiple networked processers.
    > Process managers can run on processors connected by the internet.
    >
    > Key facts are: Process managers are single threaded; they are
    > (supposed to be) fast. Mini-processes are light-weight, processing
    > isn't much heavier than function calls. Mini-processes don't know
    > where data comes from or where it goes.
    >
    > One of the issues in these kind of systems is what the inputs look
    > like. In shell programming the input ('the' is a problem in shell
    > programming) is in the form of a stream of characters. In Erlang it
    > is in the form of a list.
    >
    > In this system inputs and outputs are typed. Moreover inputs to a
    > mini-process DO NOT HAVE TO BE DECODED. They can be used much as
    > though they were arguments to a function. Similarly outputs do not
    > have to be encoded.
    >
    > Clearly, we can move data as is between the mini-processes running
    > under a single process manager. Similarly we should be able to move
    > data as is from one OS-thread to another in a single processor. After
    > that it gets a little fuzzy.
    >
    > Connecting process managers requires the cooperation of magic network
    > software. The magic network software is in charge of moving messages
    > from one process manager to another. It decides whether the data must
    > be serialized and deserialized using something JSON. (Actually it
    > tells the process managers to do it.) Don't ask about addressing.
    > You really don't want to know.
    >
    > So it comes down to this. We want a solid data format that is usable
    > within process managers. We would like it to be usable between
    > process managers if the connection are otherwise lightware and they
    > are consistent implementations. Otherwise we serialize. (And
    > lesnerize.) However that decision is left to matic network manager.
    >
    > All clear? If, not I have more mud.


    What I think you're saying is:

    Although the data (inputs/outputs) are typed, they are treated
    basically as raw bytes. Structure imposed on top of those raw
    bytes is done as part of the Erlang environment, ie, they are not
    C structs or whatever.

    A bundle of data ("payload") is enclosed in a C struct (or
    "struct-like", eg, a struct with variable-length last member) for
    the purpose of being able to reference such things from C.

    The data bundles are moved and/or copied and/or "remapped" (eg,
    may be in the same address space if there are process managers in
    sibling threads) from one process manager to another. This is
    always guaranteed to work intra-PM because a single PM is always
    done on a single implementation. The question is how to transmit
    data bundles inter-PM, because those may be implemented in very
    different implementations.

    Having said all that. my main take-away is that all that's
    important in terms of C issues is to get the top-level enclosing
    struct's to be "right", because what's inside the structs will
    work as long as the enclosing structs can be accessed definedly in
    the C environments in question. Is that right? If not, then I
    have missed something or misunderstood.


    >>Related to that, unless I am misunderstanding something, there
    >>is a significant flaw in the approach you outlined in another
    >>message, where a struct is used to place and align the payload
    >>data within a larger package structure, and an offset for
    >>locating the data is put in the package header. The problem is,
    >>the alignment guarantees for the processor (and implementation)
    >>that _produces_ the package may not be sufficient to meet the
    >>alignment requirements for the processor (and implementation)
    >>that _receives_ the package. Whatever alignment guarantees are
    >>made must suffice to meet the requirements for all processors
    >>and all implementations simultaneously. This stipulation is
    >>what takes the problem out of the realm of just writing
    >>"portable code". Getting the alignment exactly right just in
    >>the package-creating process doesn't solve the problem.

    >
    > No. You can play the same game at the receiving end to see
    > what the offset is at the receiving end. They had better be
    > the same. If not that is prima facie evidence that the
    > implementations are inconsistent.


    In that case I don't see any significant difference between this
    approach and the constant-but-checked 64-byte aligned approach I
    described earlier, except that, practically speaking, using a
    fixed 64-byte alignment is likely to work (ie, and not require any
    serialization) across a wider variety of implementations than the
    "sender-and-receiver-must-exactly-agree" approach. In particular,
    based on your explanation I believe the two approaches are equally
    "squeaky clean" in terms of not screwing up, but the fixed offset
    will probably require a fallback to serialization less often.
    Tim Rentsch, Mar 19, 2012
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark_Galeck
    Replies:
    12
    Views:
    424
    Mark Space
    Jul 29, 2008
  2. Dmitry S. Makovey

    is decorator the right thing to use?

    Dmitry S. Makovey, Sep 24, 2008, in forum: Python
    Replies:
    33
    Views:
    967
    Dmitry S. Makovey
    Sep 28, 2008
  3. Zam
    Replies:
    1
    Views:
    216
    Mark Schupp
    Mar 14, 2005
  4. Kirk Haines
    Replies:
    4
    Views:
    85
    Graham Nicholls
    Jul 9, 2004
  5. ImpalerCore

    Re: What is the right way to do this simple thing

    ImpalerCore, Mar 10, 2012, in forum: C Programming
    Replies:
    9
    Views:
    237
    Ian Collins
    Mar 13, 2012
Loading...

Share This Page