Struct on on x86_64 mac os x

Discussion in 'Python' started by Tommy Grav, Oct 20, 2009.

  1. Tommy Grav

    Tommy Grav Guest

    I have created a binary file that saves this struct from some C code:

    struct recOneData {
    char label[3][84];
    char constName[400][6];
    double timeData[3];
    long int numConst;
    double AU;
    double EMRAT;
    long int coeffPtr[12][3];
    long int DENUM;
    long int libratPtr[3];
    };

    I try to read this file with python (ActiveState 2.6.3 on x86_64 Mac
    OS X 10.6.1) using the
    code below the hdrData and constNData are fine, while from the
    timeData onwards there
    are obvious problems. The code below works fine for a 32bit binary
    read with i386 python
    2.5.2. Does anyone know what the proper format of a C double and long
    int is for a x86_64
    binary?

    def read_header(cls):
    hdrData = "84s"*3
    constNData = "6s"*400
    timeData = "d"*3
    numData = "ldd"
    coeffData = "3l"*12
    denumData = "l"
    libPtrData = "lll"
    constData = "400d"
    hformat = hdrData + constNData + timeData + numData +
    coeffData + denumData + libPtrData
    header = struct.unpack(hdrData,cls.JPLeph.read(struct.calcsize
    (hdrData)))

    constN = struct.unpack(constNData,cls.JPLeph.read
    (struct.calcsize(constNData)))

    # Reading in the time data
    cls.tstart,cls.tend,cls.tstep = struct.unpack
    (timeData,cls.JPLeph.read(struct.calcsize(timeData)))

    # Read in the number of constants and the values for AU and
    emrat
    nconst, cls.au, cls.emrat = struct.unpack
    (numData,cls.JPLeph.read(struct.calcsize(numData)))

    # Reading in the coefficient pointers
    cls.coeff = struct.unpack(coeffData,cls.JPLeph.read
    (struct.calcsize(coeffData)))
    # Read in the DENUM variable and the libratPtr(3)
    (cls.denum, ) = struct.unpack(denumData,cls.JPLeph.read
    (struct.calcsize(denumData)))
    cls.libPtr = struct.unpack(libPtrData,cls.JPLeph.read
    (struct.calcsize(libPtrData)))

    if cls.denum == 405:
    cls.asize = 1018
    elif cls.denum == 200:
    cls.asize = 826
    else:
    raise ValueError("JPL ephem file is in unknown format %d"
    % (cls.denum))

    Cheers
    Tommy
     
    Tommy Grav, Oct 20, 2009
    #1
    1. Advertising

  2. Tommy Grav wrote:
    > I have created a binary file that saves this struct from some C code:
    >
    > struct recOneData {
    > char label[3][84];
    > char constName[400][6];
    > double timeData[3];
    > long int numConst;
    > double AU;
    > double EMRAT;
    > long int coeffPtr[12][3];
    > long int DENUM;
    > long int libratPtr[3];
    > };


    The sizes of integers and floating-point numbers vary between different
    systems. Further, floating point layout and endianess also vary between
    systems. If you want to define a file format, you can't simply write a C
    struct to disk.

    Suggestions:
    1. Use JSON or XML or any other text-based metaformat to build your format
    upon.
    2. Otherwise, make up your mind byte by byte how the things should look on
    disk. That includes things like whether 'DENUM' is 2, 3, 5 or 42 bytes
    large and which byte comes first. You can then write code to write the
    structure to disk in that format, both in C and in Python. For the C part,
    also take a look at <stdint.h>, which contains useful aliases for integers
    with a well-defined size.

    Uli

    --
    Sator Laser GmbH
    Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
     
    Ulrich Eckhardt, Oct 21, 2009
    #2
    1. Advertising

  3. On Oct 20, 10:51 pm, Tommy Grav <> wrote:
    > I have created a binary file that saves this struct from some C code:
    >
    >    struct recOneData {
    >           char label[3][84];
    >           char constName[400][6];
    >         double timeData[3];
    >       long int numConst;
    >         double AU;
    >         double EMRAT;
    >       long int coeffPtr[12][3];
    >       long int DENUM;
    >       long int libratPtr[3];
    >       };
    >
    > I try to read this file with python (ActiveState 2.6.3 on x86_64 Mac  
    > OS X 10.6.1) using the
    > code below the hdrData and constNData are fine, while from the  
    > timeData onwards there
    > are obvious problems. The code below works fine for a 32bit binary  
    > read with i386 python
    > 2.5.2. Does anyone know what the proper format of a C double and long  
    > int is for a x86_64
    > binary?


    As others have said, it depends on the platform. But on OS X 10.6,
    and on most 64-bit Linuxes that I've met, I'm fairly sure that
    sizeof(long int) == sizeof(double) == 8. In contrast, 64-bit
    Windows on the same hardware will probably have sizeof(long int) == 4.

    >
    >      def read_header(cls):
    >          hdrData = "84s"*3
    >          constNData = "6s"*400


    I'm confused: why is this not "400s"*6 rather than "6s"*400?
    Doesn't constName[400][6] mean 6 lots of constName[400]?
    Similarly for "84s"*3. This might be making a difference to
    the padding.

    [...]

    --
    Mark
     
    Mark Dickinson, Oct 21, 2009
    #3
  4. On Oct 21, 9:18 am, Mark Dickinson <> wrote:
    > On Oct 20, 10:51 pm, Tommy Grav <> wrote:
    >
    > >      def read_header(cls):
    > >          hdrData = "84s"*3
    > >          constNData = "6s"*400

    >
    > I'm confused:  why is this not "400s"*6 rather than "6s"*400?
    > Doesn't constName[400][6] mean 6 lots of constName[400]?
    > Similarly for "84s"*3.  This might be making a difference to
    > the padding.


    Sorry, that's wrong. It won't make any difference to the padding,
    since arrays don't have padding in C: sizeof(char[3]) is 3,
    and sizeof(char[3][84]) is 3*84=252.

    So your problem is likely just the difference between sizeof(long)
    from one platform to another.

    Mark
     
    Mark Dickinson, Oct 21, 2009
    #4
  5. Tommy Grav

    Dave Angel Guest

    Mark Dickinson wrote:
    > On Oct 20, 10:51 pm, Tommy Grav <> wrote:
    >
    >> I have created a binary file that saves this struct from some C code:
    >>
    >> struct recOneData {
    >> char label[3][84];
    >> char constName[400][6];
    >> double timeData[3];
    >> long int numConst;
    >> double AU;
    >> double EMRAT;
    >> long int coeffPtr[12][3];
    >> long int DENUM;
    >> long int libratPtr[3];
    >> };
    >>
    >> I try to read this file with python (ActiveState 2.6.3 on x86_64 Mac
    >> OS X 10.6.1) using the
    >> code below the hdrData and constNData are fine, while from the
    >> timeData onwards there
    >> are obvious problems. The code below works fine for a 32bit binary
    >> read with i386 python
    >> 2.5.2. Does anyone know what the proper format of a C double and long
    >> int is for a x86_64
    >> binary?
    >>

    >
    > As others have said, it depends on the platform. But on OS X 10.6,
    > and on most 64-bit Linuxes that I've met, I'm fairly sure that
    > sizeof(long int) =sizeof(double) == 8. In contrast, 64-bit
    > Windows on the same hardware will probably have sizeof(long int) =4.
    >
    >
    >> def read_header(cls):
    >> hdrData =84s"*3
    >> constNData =6s"*400
    >>

    >
    > I'm confused: why is this not "400s"*6 rather than "6s"*400?
    > Doesn't constName[400][6] mean 6 lots of constName[400]?
    > Similarly for "84s"*3. This might be making a difference to
    > the padding.
    >
    > [...]
    >
    > --
    > Mark
    >
    >

    I don't know much about the 64 bit Unix platforms, since 64bits was just
    a dream last time I used Unix.

    But I'd guess that your problem is that double will be aligned to 8
    bytes on some platforms. Since your characters don't add up to a
    multiple of 8, you'd have some paddding between them.


    When you get to define your own file format, the old advice was to:
    1) use text form if possible, where the endianness and the alignment
    are non-issues.
    2) If you must use binary data, directly stored, then put the
    largest-aligned items at the beginning of the struct, and work down to
    the smallest alignments. That way there is never any padding, except at
    the end of the struct. For that, add a dummy character field of a size
    appropriate to bring the size to a multiple of the largest alignment,
    generally 8. You'd still have to worry about endianness, but at least
    padding goes away.

    Incidentally, though it doesn't affect the padding issues, I believe you
    have the meaning of constName[400][6] backwards. This refers to 400
    strings of 6 bytes each. In C, the definitions are in row-major order.

    DaveA
     
    Dave Angel, Oct 21, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris Fogelklou
    Replies:
    36
    Views:
    1,423
    Chris Fogelklou
    Apr 20, 2004
  2. Neal Becker

    Compile fails on x86_64

    Neal Becker, Sep 30, 2005, in forum: Python
    Replies:
    0
    Views:
    293
    Neal Becker
    Sep 30, 2005
  3. Neal Becker

    compile fails on x86_64 (more)

    Neal Becker, Sep 30, 2005, in forum: Python
    Replies:
    0
    Views:
    274
    Neal Becker
    Sep 30, 2005
  4. David
    Replies:
    4
    Views:
    1,833
    Andrew Thompson
    Jan 10, 2007
  5. chris grebeldinger

    2.6.4 Mac x86_64 ?

    chris grebeldinger, Nov 13, 2009, in forum: Python
    Replies:
    2
    Views:
    319
    chris grebeldinger
    Nov 17, 2009
Loading...

Share This Page