Struct on on x86_64 mac os x

[Tommy Grav]

I have created a binary file that saves this struct from some C code:

struct recOneData {
char label[3][84];
char constName[400][6];
double timeData[3];
long int numConst;
double AU;
double EMRAT;
long int coeffPtr[12][3];
long int DENUM;
long int libratPtr[3];
};

I try to read this file with python (ActiveState 2.6.3 on x86_64 Mac
OS X 10.6.1) using the
code below the hdrData and constNData are fine, while from the
timeData onwards there
are obvious problems. The code below works fine for a 32bit binary
read with i386 python
2.5.2. Does anyone know what the proper format of a C double and long
int is for a x86_64
binary?

def read_header(cls):
hdrData = "84s"*3
constNData = "6s"*400
timeData = "d"*3
numData = "ldd"
coeffData = "3l"*12
denumData = "l"
libPtrData = "lll"
constData = "400d"
hformat = hdrData + constNData + timeData + numData +
coeffData + denumData + libPtrData
header = struct.unpack(hdrData,cls.JPLeph.read(struct.calcsize
(hdrData)))

constN = struct.unpack(constNData,cls.JPLeph.read
(struct.calcsize(constNData)))

# Reading in the time data
cls.tstart,cls.tend,cls.tstep = struct.unpack
(timeData,cls.JPLeph.read(struct.calcsize(timeData)))

# Read in the number of constants and the values for AU and
emrat
nconst, cls.au, cls.emrat = struct.unpack
(numData,cls.JPLeph.read(struct.calcsize(numData)))

# Reading in the coefficient pointers
cls.coeff = struct.unpack(coeffData,cls.JPLeph.read
(struct.calcsize(coeffData)))
# Read in the DENUM variable and the libratPtr(3)
(cls.denum, ) = struct.unpack(denumData,cls.JPLeph.read
(struct.calcsize(denumData)))
cls.libPtr = struct.unpack(libPtrData,cls.JPLeph.read
(struct.calcsize(libPtrData)))

if cls.denum == 405:
cls.asize = 1018
elif cls.denum == 200:
cls.asize = 826
else:
raise ValueError("JPL ephem file is in unknown format %d"
% (cls.denum))

Cheers
Tommy

 

[Ulrich Eckhardt]

I have created a binary file that saves this struct from some C code:

struct recOneData {
char label[3][84];
char constName[400][6];
double timeData[3];
long int numConst;
double AU;
double EMRAT;
long int coeffPtr[12][3];
long int DENUM;
long int libratPtr[3];
};

The sizes of integers and floating-point numbers vary between different
systems. Further, floating point layout and endianess also vary between
systems. If you want to define a file format, you can't simply write a C
struct to disk.

Suggestions:
1. Use JSON or XML or any other text-based metaformat to build your format
upon.
2. Otherwise, make up your mind byte by byte how the things should look on
disk. That includes things like whether 'DENUM' is 2, 3, 5 or 42 bytes
large and which byte comes first. You can then write code to write the
structure to disk in that format, both in C and in Python. For the C part,
also take a look at <stdint.h>, which contains useful aliases for integers
with a well-defined size.

Uli

 

[Mark Dickinson]

I have created a binary file that saves this struct from some C code:

   struct recOneData {
          char label[3][84];
          char constName[400][6];
        double timeData[3];
      long int numConst;
        double AU;
        double EMRAT;
      long int coeffPtr[12][3];
      long int DENUM;
      long int libratPtr[3];
      };

I try to read this file with python (ActiveState 2.6.3 on x86_64 Mac  
OS X 10.6.1) using the
code below the hdrData and constNData are fine, while from the  
timeData onwards there
are obvious problems. The code below works fine for a 32bit binary  
read with i386 python
2.5.2. Does anyone know what the proper format of a C double and long  
int is for a x86_64
binary?

As others have said, it depends on the platform. But on OS X 10.6,
and on most 64-bit Linuxes that I've met, I'm fairly sure that
sizeof(long int) == sizeof(double) == 8. In contrast, 64-bit
Windows on the same hardware will probably have sizeof(long int) == 4.

     def read_header(cls):
         hdrData = "84s"*3
         constNData = "6s"*400

I'm confused: why is this not "400s"*6 rather than "6s"*400?
Doesn't constName[400][6] mean 6 lots of constName[400]?
Similarly for "84s"*3. This might be making a difference to
the padding.

[...]

 

[Mark Dickinson]

     def read_header(cls):
         hdrData = "84s"*3
         constNData = "6s"*400

I'm confused:  why is this not "400s"*6 rather than "6s"*400?
Doesn't constName[400][6] mean 6 lots of constName[400]?
Similarly for "84s"*3.  This might be making a difference to
the padding.

Sorry, that's wrong. It won't make any difference to the padding,
since arrays don't have padding in C: sizeof(char[3]) is 3,
and sizeof(char[3][84]) is 3*84=252.

So your problem is likely just the difference between sizeof(long)
from one platform to another.

Mark

 

[Dave Angel]

I have created a binary file that saves this struct from some C code:

struct recOneData {
char label[3][84];
char constName[400][6];
double timeData[3];
long int numConst;
double AU;
double EMRAT;
long int coeffPtr[12][3];
long int DENUM;
long int libratPtr[3];
};

I try to read this file with python (ActiveState 2.6.3 on x86_64 Mac
OS X 10.6.1) using the
code below the hdrData and constNData are fine, while from the
timeData onwards there
are obvious problems. The code below works fine for a 32bit binary
read with i386 python
2.5.2. Does anyone know what the proper format of a C double and long
int is for a x86_64
binary?

As others have said, it depends on the platform. But on OS X 10.6,
and on most 64-bit Linuxes that I've met, I'm fairly sure that
sizeof(long int) =sizeof(double) == 8. In contrast, 64-bit
Windows on the same hardware will probably have sizeof(long int) =4.

def read_header(cls):
hdrData =84s"*3
constNData =6s"*400

I'm confused: why is this not "400s"*6 rather than "6s"*400?
Doesn't constName[400][6] mean 6 lots of constName[400]?
Similarly for "84s"*3. This might be making a difference to
the padding.

[...]

I don't know much about the 64 bit Unix platforms, since 64bits was just
a dream last time I used Unix.

But I'd guess that your problem is that double will be aligned to 8
bytes on some platforms. Since your characters don't add up to a
multiple of 8, you'd have some paddding between them.


When you get to define your own file format, the old advice was to:
1) use text form if possible, where the endianness and the alignment
are non-issues.
2) If you must use binary data, directly stored, then put the
largest-aligned items at the beginning of the struct, and work down to
the smallest alignments. That way there is never any padding, except at
the end of the struct. For that, add a dummy character field of a size
appropriate to bring the size to a multiple of the largest alignment,
generally 8. You'd still have to worry about endianness, but at least
padding goes away.

Incidentally, though it doesn't affect the padding issues, I believe you
have the meaning of constName[400][6] backwards. This refers to 400
strings of 6 bytes each. In C, the definitions are in row-major order.

DaveA