struct.calcsize problem

[Chandu]

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

 

[Larry Bates]

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

Maybe this will help:

import struct
hdrFormats = ['16s','32s', '32s','B','8s','H','8s','H','4s','H','H',
'20s','64s','64s','64s','32s','32s','64s','L','L','B',
'B','B','B','B','64s','64s']

sumofcalcsize=0
i=0
for fmt in hdrFormats:
l=struct.calcsize(fmt)
sumofcalcsize+=l
print "fmt='%s', calcsize=%i, sumofcalcsize=%i, calcsize=%i" % \
(fmt, l, sumofcalcsize, struct.calcsize(' '.join(hdrFormats[:i+1])))
i+=1


Outputs:

fmt='16s', calcsize=16, sumofcalcsize=16, calcsize=16
fmt='32s', calcsize=32, sumofcalcsize=48, calcsize=48
fmt='32s', calcsize=32, sumofcalcsize=80, calcsize=80
fmt='B', calcsize=1, sumofcalcsize=81, calcsize=81
fmt='8s', calcsize=8, sumofcalcsize=89, calcsize=89
fmt='H', calcsize=2, sumofcalcsize=91, calcsize=92 <====
fmt='8s', calcsize=8, sumofcalcsize=99, calcsize=100
fmt='H', calcsize=2, sumofcalcsize=101, calcsize=102
fmt='4s', calcsize=4, sumofcalcsize=105, calcsize=106
fmt='H', calcsize=2, sumofcalcsize=107, calcsize=108
fmt='H', calcsize=2, sumofcalcsize=109, calcsize=110
fmt='20s', calcsize=20, sumofcalcsize=129, calcsize=130
fmt='64s', calcsize=64, sumofcalcsize=193, calcsize=194
fmt='64s', calcsize=64, sumofcalcsize=257, calcsize=258
fmt='64s', calcsize=64, sumofcalcsize=321, calcsize=322
fmt='32s', calcsize=32, sumofcalcsize=353, calcsize=354
fmt='32s', calcsize=32, sumofcalcsize=385, calcsize=386
fmt='64s', calcsize=64, sumofcalcsize=449, calcsize=450
fmt='L', calcsize=4, sumofcalcsize=453, calcsize=456 <====
fmt='L', calcsize=4, sumofcalcsize=457, calcsize=460
fmt='B', calcsize=1, sumofcalcsize=458, calcsize=461
fmt='B', calcsize=1, sumofcalcsize=459, calcsize=462
fmt='B', calcsize=1, sumofcalcsize=460, calcsize=463
fmt='B', calcsize=1, sumofcalcsize=461, calcsize=464
fmt='B', calcsize=1, sumofcalcsize=462, calcsize=465
fmt='64s', calcsize=64, sumofcalcsize=526, calcsize=529
fmt='64s', calcsize=64, sumofcalcsize=590, calcsize=593

Larry Bates

 

[Bengt Richter]

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

Looks to me like you are getting default native byte order and _alignment_
and some pad bytes are getting added in. For native order with no padding,
try prefixing the format string with '='

'16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s 64s L L B B B B B 64s 64s'

If you add up the individual sizes, padding doesn't happen, apparently:

>>> map(struct.calcsize, hdrFormat.split()) [16, 32, 32, 1, 8, 2, 8, 2, 4, 2, 2, 20, 64, 64, 64, 32, 32, 64, 4, 4, 1, 1, 1, 1, 1, 64, 64]
>>> sum(map(struct.calcsize, hdrFormat.split()))

590

But default: 593
Apparently is native, with native alignment & I get the same as you: 593
Whereas native order standard (no pad) alignment is: 590
Little endian, standard alignment: 590
Big endian, standard alignment 590
Network (big endian), standard alignment: 590

I guess if you want alignment for anything non-native, you have to specify pad bytes
where you need them (with x format character).

Regards,
Bengt Richter

 

[Chandu]

Thanks for the helpful feedback. I guessed it was the alignment issue,
but could not find the exact format for changing the default. It is not
mystery any more!

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

Looks to me like you are getting default native byte order and _alignment_
and some pad bytes are getting added in. For native order with no padding,
try prefixing the format string with '='

'16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s 64s L L B B B B B 64s 64s'

If you add up the individual sizes, padding doesn't happen, apparently:

map(struct.calcsize, hdrFormat.split()) [16, 32, 32, 1, 8, 2, 8, 2, 4, 2, 2, 20, 64, 64, 64, 32, 32, 64, 4, 4, 1, 1, 1, 1, 1, 64, 64]
sum(map(struct.calcsize, hdrFormat.split()))

590

But default:593
Apparently is native, with native alignment & I get the same as you:593
Whereas native order standard (no pad) alignment is:590
Little endian, standard alignment:590
Big endian, standard alignment590
Network (big endian), standard alignment:590

I guess if you want alignment for anything non-native, you have to specify pad bytes
where you need them (with x format character).

Regards,
Bengt Richter