FAQ: how to vary the byte offset of a field of a ctypes.Structure

[p.lavarre]

How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.

The errors I get as I try techniques that don't work include:

AttributeError: '_fields_' must be a sequence of pairs
AttributeError: _fields_ is final
ValueError: Memory cannot be resized because this object doesn't own
it
TypeError: incompatible types, c_ubyte_Array_2 instance instead of
c_ubyte_Array_1 instance

How do I change the offset of a field of a ctypes.Structure?

Is the answer to contribute to the next version of CTypes? Or is this
feature already supported somehow?

Curiously yours, thank in advance,

http://www.google.com/search?q=ctypes+variable+size
http://www.google.com/search?q=ctypes+variable+length
http://www.google.com/search?q=ctypes+variable+offset
http://www.google.com/search?q=ctypes+"string+of+strings"
http://www.google.com/search?q=ctypes+"vary+the+byte+offset"
http://www.google.com/search?q=ctypes+"change+the+byte+offset"

 

[Larry Bates]

How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.

The errors I get as I try techniques that don't work include:

AttributeError: '_fields_' must be a sequence of pairs
AttributeError: _fields_ is final
ValueError: Memory cannot be resized because this object doesn't own
it
TypeError: incompatible types, c_ubyte_Array_2 instance instead of
c_ubyte_Array_1 instance

How do I change the offset of a field of a ctypes.Structure?

Is the answer to contribute to the next version of CTypes? Or is this
feature already supported somehow?

Curiously yours, thank in advance,

How about something like:

class fooStruct(ctypes.Structure):
_pack_ = 1
_fields_=[]
def __init__(self, fields):
self._fields_=fields
ctypes.Structure.__init__(self)

a=fooStruct([('first', 3*ctypes.c_ubyte),
('second', 4*ctypes.c_ubyte)])

print a._fields_


-Larry

 

[Thomas Heller]

How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.

Often it helps to ask yourself the question: How would I do this in C?

IMO, the answer to this question, applied to your problem, would be:
*Not* by using a structure. A structure is fine if the definition is fixed,
or at most has *one* variable sized field at the very end. Nothing
is true for your problem.

Thomas

 

[p.lavarre]

ctypes.sizeof(a) is still zero, as if ctypes.Structure.__init__
fetches a.__class__._fields_ rather than a._fields_

 

[p.lavarre]

""" Thomas,

Ouch ouch I must have misunderstood what you meant by "use the dynamic
nature of Python, and (re-)define the data type after the required
size is already known, on a case by case basis".

Do you have an example of what you meant? I searched but did not find.
Are those your words?

Yes, to declare strings of strings in Python would be to express a
familiar idea that I don't know how to say well in C.

These are standard structs that I exchange with third parties, e.g.,
me editing Class files read later by a Java interpreter, so I can't
avoid having to deal with this complexity designed into them. For
discussion I've simplified the problem: back in real life I have a few
dozen variations of structs like this to deal with.

The following Python mostly works, but only at the cost of rewriting
the small part of CTypes that I need for this app.
"""

import binascii
import struct

class Struct:

def __init__(self, declarations = []):

"""Add initial values to self and list the names declared."""

names = []
for (initial, name) in declarations:
names += [name]
python = ' '.join(['self.' + name, '=',
'initial'])
exec python
self._names_ = names

def _fields_(self):

"""List the fields."""

fields = []
for name in self._names_:
python = 'self.' + name
fields += [eval(python)]
return fields

def _pack_(self):

"""Pack a copy of the fields."""

packs = ''
for field in self._fields_():
packs += field._pack_()
return packs

def _size_(self, bytes = None):

"""Count the bytes of the fields."""

return len(self._pack_())

def _unpack_(self, bytes):

"""Count the bytes of a copy of the fields."""

offset = 0
for field in self._fields_():
size = field._size_(bytes[offset:])
field._unpack_(bytes[offset:][:size])
offset += size
if offset != len(bytes):
why = ('_unpack_ requires a string argument'
'of length %d' % offset)
raise struct.error(why)

class Field(Struct):

"""Contain one value."""

def __init__(self, value = 0):
self._value_ = value

def _pack_(self):
raise AttributeError('abstract')

def _unpack_(self, bytes):
raise AttributeError('abstract')

class Byte(Field):

"""Contain one byte."""

def _pack_(self):
return struct.pack('B', self._value_)

def _unpack_(self, bytes):
self._value_ = struct.unpack('B', bytes)[0]

class ByteArray(Field):

"""Contain the same nonnegative number of bytes always."""

def _pack_(self):
return self._value_

def _unpack_(self, bytes):
if len(bytes) == len(self._value_):
self._value_ = bytes
else:

why = ('_unpack_ requires a string argument'
'of length %d' % len(self._value_))
raise struct.error(why)

class Symbol(Struct):

"""Contain a count of bytes."""

def __init__(self, value = ''):
Struct.__init__(self, [
(Byte(), 'length'),
(ByteArray(value), 'bytes')])
self._pack_()


def _size_(self, bytes = None):
return ord(bytes[0])

def _pack_(self):
self.length = Byte(self.length._size_() +
self.bytes._size_())
return Struct._pack_(self)

class TwoSymbols(Struct):

"""Contain two Symbols."""

def __init__(self, values = ['', '']):
Struct.__init__(self, [
(Symbol(values[0]), 'first'),
(Symbol(values[1]), 'second')])

zz = Symbol()
print binascii.hexlify(zz._pack_()).upper()
# 01

zz = Symbol()
zz.bytes = ByteArray('left')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x05left'

zz = Symbol()
zz.bytes = ByteArray('right')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x06right'

zz = TwoSymbols()
zz.first.bytes = ByteArray('hi')
zz.second.bytes = ByteArray('bye')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x03hi\x04bye'

print zz._size_()
# 7

yy = '''
def yyFunc(self):
return [self.length, self.bytes]
'''
exec yy
Symbol._fields_ = yyFunc
zz = TwoSymbols(['alef', 'bet'])
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())

 

[p.lavarre]

I see that changing self._fields_ doesn't change ctypes.sizeof(self).

I guess ctypes.Structure.__init__(self) fetches
self.__class__._fields_ not self._fields_.

 

[p.lavarre]

Often it helps to ask yourself the question: How would I do this in C? ...

*Not* by using a structure. A structure is fine if the definition is fixed,
or at most has *one* variable sized field at the very end.

http://docs.python.org/lib/module-pickle.html
might be near where I'll find concise Python ways of pickling and
unpickling the (0xFF ** N) possible ways of packing N strings of byte
lengths of 0..xFE together ...

I notice by now I need what ctypes does well often enough that I have
to make my corrupt copy of a subset of ctypes coexist with ctypes per
se. So I have two different BYTE definitions. One is the
ctypes.c_ubyte. The other is my corrupt copy. And likewise for a big-
endian BIG_WORD and so on. So I end up naming the one BYTE and the
other BITE, the one BIG_WORD and the other BIG_WURD, yuck.

 

[p.lavarre]

http://docs.python.org/lib/module-pickle.html
Aye, looks like an exercise left open for the student to complete:
....
TypeError: abstract class