FAQ: how to vary the byte offset of a field of a ctypes.Structure

Discussion in 'Python' started by p.lavarre@ieee.org, May 31, 2007.

  1. Guest

    How do I vary the byte offset of a field of a ctypes.Structure?

    How do I "use the dynamic nature of Python, and (re-)define the data
    type after the required size is already known, on a case by case
    basis"?

    \\\

    For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
    for the struct:

    class Struct34(ctypes.Structure):
    _pack_ = 1
    _fields_ = [('first', 3 * ctypes.c_ubyte),
    ('second', 4 * ctypes.c_ubyte)]

    but then sometimes instead I receive the value '\x05left' + \x06right'
    for the struct:

    class Struct56(ctypes.Structure):
    _pack_ = 1
    _fields_ = [('first', 5 * ctypes.c_ubyte),
    ('second', 6 * ctypes.c_ubyte)]

    Thus in general I receive (0xFF ** 2) possible combinations of field
    lengths.

    ///

    How do I declare all those hugely many simply regular combinations as
    one CTypes.structure?

    I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
    But always the byte offsets of the subsequent fields vary when the
    byte sizes of the preceding fields vary. The byte size of the
    enclosing packed struct varies as the length of the packed bytes it
    contains.

    The errors I get as I try techniques that don't work include:

    AttributeError: '_fields_' must be a sequence of pairs
    AttributeError: _fields_ is final
    ValueError: Memory cannot be resized because this object doesn't own
    it
    TypeError: incompatible types, c_ubyte_Array_2 instance instead of
    c_ubyte_Array_1 instance

    How do I change the offset of a field of a ctypes.Structure?

    Is the answer to contribute to the next version of CTypes? Or is this
    feature already supported somehow?

    Curiously yours, thank in advance,

    http://www.google.com/search?q=ctypes variable size
    http://www.google.com/search?q=ctypes variable length
    http://www.google.com/search?q=ctypes variable offset
    http://www.google.com/search?q=ctypes "string of strings"
    http://www.google.com/search?q=ctypes "vary the byte offset"
    http://www.google.com/search?q=ctypes "change the byte offset"
    , May 31, 2007
    #1
    1. Advertising

  2. Larry Bates Guest

    wrote:
    > How do I vary the byte offset of a field of a ctypes.Structure?
    >
    > How do I "use the dynamic nature of Python, and (re-)define the data
    > type after the required size is already known, on a case by case
    > basis"?
    >
    > \\\
    >
    > For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
    > for the struct:
    >
    > class Struct34(ctypes.Structure):
    > _pack_ = 1
    > _fields_ = [('first', 3 * ctypes.c_ubyte),
    > ('second', 4 * ctypes.c_ubyte)]
    >
    > but then sometimes instead I receive the value '\x05left' + \x06right'
    > for the struct:
    >
    > class Struct56(ctypes.Structure):
    > _pack_ = 1
    > _fields_ = [('first', 5 * ctypes.c_ubyte),
    > ('second', 6 * ctypes.c_ubyte)]
    >
    > Thus in general I receive (0xFF ** 2) possible combinations of field
    > lengths.
    >
    > ///
    >
    > How do I declare all those hugely many simply regular combinations as
    > one CTypes.structure?
    >
    > I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
    > But always the byte offsets of the subsequent fields vary when the
    > byte sizes of the preceding fields vary. The byte size of the
    > enclosing packed struct varies as the length of the packed bytes it
    > contains.
    >
    > The errors I get as I try techniques that don't work include:
    >
    > AttributeError: '_fields_' must be a sequence of pairs
    > AttributeError: _fields_ is final
    > ValueError: Memory cannot be resized because this object doesn't own
    > it
    > TypeError: incompatible types, c_ubyte_Array_2 instance instead of
    > c_ubyte_Array_1 instance
    >
    > How do I change the offset of a field of a ctypes.Structure?
    >
    > Is the answer to contribute to the next version of CTypes? Or is this
    > feature already supported somehow?
    >
    > Curiously yours, thank in advance,
    >


    How about something like:

    class fooStruct(ctypes.Structure):
    _pack_ = 1
    _fields_=[]
    def __init__(self, fields):
    self._fields_=fields
    ctypes.Structure.__init__(self)

    a=fooStruct([('first', 3*ctypes.c_ubyte),
    ('second', 4*ctypes.c_ubyte)])

    print a._fields_


    -Larry
    Larry Bates, May 31, 2007
    #2
    1. Advertising

  3. schrieb:
    > How do I vary the byte offset of a field of a ctypes.Structure?
    >
    > How do I "use the dynamic nature of Python, and (re-)define the data
    > type after the required size is already known, on a case by case
    > basis"?
    >
    > \\\
    >
    > For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
    > for the struct:
    >
    > class Struct34(ctypes.Structure):
    > _pack_ = 1
    > _fields_ = [('first', 3 * ctypes.c_ubyte),
    > ('second', 4 * ctypes.c_ubyte)]
    >
    > but then sometimes instead I receive the value '\x05left' + \x06right'
    > for the struct:
    >
    > class Struct56(ctypes.Structure):
    > _pack_ = 1
    > _fields_ = [('first', 5 * ctypes.c_ubyte),
    > ('second', 6 * ctypes.c_ubyte)]
    >
    > Thus in general I receive (0xFF ** 2) possible combinations of field
    > lengths.
    >
    > ///
    >
    > How do I declare all those hugely many simply regular combinations as
    > one CTypes.structure?
    >
    > I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
    > But always the byte offsets of the subsequent fields vary when the
    > byte sizes of the preceding fields vary. The byte size of the
    > enclosing packed struct varies as the length of the packed bytes it
    > contains.


    Often it helps to ask yourself the question: How would I do this in C?

    IMO, the answer to this question, applied to your problem, would be:
    *Not* by using a structure. A structure is fine if the definition is fixed,
    or at most has *one* variable sized field at the very end. Nothing
    is true for your problem.

    Thomas
    Thomas Heller, May 31, 2007
    #3
  4. Guest

    ctypes.sizeof(a) is still zero, as if ctypes.Structure.__init__
    fetches a.__class__._fields_ rather than a._fields_
    , May 31, 2007
    #4
  5. Guest

    """ Thomas,

    Ouch ouch I must have misunderstood what you meant by "use the dynamic
    nature of Python, and (re-)define the data type after the required
    size is already known, on a case by case basis".

    Do you have an example of what you meant? I searched but did not find.
    Are those your words?

    Yes, to declare strings of strings in Python would be to express a
    familiar idea that I don't know how to say well in C.

    These are standard structs that I exchange with third parties, e.g.,
    me editing Class files read later by a Java interpreter, so I can't
    avoid having to deal with this complexity designed into them. For
    discussion I've simplified the problem: back in real life I have a few
    dozen variations of structs like this to deal with.

    The following Python mostly works, but only at the cost of rewriting
    the small part of CTypes that I need for this app.
    """

    import binascii
    import struct

    class Struct:

    def __init__(self, declarations = []):

    """Add initial values to self and list the names declared."""

    names = []
    for (initial, name) in declarations:
    names += [name]
    python = ' '.join(['self.' + name, '=',
    'initial'])
    exec python
    self._names_ = names

    def _fields_(self):

    """List the fields."""

    fields = []
    for name in self._names_:
    python = 'self.' + name
    fields += [eval(python)]
    return fields

    def _pack_(self):

    """Pack a copy of the fields."""

    packs = ''
    for field in self._fields_():
    packs += field._pack_()
    return packs

    def _size_(self, bytes = None):

    """Count the bytes of the fields."""

    return len(self._pack_())

    def _unpack_(self, bytes):

    """Count the bytes of a copy of the fields."""

    offset = 0
    for field in self._fields_():
    size = field._size_(bytes[offset:])
    field._unpack_(bytes[offset:][:size])
    offset += size
    if offset != len(bytes):
    why = ('_unpack_ requires a string argument'
    'of length %d' % offset)
    raise struct.error(why)

    class Field(Struct):

    """Contain one value."""

    def __init__(self, value = 0):
    self._value_ = value

    def _pack_(self):
    raise AttributeError('abstract')

    def _unpack_(self, bytes):
    raise AttributeError('abstract')

    class Byte(Field):

    """Contain one byte."""

    def _pack_(self):
    return struct.pack('B', self._value_)

    def _unpack_(self, bytes):
    self._value_ = struct.unpack('B', bytes)[0]

    class ByteArray(Field):

    """Contain the same nonnegative number of bytes always."""

    def _pack_(self):
    return self._value_

    def _unpack_(self, bytes):
    if len(bytes) == len(self._value_):
    self._value_ = bytes
    else:

    why = ('_unpack_ requires a string argument'
    'of length %d' % len(self._value_))
    raise struct.error(why)

    class Symbol(Struct):

    """Contain a count of bytes."""

    def __init__(self, value = ''):
    Struct.__init__(self, [
    (Byte(), 'length'),
    (ByteArray(value), 'bytes')])
    self._pack_()


    def _size_(self, bytes = None):
    return ord(bytes[0])

    def _pack_(self):
    self.length = Byte(self.length._size_() +
    self.bytes._size_())
    return Struct._pack_(self)

    class TwoSymbols(Struct):

    """Contain two Symbols."""

    def __init__(self, values = ['', '']):
    Struct.__init__(self, [
    (Symbol(values[0]), 'first'),
    (Symbol(values[1]), 'second')])

    zz = Symbol()
    print binascii.hexlify(zz._pack_()).upper()
    # 01

    zz = Symbol()
    zz.bytes = ByteArray('left')
    zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
    # '\x05left'

    zz = Symbol()
    zz.bytes = ByteArray('right')
    zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
    # '\x06right'

    zz = TwoSymbols()
    zz.first.bytes = ByteArray('hi')
    zz.second.bytes = ByteArray('bye')
    zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
    # '\x03hi\x04bye'

    print zz._size_()
    # 7

    yy = '''
    def yyFunc(self):
    return [self.length, self.bytes]
    '''
    exec yy
    Symbol._fields_ = yyFunc
    zz = TwoSymbols(['alef', 'bet'])
    zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
    , May 31, 2007
    #5
  6. Guest

    I see that changing self._fields_ doesn't change ctypes.sizeof(self).

    I guess ctypes.Structure.__init__(self) fetches
    self.__class__._fields_ not self._fields_.
    , Jun 1, 2007
    #6
  7. Guest

    > Often it helps to ask yourself the question: How would I do this in C? ...
    >
    > *Not* by using a structure. A structure is fine if the definition is fixed,
    > or at most has *one* variable sized field at the very end.


    http://docs.python.org/lib/module-pickle.html
    might be near where I'll find concise Python ways of pickling and
    unpickling the (0xFF ** N) possible ways of packing N strings of byte
    lengths of 0..xFE together ...

    I notice by now I need what ctypes does well often enough that I have
    to make my corrupt copy of a subset of ctypes coexist with ctypes per
    se. So I have two different BYTE definitions. One is the
    ctypes.c_ubyte. The other is my corrupt copy. And likewise for a big-
    endian BIG_WORD and so on. So I end up naming the one BYTE and the
    other BITE, the one BIG_WORD and the other BIG_WURD, yuck.
    , Jun 2, 2007
    #7
  8. Guest

    > > http://docs.python.org/lib/module-pickle.html
    > > ... concise Python ways of pickling and unpickling
    > > the (0xFF ** N) possible ways of
    > > packing N strings of byte lengths of 0..xFE together ...


    Aye, looks like an exercise left open for the student to complete:

    >>> pickle.dumps("")

    "S''\np0\n."
    >>>
    >>> pickle.dumps("abc")

    "S'abc'\np0\n."
    >>>
    >>> pickle.loads(pickle.dumps("abc"))

    'abc'
    >>>
    >>> pickle.dumps(ctypes.c_ubyte(0))

    ....
    TypeError: abstract class
    >>>
    , Jun 4, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ryan Moore

    Vary by custom

    Ryan Moore, Jan 27, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    4,691
    GeekyMonkey
    Aug 24, 2006
  2. Lance Riedel

    Translated Offset to Source Offset

    Lance Riedel, Oct 14, 2003, in forum: XML
    Replies:
    2
    Views:
    485
    Patrick TJ McPhee
    Oct 15, 2003
  3. HeroOfSpielburg
    Replies:
    1
    Views:
    373
    Alf P. Steinbach
    Aug 6, 2003
  4. Carl Banks

    ctypes pointer from offset into array?

    Carl Banks, Dec 5, 2009, in forum: Python
    Replies:
    2
    Views:
    2,069
    sturlamolden
    Dec 5, 2009
  5. Roy Smith
    Replies:
    4
    Views:
    240
    Roy Smith
    Jan 27, 2013
Loading...

Share This Page