struct.Struct random access

[castironpi]

I'd like to seriously nominate this idea and get a considered opinion
on it.

struct.Struct lets you encode Python objects into structured memory.
It accepts a format string, and optionally a buffer and offset to/from
which to read/write the structure. What do you think of random access
to the results? To avoid Marc's concern about meaningless anonymous
record entries, model the constructor after 'namedtuple' type.

(unproduced)
Even in sequential access speed benefits, by avoiding the construction
of n-1 objects.

PS: If you don't have time or don't want to consider it seriously, or
want to see more specifics, just say so.

 

[Tim Roberts]

I'd like to seriously nominate this idea and get a considered opinion
on it.

struct.Struct lets you encode Python objects into structured memory.
It accepts a format string, and optionally a buffer and offset to/from
which to read/write the structure. What do you think of random access
to the results? To avoid Marc's concern about meaningless anonymous
record entries, model the constructor after 'namedtuple' type.

(unproduced)

What data type would you expect "buf" to be? Your design requires it to be
both an input and an output parameter. Today's "struct" converts into and
out of a string, and strings are immutable. So, to continue with that
standard, you would have to pass a string into "pack_into" and have the
modified result returned as an output:

buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
buf = packer.pack_into( buf, 0, 'i1', 12 )

In the end, I'm not sure you are really saving anything.

Even in sequential access speed benefits, by avoiding the construction
of n-1 objects.

This is a fairly major change, because it requires a "struct.Struct" object
to maintain state, something that the "struct" module currently does not
do.

In my personal opinion, this is a rather large and confusing change, for
very little benefit.

 

[castironpi]

What data type would you expect "buf" to be?  Your design requires it to be
both an input and an output parameter.  Today's "struct" converts into and
out of a string, and strings are immutable.  So, to continue with that
standard, you would have to pass a string into "pack_into" and have the
modified result returned as an output:

    buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
    buf = packer.pack_into( buf, 0, 'i1', 12 )

In the end, I'm not sure you are really saving anything.


This is a fairly major change, because it requires a "struct.Struct" object
to maintain state, something that the "struct" module currently does not
do.

In my personal opinion, this is a rather large and confusing change, for
very little benefit.

CMIIW correct me if I'm wrong, I don't think that pack_into returns a
value the way that pack does.

The syntax is ambiguous in the case of two-element structs: are you
packing a new struct or assigning a field? Maybe the field name needs
a keyword. A field name can be a third argument in unpack_from
regardless, however. Or use the field name as keyword:
strt.pack_into( buf, off, i1= 32 ).

I just tried an experiment with an Mmap and PyObject_AsWriteBuffer.
Mmaps could work but you would not be able to use arbitrary strings.
You would have to specially allocate a ctypes buffer with a size to
match the packed size of the structure.

 

[Tim Roberts]

CMIIW correct me if I'm wrong, I don't think that pack_into returns a
value the way that pack does.

Sorry, I was not aware that struct.pack_into and struct.unpack_from already
existed (they were added in Python 2.5). I thought you were proposing them
as new additions.

Because of that, the rest of my post doesn't make much sense.

 

[castironpi]

Sorry, I was not aware that struct.pack_into and struct.unpack_from already
existed (they were added in Python 2.5).  I thought you were proposing them
as new additions.

Because of that, the rest of my post doesn't make much sense.

I was a lot less enthusiastic too when I discovered
'PyObject_AsWriteBuffer'. Nevertheless I think it could be
instructive as well as a benefit. The ctypes conversion,
cast( buffer_p + offset ), is a bit of a kludge. With the
introduction of 'namedtuple' type, I think it falls right in line with
the direction Python is going in.

It might suffice to merely expose the 'offset' member of the items in
the array of 'formatcode'. Though, the addition of the dictionary to
lookup offsets by field name could offset the benefit somewhat.