struct.Struct random access

Discussion in 'Python' started by castironpi, Aug 26, 2008.

  1. castironpi

    castironpi Guest

    I'd like to seriously nominate this idea and get a considered opinion
    on it.

    struct.Struct lets you encode Python objects into structured memory.
    It accepts a format string, and optionally a buffer and offset to/from
    which to read/write the structure. What do you think of random access
    to the results? To avoid Marc's concern about meaningless anonymous
    record entries, model the constructor after 'namedtuple' type.

    (unproduced)
    >>> packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
    >>> packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
    >>> packer.unpack_from( buf, off, 'i3' )

    30
    >>> packer.unpack_from( buf, off )

    ( 10, 20, 30, 0.5, 'abc' )
    >>> packer.pack_into( buf, off, 'i1', 12 )


    Even in sequential access speed benefits, by avoiding the construction
    of n-1 objects.

    PS: If you don't have time or don't want to consider it seriously, or
    want to see more specifics, just say so.
     
    castironpi, Aug 26, 2008
    #1
    1. Advertising

  2. castironpi

    Tim Roberts Guest

    castironpi <> wrote:
    >
    >I'd like to seriously nominate this idea and get a considered opinion
    >on it.
    >
    >struct.Struct lets you encode Python objects into structured memory.
    >It accepts a format string, and optionally a buffer and offset to/from
    >which to read/write the structure. What do you think of random access
    >to the results? To avoid Marc's concern about meaningless anonymous
    >record entries, model the constructor after 'namedtuple' type.
    >
    >(unproduced)
    >>>> packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
    >>>> packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
    >>>> packer.unpack_from( buf, off, 'i3' )

    >30
    >>>> packer.unpack_from( buf, off )

    >( 10, 20, 30, 0.5, 'abc' )
    >>>> packer.pack_into( buf, off, 'i1', 12 )


    What data type would you expect "buf" to be? Your design requires it to be
    both an input and an output parameter. Today's "struct" converts into and
    out of a string, and strings are immutable. So, to continue with that
    standard, you would have to pass a string into "pack_into" and have the
    modified result returned as an output:

    buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
    buf = packer.pack_into( buf, 0, 'i1', 12 )

    In the end, I'm not sure you are really saving anything.

    >Even in sequential access speed benefits, by avoiding the construction
    >of n-1 objects.


    This is a fairly major change, because it requires a "struct.Struct" object
    to maintain state, something that the "struct" module currently does not
    do.

    In my personal opinion, this is a rather large and confusing change, for
    very little benefit.
    --
    Tim Roberts,
    Providenza & Boekelheide, Inc.
     
    Tim Roberts, Aug 28, 2008
    #2
    1. Advertising

  3. castironpi

    castironpi Guest

    On Aug 28, 1:59 am, Tim Roberts <> wrote:
    > castironpi <> wrote:
    >
    > >I'd like to seriously nominate this idea and get a considered opinion
    > >on it.

    >
    > >struct.Struct lets you encode Python objects into structured memory.
    > >It accepts a format string, and optionally a buffer and offset to/from
    > >which to read/write the structure.  What do you think of random access
    > >to the results?  To avoid Marc's concern about meaningless anonymous
    > >record entries, model the constructor after 'namedtuple' type.

    >
    > >(unproduced)
    > >>>> packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
    > >>>> packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
    > >>>> packer.unpack_from( buf, off, 'i3' )

    > >30
    > >>>> packer.unpack_from( buf, off )

    > >( 10, 20, 30, 0.5, 'abc' )
    > >>>> packer.pack_into( buf, off, 'i1', 12 )

    >
    > What data type would you expect "buf" to be?  Your design requires it to be
    > both an input and an output parameter.  Today's "struct" converts into and
    > out of a string, and strings are immutable.  So, to continue with that
    > standard, you would have to pass a string into "pack_into" and have the
    > modified result returned as an output:
    >
    >     buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
    >     buf = packer.pack_into( buf, 0, 'i1', 12 )
    >
    > In the end, I'm not sure you are really saving anything.
    >
    > >Even in sequential access speed benefits, by avoiding the construction
    > >of n-1 objects.

    >
    > This is a fairly major change, because it requires a "struct.Struct" object
    > to maintain state, something that the "struct" module currently does not
    > do.
    >
    > In my personal opinion, this is a rather large and confusing change, for
    > very little benefit.
    > --
    > Tim Roberts,
    > Providenza & Boekelheide, Inc.


    CMIIW correct me if I'm wrong, I don't think that pack_into returns a
    value the way that pack does.

    The syntax is ambiguous in the case of two-element structs: are you
    packing a new struct or assigning a field? Maybe the field name needs
    a keyword. A field name can be a third argument in unpack_from
    regardless, however. Or use the field name as keyword:
    strt.pack_into( buf, off, i1= 32 ).

    I just tried an experiment with an Mmap and PyObject_AsWriteBuffer.
    Mmaps could work but you would not be able to use arbitrary strings.
    You would have to specially allocate a ctypes buffer with a size to
    match the packed size of the structure.
     
    castironpi, Aug 28, 2008
    #3
  4. castironpi

    Tim Roberts Guest

    castironpi <> wrote:
    >
    >CMIIW correct me if I'm wrong, I don't think that pack_into returns a
    >value the way that pack does.


    Sorry, I was not aware that struct.pack_into and struct.unpack_from already
    existed (they were added in Python 2.5). I thought you were proposing them
    as new additions.

    Because of that, the rest of my post doesn't make much sense.
    --
    Tim Roberts,
    Providenza & Boekelheide, Inc.
     
    Tim Roberts, Aug 30, 2008
    #4
  5. castironpi

    castironpi Guest

    On Aug 29, 10:30 pm, Tim Roberts <> wrote:
    > castironpi <> wrote:
    >
    > >CMIIW correct me if I'm wrong, I don't think that pack_into returns a
    > >value the way that pack does.

    >
    > Sorry, I was not aware that struct.pack_into and struct.unpack_from already
    > existed (they were added in Python 2.5).  I thought you were proposing them
    > as new additions.
    >
    > Because of that, the rest of my post doesn't make much sense.
    > --
    > Tim Roberts,
    > Providenza & Boekelheide, Inc.


    I was a lot less enthusiastic too when I discovered
    'PyObject_AsWriteBuffer'. Nevertheless I think it could be
    instructive as well as a benefit. The ctypes conversion,
    cast( buffer_p + offset ), is a bit of a kludge. With the
    introduction of 'namedtuple' type, I think it falls right in line with
    the direction Python is going in.

    It might suffice to merely expose the 'offset' member of the items in
    the array of 'formatcode'. Though, the addition of the dictionary to
    lookup offsets by field name could offset the benefit somewhat.
     
    castironpi, Aug 30, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin
    Replies:
    19
    Views:
    1,153
    Tris Orendorff
    Feb 13, 2006
  2. Chris Fogelklou
    Replies:
    36
    Views:
    1,430
    Chris Fogelklou
    Apr 20, 2004
  3. globalrev
    Replies:
    4
    Views:
    797
    Gabriel Genellina
    Apr 20, 2008
  4. castironpi

    Struct class random access

    castironpi, Aug 25, 2008, in forum: Python
    Replies:
    6
    Views:
    315
    castironpi
    Aug 27, 2008
  5. VK
    Replies:
    15
    Views:
    1,283
    Dr J R Stockton
    May 2, 2010
Loading...

Share This Page