issue with struct.unpack

Discussion in 'Python' started by 9bizy, Aug 25, 2012.

  1. 9bizy

    9bizy Guest

    I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?

    I checked the python documentation for struct and I can seen to find any argument for this.

    I have data = struct.unpack('char',data) but I still get errors
    9bizy, Aug 25, 2012
    #1
    1. Advertising

  2. On 25/08/2012 19:34, 9bizy wrote:
    > I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?
    >
    > I checked the python documentation for struct and I can seen to find any argument for this.
    >
    > I have data = struct.unpack('char',data) but I still get errors
    >


    We have two options here. Either

    a) People reading your original request go on a mind reading course or
    similar in an attempt to find out what the errors are, though I'm
    confused as to how you get errors from one line of code.

    Or

    b) Provide the smallest sample of code that allows the problem to be
    reproduced together with the complete traceback so we can see exactly
    what happened.

    --
    Cheers.

    Mark Lawrence.
    Mark Lawrence, Aug 25, 2012
    #2
    1. Advertising

  3. 9bizy

    MRAB Guest

    On 25/08/2012 19:34, 9bizy wrote:
    > I am trying to unpack values from sensor data I am retrieving through
    > a serial cable, but I get errors while using struct.unpack, how can I
    > use struct.unpack to unload the data in a readable format?
    >
    > I checked the python documentation for struct and I can seen to find
    > any argument for this.
    >
    > I have data = struct.unpack('char',data) but I still get errors
    >

    The format strings are described here for Python 3:

    http://docs.python.org/3.2/library/struct.html

    and here for Python 2:

    http://docs.python.org/2.7/library/struct.html
    MRAB, Aug 25, 2012
    #3
  4. On Sat, 25 Aug 2012 11:34:39 -0700 (PDT), 9bizy
    <> declaimed the following in
    gmane.comp.python.general:

    > I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?
    >
    > I checked the python documentation for struct and I can seen to find any argument for this.
    >

    Pardon "... I can seen to find..."?

    > I have data = struct.unpack('char',data) but I still get errors


    No surprise... You've asked for a single-character, a short integer,
    and two unknown field types.

    What about

    """
    s char[] string
    """

    and

    """
    For the "s" format character, the count is interpreted as the size of
    the string, not a repeat count like for the other format characters; for
    example, '10s' means a single 10-byte string, while '10c' means 10
    characters. For packing, the string is truncated or padded with null
    bytes as appropriate to make it fit. For unpacking, the resulting string
    always has exactly the specified number of bytes. As a special case,
    '0s' means a single, empty string (while '0c' means 0 characters).
    """

    (both from the 2.5 help file).

    The struct module relies upon the user knowing the format of the data.
    If your problem is that you have some null-terminated string data in a
    variable width field, you will have to locate the position of the null
    FIRST, and specify the appropriate "count" for the s format.
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Aug 26, 2012
    #4
  5. On 26.08.2012 01:31, Dennis Lee Bieber wrote:
    > The struct module relies upon the user knowing the format of the data.
    > If your problem is that you have some null-terminated string data in a
    > variable width field, you will have to locate the position of the null
    > FIRST, and specify the appropriate "count" for the s format.


    This gave me the idea of an Enhancement of the Struct class with an
    additional format character (perhaps 'n') which corresponds to a
    null-terminated string:

    -----
    # -*- coding: utf-8 -*-

    import struct

    class Nstr(object):
    def __init__(self, ncount):
    self.ncount = ncount

    def unpack(self, s):
    s = s.split('\0')
    return s[:self.ncount], '\0'.join(s[self.ncount:])

    def pack(self, *s):
    if len(s)!=self.ncount:
    raise ValueError
    for st in s:
    if '\0' in st:
    raise ValueError
    return '\0'.join(s)+'\0'

    def __repr__(self):
    return 'Nstr('+repr(self.ncount)+')'

    class NStruct(object):
    def __init__(self, format):
    self.format = format
    if format[0] in '!=<>@':
    self.endianness = format[0]
    format = format[1:]
    else:
    self.endianness = ''
    self.chunks = []
    while len(format)>0:
    j = format.find('n')
    if j > -1:
    k = j-1
    while format[k].isdigit():
    k-=1
    chunkformat, ncount, format = format[:k+1],\
    format[k+1:j], format[j+1:]
    ncount = 1 if len(ncount)==0 else int(ncount)
    else:
    chunkformat, ncount, format = format, '', ''
    ncount = 0
    stru = struct.Struct(self.endianness+chunkformat)
    l = len(stru.unpack("0"*stru.size))
    self.chunks.append((stru, l))
    if ncount > 0:
    self.chunks.append((Nstr(ncount), ncount))

    def unpack(self, data):
    res = []
    for sth, n in self.chunks:
    if isinstance(sth, struct.Struct):
    chunk, data = data[:sth.size], data[sth.size:]
    res.extend(sth.unpack(chunk))
    elif isinstance(sth, Nstr):
    chunk, data = sth.unpack(data)
    res.extend(chunk)
    return res

    def pack(self, *data):
    res = []
    for sth, n in self.chunks:
    chunk, data = data[:n], data[n:]
    res.append(sth.pack(*chunk))
    return ''.join(res)

    def __repr__(self):
    return 'NStruct('+repr(self.format)+')'

    if __name__=="__main__":
    a = NStruct('h b 2n 2h')
    print repr(a)
    d = 'asdblah blah\0haha\0asdf'
    r = a.unpack(d)
    assert r == [29537, 100, 'blah blah', 'haha', 29537, 26212]
    print repr(d), repr(r)
    dd = a.pack(*r)
    print r, repr(dd)
    assert dd == d

    -----

    beware of bugs in the above code, i haven't testet it much yet.

    Alex
    Alexander Blinne, Aug 26, 2012
    #5
  6. 9bizy

    9bizy Guest

    This is what I have to reproduce the challenge I am having below:


    import csv
    import struct


    data = []

    for Node in csv.reader(file('s_data.xls')):
    data.append(list((file('s_data.xls'))))


    data = struct.unpack('!B4HH', data)
    print "s_data.csv: ", data

    I tries so many format for the struct.unpack but I got this errors:

    Traceback (most recent call last):

    data = struct.unpack('!B4HH', data)
    struct.error: unpack requires a string argument of length 11

    On Saturday, 25 August 2012 19:34:39 UTC+1, 9bizy wrote:
    > I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?
    >
    >
    >
    > I checked the python documentation for struct and I can seen to find any argument for this.
    >
    >
    >
    > I have data = struct.unpack('char',data) but I still get errors
    9bizy, Aug 28, 2012
    #6
  7. 9bizy

    9bizy Guest

    On Saturday, 25 August 2012 20:16:54 UTC+1, MRAB wrote:
    > On 25/08/2012 19:34, 9bizy wrote:
    >
    > > I am trying to unpack values from sensor data I am retrieving through

    >
    > > a serial cable, but I get errors while using struct.unpack, how can I

    >
    > > use struct.unpack to unload the data in a readable format?

    >
    > >

    >
    > > I checked the python documentation for struct and I can seen to find

    >
    > > any argument for this.

    >
    > >

    >
    > > I have data = struct.unpack('char',data) but I still get errors

    >
    > >

    >
    > The format strings are described here for Python 3:
    >
    >
    >
    > http://docs.python.org/3.2/library/struct.html
    >
    >
    >
    > and here for Python 2:
    >
    >
    >
    > http://docs.python.org/2.7/library/struct.html


    I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.
    9bizy, Aug 28, 2012
    #7
  8. 9bizy

    9bizy Guest

    On Saturday, 25 August 2012 20:16:54 UTC+1, MRAB wrote:
    > On 25/08/2012 19:34, 9bizy wrote:
    >
    > > I am trying to unpack values from sensor data I am retrieving through

    >
    > > a serial cable, but I get errors while using struct.unpack, how can I

    >
    > > use struct.unpack to unload the data in a readable format?

    >
    > >

    >
    > > I checked the python documentation for struct and I can seen to find

    >
    > > any argument for this.

    >
    > >

    >
    > > I have data = struct.unpack('char',data) but I still get errors

    >
    > >

    >
    > The format strings are described here for Python 3:
    >
    >
    >
    > http://docs.python.org/3.2/library/struct.html
    >
    >
    >
    > and here for Python 2:
    >
    >
    >
    > http://docs.python.org/2.7/library/struct.html


    I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.
    9bizy, Aug 28, 2012
    #8
  9. 9bizy

    MRAB Guest

    On 28/08/2012 23:34, 9bizy wrote:
    > This is what I have to reproduce the challenge I am having below:
    >
    >
    > import csv
    > import struct
    >
    >
    > data = []
    >
    > for Node in csv.reader(file('s_data.xls')):


    That tries to read the file as CSV, but, judging from the extension,
    it's in Excel's format. You don't even use what is read, i.e. Node.

    > data.append(list((file('s_data.xls'))))
    >

    That opens the file again and 'list' causes it to read the file as
    though it were a series of lines in a text file, which, as I've said,
    it looks like it isn't. The list of 'lines' is appended to the list
    'data', so that's a list of lists.
    >
    > data = struct.unpack('!B4HH', data)
    > print "s_data.csv: ", data
    >
    > I tries so many format for the struct.unpack but I got this errors:
    >
    > Traceback (most recent call last):
    >
    > data = struct.unpack('!B4HH', data)
    > struct.error: unpack requires a string argument of length 11
    >

    [snip]
    It's complaining because it's expecting a string argument but you're
    giving it a list instead.
    MRAB, Aug 28, 2012
    #9
  10. 9bizy

    MRAB Guest

    On 28/08/2012 23:35, 9bizy wrote:
    > On Saturday, 25 August 2012 20:16:54 UTC+1, MRAB wrote:
    >> On 25/08/2012 19:34, 9bizy wrote:
    >>
    >> > I am trying to unpack values from sensor data I am retrieving through

    >>
    >> > a serial cable, but I get errors while using struct.unpack, how can I

    >>
    >> > use struct.unpack to unload the data in a readable format?

    >>
    >> >

    >>
    >> > I checked the python documentation for struct and I can seen to find

    >>
    >> > any argument for this.

    >>
    >> >

    >>
    >> > I have data = struct.unpack('char',data) but I still get errors

    >>
    >> >

    >>
    >> The format strings are described here for Python 3:
    >>
    >>
    >>
    >> http://docs.python.org/3.2/library/struct.html
    >>
    >>
    >>
    >> and here for Python 2:
    >>
    >>
    >>
    >> http://docs.python.org/2.7/library/struct.html

    >
    > I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.
    >

    If you want to read from an Excel file you should be using the 'xlrd'
    module. You can find it here: http://www.python-excel.org/
    MRAB, Aug 28, 2012
    #10
  11. 9bizy

    9bizy Guest

    On Tuesday, 28 August 2012 23:49:54 UTC+1, MRAB wrote:
    > On 28/08/2012 23:34, 9bizy wrote:
    >
    > > This is what I have to reproduce the challenge I am having below:

    >
    > >

    >
    > >

    >
    > > import csv

    >
    > > import struct

    >
    > >

    >
    > >

    >
    > > data = []

    >
    > >

    >
    > > for Node in csv.reader(file('s_data.xls')):

    >
    >
    >
    > That tries to read the file as CSV, but, judging from the extension,
    >
    > it's in Excel's format. You don't even use what is read, i.e. Node.
    >
    >
    >
    > > data.append(list((file('s_data.xls'))))

    >
    > >

    >
    > That opens the file again and 'list' causes it to read the file as
    >
    > though it were a series of lines in a text file, which, as I've said,
    >
    > it looks like it isn't. The list of 'lines' is appended to the list
    >
    > 'data', so that's a list of lists.
    >
    > >

    >
    > > data = struct.unpack('!B4HH', data)

    >
    > > print "s_data.csv: ", data

    >
    > >

    >
    > > I tries so many format for the struct.unpack but I got this errors:

    >
    > >

    >
    > > Traceback (most recent call last):

    >
    > >

    >
    > > data = struct.unpack('!B4HH', data)

    >
    > > struct.error: unpack requires a string argument of length 11

    >
    > >

    >
    > [snip]
    >
    > It's complaining because it's expecting a string argument but you're
    >
    > giving it a list instead.


    How do I then convert data to a string argument in this case?
    9bizy, Aug 29, 2012
    #11
  12. 9bizy

    9bizy Guest

    On Tuesday, 28 August 2012 23:49:54 UTC+1, MRAB wrote:
    > On 28/08/2012 23:34, 9bizy wrote:
    >
    > > This is what I have to reproduce the challenge I am having below:

    >
    > >

    >
    > >

    >
    > > import csv

    >
    > > import struct

    >
    > >

    >
    > >

    >
    > > data = []

    >
    > >

    >
    > > for Node in csv.reader(file('s_data.xls')):

    >
    >
    >
    > That tries to read the file as CSV, but, judging from the extension,
    >
    > it's in Excel's format. You don't even use what is read, i.e. Node.
    >
    >
    >
    > > data.append(list((file('s_data.xls'))))

    >
    > >

    >
    > That opens the file again and 'list' causes it to read the file as
    >
    > though it were a series of lines in a text file, which, as I've said,
    >
    > it looks like it isn't. The list of 'lines' is appended to the list
    >
    > 'data', so that's a list of lists.
    >
    > >

    >
    > > data = struct.unpack('!B4HH', data)

    >
    > > print "s_data.csv: ", data

    >
    > >

    >
    > > I tries so many format for the struct.unpack but I got this errors:

    >
    > >

    >
    > > Traceback (most recent call last):

    >
    > >

    >
    > > data = struct.unpack('!B4HH', data)

    >
    > > struct.error: unpack requires a string argument of length 11

    >
    > >

    >
    > [snip]
    >
    > It's complaining because it's expecting a string argument but you're
    >
    > giving it a list instead.


    How do I then convert data to a string argument in this case?
    9bizy, Aug 29, 2012
    #12
  13. 9bizy

    MRAB Guest

    On 29/08/2012 00:01, 9bizy wrote:> On Tuesday, 28 August 2012 23:49:54
    UTC+1, MRAB wrote:
    >> On 28/08/2012 23:34, 9bizy wrote:
    >> > This is what I have to reproduce the challenge I am having below:
    >> >
    >> > import csv
    >> > import struct
    >> >
    >> > data = []
    >> >
    >> > for Node in csv.reader(file('s_data.xls')):

    >>
    >> That tries to read the file as CSV, but, judging from the extension,
    >> it's in Excel's format. You don't even use what is read, i.e. Node.
    >>
    >> > data.append(list((file('s_data.xls'))))
    >> >

    >> That opens the file again and 'list' causes it to read the file as
    >> though it were a series of lines in a text file, which, as I've said,
    >> it looks like it isn't. The list of 'lines' is appended to the list
    >> 'data', so that's a list of lists.
    >> >
    >> > data = struct.unpack('!B4HH', data)
    >> > print "s_data.csv: ", data
    >> >
    >> > I tries so many format for the struct.unpack but I got this errors:
    >> >
    >> > Traceback (most recent call last):
    >> >
    >> > data = struct.unpack('!B4HH', data)
    >> > struct.error: unpack requires a string argument of length 11
    >> >

    >> [snip]
    >> It's complaining because it's expecting a string argument but you're
    >> giving it a list instead.

    >
    > How do I then convert data to a string argument in this case?
    >

    The question is: what are you trying to do?

    If you're trying to read an Excel file, then you should be trying the
    'xlrd' module. You can find it here: http://www.python-excel.org/

    If your trying to 'decode' a binary file, then you should open it in
    binary mode (with "rb"), read (some of) it as a byte string and then
    pass it to struct.unpack.
    MRAB, Aug 29, 2012
    #13
  14. 9bizy

    9bizy Guest

    On Wednesday, 29 August 2012 00:36:40 UTC+1, MRAB wrote:
    > On 29/08/2012 00:01, 9bizy wrote:> On Tuesday, 28 August 2012 23:49:54
    >
    > UTC+1, MRAB wrote:
    >
    > >> On 28/08/2012 23:34, 9bizy wrote:

    >
    > >> > This is what I have to reproduce the challenge I am having below:

    >
    > >> >

    >
    > >> > import csv

    >
    > >> > import struct

    >
    > >> >

    >
    > >> > data = []

    >
    > >> >

    >
    > >> > for Node in csv.reader(file('s_data.xls')):

    >
    > >>

    >
    > >> That tries to read the file as CSV, but, judging from the extension,

    >
    > >> it's in Excel's format. You don't even use what is read, i.e. Node.

    >
    > >>

    >
    > >> > data.append(list((file('s_data.xls'))))

    >
    > >> >

    >
    > >> That opens the file again and 'list' causes it to read the file as

    >
    > >> though it were a series of lines in a text file, which, as I've said,

    >
    > >> it looks like it isn't. The list of 'lines' is appended to the list

    >
    > >> 'data', so that's a list of lists.

    >
    > >> >

    >
    > >> > data = struct.unpack('!B4HH', data)

    >
    > >> > print "s_data.csv: ", data

    >
    > >> >

    >
    > >> > I tries so many format for the struct.unpack but I got this errors:

    >
    > >> >

    >
    > >> > Traceback (most recent call last):

    >
    > >> >

    >
    > >> > data = struct.unpack('!B4HH', data)

    >
    > >> > struct.error: unpack requires a string argument of length 11

    >
    > >> >

    >
    > >> [snip]

    >
    > >> It's complaining because it's expecting a string argument but you're

    >
    > >> giving it a list instead.

    >
    > >

    >
    > > How do I then convert data to a string argument in this case?

    >
    > >

    >
    > The question is: what are you trying to do?
    >
    >
    >
    > If you're trying to read an Excel file, then you should be trying the
    >
    > 'xlrd' module. You can find it here: http://www.python-excel.org/
    >
    >
    >
    > If your trying to 'decode' a binary file, then you should open it in
    >
    > binary mode (with "rb"), read (some of) it as a byte string and then
    >
    > pass it to struct.unpack.


    Thank you MRAB this was helpful.
    9bizy, Aug 29, 2012
    #14
  15. 9bizy

    9bizy Guest

    On Wednesday, 29 August 2012 00:36:40 UTC+1, MRAB wrote:
    > On 29/08/2012 00:01, 9bizy wrote:> On Tuesday, 28 August 2012 23:49:54
    >
    > UTC+1, MRAB wrote:
    >
    > >> On 28/08/2012 23:34, 9bizy wrote:

    >
    > >> > This is what I have to reproduce the challenge I am having below:

    >
    > >> >

    >
    > >> > import csv

    >
    > >> > import struct

    >
    > >> >

    >
    > >> > data = []

    >
    > >> >

    >
    > >> > for Node in csv.reader(file('s_data.xls')):

    >
    > >>

    >
    > >> That tries to read the file as CSV, but, judging from the extension,

    >
    > >> it's in Excel's format. You don't even use what is read, i.e. Node.

    >
    > >>

    >
    > >> > data.append(list((file('s_data.xls'))))

    >
    > >> >

    >
    > >> That opens the file again and 'list' causes it to read the file as

    >
    > >> though it were a series of lines in a text file, which, as I've said,

    >
    > >> it looks like it isn't. The list of 'lines' is appended to the list

    >
    > >> 'data', so that's a list of lists.

    >
    > >> >

    >
    > >> > data = struct.unpack('!B4HH', data)

    >
    > >> > print "s_data.csv: ", data

    >
    > >> >

    >
    > >> > I tries so many format for the struct.unpack but I got this errors:

    >
    > >> >

    >
    > >> > Traceback (most recent call last):

    >
    > >> >

    >
    > >> > data = struct.unpack('!B4HH', data)

    >
    > >> > struct.error: unpack requires a string argument of length 11

    >
    > >> >

    >
    > >> [snip]

    >
    > >> It's complaining because it's expecting a string argument but you're

    >
    > >> giving it a list instead.

    >
    > >

    >
    > > How do I then convert data to a string argument in this case?

    >
    > >

    >
    > The question is: what are you trying to do?
    >
    >
    >
    > If you're trying to read an Excel file, then you should be trying the
    >
    > 'xlrd' module. You can find it here: http://www.python-excel.org/
    >
    >
    >
    > If your trying to 'decode' a binary file, then you should open it in
    >
    > binary mode (with "rb"), read (some of) it as a byte string and then
    >
    > pass it to struct.unpack.


    Thank you MRAB this was helpful.
    9bizy, Aug 29, 2012
    #15
  16. On Tue, 28 Aug 2012 15:34:11 -0700 (PDT), 9bizy
    <> declaimed the following in
    gmane.comp.python.general:

    >
    > for Node in csv.reader(file('s_data.xls')):
    > data.append(list((file('s_data.xls'))))
    >
    >
    > data = struct.unpack('!B4HH', data)
    > print "s_data.csv: ", data
    >
    > I tries so many format for the struct.unpack but I got this errors:
    >
    > Traceback (most recent call last):
    >
    > data = struct.unpack('!B4HH', data)
    > struct.error: unpack requires a string argument of length 11
    >


    Which seems reasonable -- you've specified 1 byte, and 5 16-bit
    integers => 11 bytes needed for input.

    Have you examined what your input really contains? I see a list
    given the combination of .append() and list()
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Aug 29, 2012
    #16
  17. On Tue, 28 Aug 2012 15:35:58 -0700 (PDT), 9bizy
    <> declaimed the following in
    gmane.comp.python.general:

    >
    > I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.


    Why should it?... struct doesn't care WHERE the data came from; it
    just takes a /string of bytes/ and interprets the contents as a series
    of raw binary fields based upon the format character.

    >>> import struct
    >>> data = "elevenbytes"
    >>> struct.unpack("!B4HH", data)

    (101, 27749, 30309, 28258, 31092, 25971)
    >>> ndata = [30, 138, 32260, 563, 3238, 23537]
    >>> struct.pack("!B4HH", *ndata)

    '\x1e\x00\x8a~\x04\x023\x0c\xa6[\xf1'
    >>>

    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Aug 29, 2012
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris Fogelklou
    Replies:
    36
    Views:
    1,371
    Chris Fogelklou
    Apr 20, 2004
  2. Matthew Barnes
    Replies:
    0
    Views:
    372
    Matthew Barnes
    Nov 19, 2003
  3. mikeSpindler

    struct.unpack() and bit operations

    mikeSpindler, Sep 23, 2004, in forum: Python
    Replies:
    2
    Views:
    577
    Peter Otten
    Sep 23, 2004
  4. Geoffrey
    Replies:
    5
    Views:
    589
    Geoffrey
    Dec 1, 2004
  5. Peter Otten

    struct unpack issue

    Peter Otten, Jun 13, 2008, in forum: Python
    Replies:
    1
    Views:
    383
    Ping Zhao
    Jun 14, 2008
Loading...

Share This Page