Fast reading and unpacking of binary data (struct module)

[Daniel Platz]

Hi,

I have a Python newbie question about reading data from a binary file.
I have an huge binary file from an external program. I want to read
and process the data in this file in a reasonable time. It turns out
that the reading of the data itself and the processing do not need
most of the time. However, when using the read(bytes) method Python
returns a string representing the binary information in hex. This
string I have to "cast/translate" into a number (in my case a signed
short). For this I am using the method struct.unpack from the struct
module. This unpacking part of the program takes by far the most time.
Is there a way to speed this up or to do it the unpacking more
cleverly than with the struct module?

Thanks in advance.

With kind regards,

Daniel

 

[Gabriel Genellina]

En Tue, 21 Jul 2009 21:00:13 -0300, Daniel Platz

I have an huge binary file from an external program. I want to read
and process the data in this file in a reasonable time. It turns out
that the reading of the data itself and the processing do not need
most of the time. However, when using the read(bytes) method Python
returns a string representing the binary information in hex. This
string I have to "cast/translate" into a number (in my case a signed
short). For this I am using the method struct.unpack from the struct
module. This unpacking part of the program takes by far the most time.
Is there a way to speed this up or to do it the unpacking more
cleverly than with the struct module?

Try creating a Struct object with your format and use its unpack() method.
http://docs.python.org/library/struct.html#struct-objects

If your format consists of just integers, probably an array is more
efficient:
http://docs.python.org/library/array.html#array.array.fromfile

 

[Neal Becker]

Hi,

I have a Python newbie question about reading data from a binary file.
I have an huge binary file from an external program. I want to read
and process the data in this file in a reasonable time. It turns out
that the reading of the data itself and the processing do not need
most of the time. However, when using the read(bytes) method Python
returns a string representing the binary information in hex. This
string I have to "cast/translate" into a number (in my case a signed
short). For this I am using the method struct.unpack from the struct
module. This unpacking part of the program takes by far the most time.
Is there a way to speed this up or to do it the unpacking more
cleverly than with the struct module?

Thanks in advance.

With kind regards,

Daniel

Consider mmap
Consider numpy
Consider numpy+mmap