bitstreams

[Andreas Lobinger]

Aloha,
can anyone here recommend a implementation for bitstreams?

For a project i need to extract bitfields from a file containing
one large bitstream (lenght >> 1000bit). The bitfields (1bit-24bit)
are not aligned to char boundaries (8bit).

Something like
b1 = bitstream.fromfile('m1.bin')
v1 = b1.getbits(startpos=3,length=10) # v ist a reg. int
v2 = b1.getnext(8) # bits 0-7
v3 = b1.getnext(10) # bits 8-17

Hoping for an answer and wishing a happy day
LOBI

 

[Scott David Daniels]

Aloha,
can anyone here recommend a implementation for bitstreams?

For a project i need to extract bitfields from a file containing
one large bitstream (lenght >> 1000bit). The bitfields (1bit-24bit)
are not aligned to char boundaries (8bit).

Something like
b1 = bitstream.fromfile('m1.bin')
v1 = b1.getbits(startpos=3,length=10) # v ist a reg. int
v2 = b1.getnext(8) # bits 0-7
v3 = b1.getnext(10) # bits 8-17

Hoping for an answer and wishing a happy day
LOBI

You might start with something like the following as a primitive,
tweak it to deal correctly with your byte sex (big-endian vs. little
endian), bit numbering, and such. Then when that works, add a
current position and auto-advance stuff. Then you can decide if you
want to go in bigger "bytes" by something like 'L' instead of 'B'
below. Note that 1000 bits is not so very big that reading the file
into memory is a bad idea. In fact, I'd only think about that above
about a million bits.

import array

class Bitvector(object):
def __init__(self, filename):
datafile = file(filename, 'rb')
self.data = array.array('B', datafile.read())
datafile.close()
self.elbits = self.data.itemsize * 8 # 8 bits/byte
def grab(self, start, length):
word, part = divmod(start, self.elbits)
sofar = self.elbits - part
result = self.data[word] & ((1L << sofar) - 1)
# 1L above so no sign bit hassles below
while sofar < length:
word += 1
result = (result << self.elbits) | self.data[word]
sofar += self.elbits
return result >> sofar - length

 

[Michael Hudson]

Aloha,
can anyone here recommend a implementation for bitstreams?

For a project i need to extract bitfields from a file containing
one large bitstream (lenght >> 1000bit). The bitfields (1bit-24bit)
are not aligned to char boundaries (8bit).

Something like
b1 = bitstream.fromfile('m1.bin')
v1 = b1.getbits(startpos=3,length=10) # v ist a reg. int
v2 = b1.getnext(8) # bits 0-7
v3 = b1.getnext(10) # bits 8-17

Well, a thousand bits really isn't very many. Read the lot and slurp
it into a long? Hmm, not sure of an easy way to do that... something
like

a = array.array('B')
a.fromstring(open('m1.bin').read())
v = 0L
m = 1L
for b in a:
v += m*b
m *= 256

then use shifts & masks as desired on v.

There ought to be a cuter way to go from m1.bin to v, though...

Cheers,
mwh

 

[Andreas Lobinger]

Aloha,

Well, a thousand bits really isn't very many. Read the lot and slurp
it into a long?

Thanks to both replies by now. Two points. length>>1000bits means
that i'm talking about an implementation that runs up to 1000000bits.
In the moment i'm working with a string in memory and use a state
maschine to get to the first byte, use string.struct to get
the word/long containing all bits and a little bit masking.

There is no real problem writing code for this, but i thought that
it could be that there are already implementations for bitstreams
f.e. mpeg-reading, a zlib implementation or other coding/decoding
things...

Wishing a happy day
LOBI

 

[Michael Hudson]

Aloha,




Thanks to both replies by now. Two points. length>>1000bits means
that i'm talking about an implementation that runs up to 1000000bits.

Oh

In the moment i'm working with a string in memory and use a state
maschine to get to the first byte, use string.struct to get
the word/long containing all bits and a little bit masking.
Right.

There is no real problem writing code for this, but i thought that
it could be that there are already implementations for bitstreams
f.e. mpeg-reading, a zlib implementation or other coding/decoding
things...

Not to my knowledge! Such things are generally done by C libraries
that are then wrapped for use by Python, in my experience.

Writing a C extension to do what you need probably isn't amazingly
hard (so long as you don't need bitfields wider than the machine word
size I guess).

Cheers,
mwh