Python serial data aquisition

[Flavio codeco coelho]

Hi,

I am using pyserial to acquire data from an A/D converter plugged to
my serial port.

my hardware represents analog voltages as 12bit numbers. So, according
to the manufacturer, this number will be stored in two bytes like
this;
|-------------bits(1-8)-----------|
Byte1: x x x n1 n2 n3 n4 n5
Byte2: x n6 n7 n8 n9 n10 n11 n12

where x is some other information, and nx are the digits of my number.

My problem is to how to recover my reading from these bytes, since
pyserial gives me a character (string) from each byte... I dont know
how to throw away the unneeded bits and concatenate the remaining
bits to form a number...


Flávio Codeço Coelho

 

[Steve Holden]

Hi,

I am using pyserial to acquire data from an A/D converter plugged to
my serial port.

my hardware represents analog voltages as 12bit numbers. So, according
to the manufacturer, this number will be stored in two bytes like
this;
|-------------bits(1-8)-----------|
Byte1: x x x n1 n2 n3 n4 n5
Byte2: x n6 n7 n8 n9 n10 n11 n12

where x is some other information, and nx are the digits of my number.

My problem is to how to recover my reading from these bytes, since
pyserial gives me a character (string) from each byte... I dont know
how to throw away the unneeded bits and concatenate the remaining
bits to form a number...


Flávio Codeço Coelho

Try something like (here I'm assuming that n12 is the least-significant
bit) (untested):

((ord(byte1) & 31) << 7 ) + (ord(byte2) & 127)

This takes the rightmost five bits from the first byte's integer value,
shifts them up seven bits, and then adds the rightmost seven bits of the
second byte's integer value.

regards
Steve

 

[Michael Fuhr]

my hardware represents analog voltages as 12bit numbers. So, according
to the manufacturer, this number will be stored in two bytes like
this;
|-------------bits(1-8)-----------|
Byte1: x x x n1 n2 n3 n4 n5
Byte2: x n6 n7 n8 n9 n10 n11 n12

where x is some other information, and nx are the digits of my number.

Is byte1 the high-order byte or the low-order byte? Is n1 the
high-order bit or the low-order bit? In my example below I'll
assume that byte1 and n1 are in the high-order positions.

My problem is to how to recover my reading from these bytes, since
pyserial gives me a character (string) from each byte...

You could convert the characters to their integer ASCII values with
ord() and perform bitwise operations on them. For example, suppose
you have the following characters:

byte1 = '\xd2' # 1101 0010
byte2 = '\xb5' # 1011 0101

If byte1 is the high-order byte and n1 is the high-order bit, then
you need to mask off the upper three bits of byte1 (giving 0001 0010)
and the upper bit of byte2 (giving 0011 0101), then shift byte1 left 7
positions (not 8) and OR it with byte2 (giving 1001 0011 0101).

value = ((ord(byte1) & 0x1f) << 7) | (ord(byte2) & 0x7f)

If the actual byte and/or bit order is different then you'll have
to modify the expression, but this should at least give you ideas.

 

[Flavio codeco coelho]

If the actual byte and/or bit order is different then you'll have
to modify the expression, but this should at least give you ideas.

Thanks Michael and Steve,

I'll put your Ideas to the test ASAP, meanwhile, could you point me to
references to these bit operations in Python? I am new to this stuff,
and might need to do more of this to support other hardware...

I havent been able to find anything about this on the python
documentation..

Thanks a lot!!

Flavio

 

[Paul Rubin]

I'll put your Ideas to the test ASAP, meanwhile, could you point me to
references to these bit operations in Python? I am new to this stuff,
and might need to do more of this to support other hardware...

I havent been able to find anything about this on the python
documentation..

ord(ch) turns a character into an integer, i.e. ord('a') = 97, etc.

You can use the << (left shift), >> (right shift), | (logical or),
and & (logical and) operations just like in C.

So for your diagram

|-------------bits(1-8)-----------|
Byte1: x x x n1 n2 n3 n4 n5
Byte2: x n6 n7 n8 n9 n10 n11 n12

assuming n1 is the high order bit, you'd just say

value = ((ord(byte1) & 0x1f) << 7) | (ord(byte2) & 0x3f)

or something like that.

 

[Steve Holden]

Thanks Michael and Steve,

I'll put your Ideas to the test ASAP, meanwhile, could you point me to
references to these bit operations in Python? I am new to this stuff,
and might need to do more of this to support other hardware...

I havent been able to find anything about this on the python
documentation..

Thanks a lot!!

Flavio

ord() is documented in section 2.1 of the library reference manual
(Built-in Functions) - see
http://docs.python.org/lib/built-in-funcs.html#l2h-53 for further
information.

Bitstring operations are at
http://docs.python.org/lib/bitstring-ops.html#l2h-150

Hope this helps.

regards
Steve

 

[Flavio codeco coelho]

If the actual byte and/or bit order is different then you'll have
to modify the expression, but this should at least give you ideas.

Hi Michael,

It all looks pretty god but there is a couple of things I still don't
understand, 1) in Steve's solution (which seems equivalent to your
own), he does the masking, shifts by seven, and then sums the two
numbers while you, instead of summing, use a logical or. How can these
operations be equivalent? or are they? Is the logical or equivalent
to a concatenation?

2) why 7? why do we have shift 7 bits? is that so the 8th bit on byte
one is aligned with the first bit on the second byte?

thanks:

 

[Flavio codeco coelho]

or something like that.

Hi Paul,

thanks for your answer.
I Noticed, however that although your solution is almost identical to
that of Michael (earlier in the thread) your masking for the second
byte is different than the one he used. Since hex numbers get me all
confused (and python doesn't convert to binary), I was wondering which
one is the correct masking...

thnaks,

Flavio

 

[Bengt Richter]

Hi,

I am using pyserial to acquire data from an A/D converter plugged to
my serial port.

my hardware represents analog voltages as 12bit numbers. So, according
to the manufacturer, this number will be stored in two bytes like
this;
|-------------bits(1-8)-----------|
Byte1: x x x n1 n2 n3 n4 n5
Byte2: x n6 n7 n8 n9 n10 n11 n12

where x is some other information, and nx are the digits of my number.

My problem is to how to recover my reading from these bytes, since
pyserial gives me a character (string) from each byte... I dont know
how to throw away the unneeded bits and concatenate the remaining
bits to form a number...

The others have shown how to recover a 12 bit positive value 0 through 4095,
but if the number is signed, and you want the signed value, you'll have to
find out how signed numbers are represented. An offset is common, in which
case you would subtract 2048 (2**11). If it's two's complement by some chance,
you'll want (I think -- untested ;-) to do num -= 2*(num&2048) instead of always num -= 2048

If you need speed in converting large strings of byte pairs, you could
convert pairs of bytes with the array module ('H' for unsigned 2-byte numbers)
and use the resulting 16-bit numbers as indices into another array of final values
prepared beforehand with redundant information that will accomplish the effect
of masking and shifting and adjusting sign.

If you need this speed, volunteers will magically appear. Maybe even if you don't ;-)

Regards,
Bengt Richter

 

[Diez B. Roggisch]

It all looks pretty god but there is a couple of things I still don't

understand, 1) in Steve's solution (which seems equivalent to your
own), he does the masking, shifts by seven, and then sums the two
numbers while you, instead of summing, use a logical or. How can these
operations be equivalent? or are they? Is the logical or equivalent
to a concatenation?

No, but look at this:

111000 + 000111 = 111111
111000 | 000111 = 111111

Adding and or are yielding the same results as long as at least one of the
bits combined is zero. By shifting the appropriate number of bits and thus
clearing the lower ones and "anding" the second argument with 0x7f, this is
ensured for the whole numbers.

2) why 7? why do we have shift 7 bits? is that so the 8th bit on byte
one is aligned with the first bit on the second byte?

Because of the specification you gave - look at the Byte2 spec closely.

 

[Flavio codeco coelho]

The others have shown how to recover a 12 bit positive value 0 through 4095,
but if the number is signed, and you want the signed value, you'll have to
find out how signed numbers are represented. An offset is common, in which
case you would subtract 2048 (2**11). If it's two's complement by some chance,
you'll want (I think -- untested ;-) to do num -= 2*(num&2048) instead of always num -= 2048
If you need speed in converting large strings of byte pairs, you could
convert pairs of bytes with the array module ('H' for unsigned 2-byte numbers)
and use the resulting 16-bit numbers as indices into another array of final values
prepared beforehand with redundant information that will accomplish the effect
of masking and shifting and adjusting sign.
If you need this speed, volunteers will magically appear. Maybe even if you don't ;-)
Regards,
Bengt Richter

Hi Bengt,

The Idea of using Array is realy cool Though I have to think about it
since I would to plot the values as they are sampled...

Anyway, how would you set up this array to do the shifting and
masking?

BTW, since this thread is generating quite a bit of attention let me
post a complete description of my problem so that it may serve as
reference to others:

Hardware: DI-151RS from Dataq (2 analog channels, 2 digital input,
single ended/differential recording, max sampling rate 240Hz) connects
to the serial pro through a standard db9 plug.

Encryption table:


B7 B6 B5 B4 B3 B2 B1 B0
Byte1 A4 A3 A2 A1 A0 1 Din 0
Byte2 A11 A10 A9 A8 A7 A6 A5 1
Byte3 B4 B3 B2 B1 B0 1 Din 1
Byte4 B11 B10 B9 B8 B7 B6 B5 1

first two bytes are for analog ch 1 and remaining two are for ch 2.
Din stands for digital in. AXX and BXX are the nth bits of each
reading. A0 and B0 are the least significant bits.

The latest and preferred solution on how to convert these bytes is,
according to the suggestion of Chris Liechti (author of pyserial) is:
(this is for the first channel only, repeat for the second)

import struct

l, h = struct.unpack(">BB", ser.read(2))
n = (l >> 3) + ((h >> 1)<<5)

struct.unpack returns a tuple of values represented by a string(the
output of the read command) packed according to the format specified
by ">BB"
In this forma string, ">" stands for big Endian representation and "B"
stands for unsigned char.

If anyone has a better suggestion, speack up!

oof! I started this thread knowing next to nothing about this stuff,
It seem that I am finally getting the idea! )

cheers,

Flávio

 

[Bengt Richter]

Hi Bengt,

The Idea of using Array is realy cool Though I have to think about it
since I would to plot the values as they are sampled...

Anyway, how would you set up this array to do the shifting and
masking?

See below.

BTW, since this thread is generating quite a bit of attention let me
post a complete description of my problem so that it may serve as
reference to others:

Hardware: DI-151RS from Dataq (2 analog channels, 2 digital input,
single ended/differential recording, max sampling rate 240Hz) connects
to the serial pro through a standard db9 plug.

Encryption table:


B7 B6 B5 B4 B3 B2 B1 B0
Byte1 A4 A3 A2 A1 A0 1 Din 0
Byte2 A11 A10 A9 A8 A7 A6 A5 1
Byte3 B4 B3 B2 B1 B0 1 Din 1
Byte4 B11 B10 B9 B8 B7 B6 B5 1

first two bytes are for analog ch 1 and remaining two are for ch 2.
Din stands for digital in. AXX and BXX are the nth bits of each
reading. A0 and B0 are the least significant bits.

The latest and preferred solution on how to convert these bytes is,
according to the suggestion of Chris Liechti (author of pyserial) is:
(this is for the first channel only, repeat for the second)

import struct

l, h = struct.unpack(">BB", ser.read(2))
n = (l >> 3) + ((h >> 1)<<5)

struct.unpack returns a tuple of values represented by a string(the
output of the read command) packed according to the format specified
by ">BB"
In this forma string, ">" stands for big Endian representation and "B"
stands for unsigned char.

If anyone has a better suggestion, speack up!

oof! I started this thread knowing next to nothing about this stuff,
It seem that I am finally getting the idea! )

----< flavio.py >----------------------------------------------------
import array
# set up conversion table for all 8-bit little-endian byte pair possibilities
convert = array.array('l',[((ahi>>1)<<5)+(alo>>3) for ahi in xrange(256) for alo in xrange(256)])

def samptoval(samples):
# convert many-sample string to array of 16-bit unnsigned integers
data = array.array('H')
data.fromstring(samples)
# replace 16-bit byte pair values with 12-bit values
for i, v in enumerate(data):
data = convert[v]
return data

def fakedata(i12, b0=0, din=0):
"""convert unsigned 12-bit integer to little-endian byte string"""
return chr(((i12&0x1f)<<3)+4+din*2+b0) + chr(((i12&0xfe0)>>4)+1)

def test(v1=0, v2=256):
assert 0 <= v1 <= v2 <=4096, "values must be in range(4096)"
# default (0, 256) => fake a lot of A, B input pairs like ser.read(4*256)
samples = ''.join([fakedata(i)+fakedata(i,1) for i in xrange(v1,v2)])
data = samptoval(samples)
for i, v in enumerate(xrange(v1,v2)):
#print i, data[i*2:i*2+2], v
assert data[i*2] == data[i*2+1] == v
return samples, data

if __name__ == '__main__':
import sys
args = sys.argv[1:3]
result = test(*map(int, args))
print '%r\n%r' % result
---------------------------------------------------------------------

This creates pairs of A and B data for a sequence of 12-bit values and converts
them back for a test, e.g.,

[13:40] C:\pywk\clp>flavio.py 0 4
'\x04\x01\x05\x01\x0c\x01\r\x01\x14\x01\x15\x01\x1c\x01\x1d\x01'
array('H', [0, 0, 1, 1, 2, 2, 3, 3])

[13:45] C:\pywk\clp>flavio.py 30 34
'\xf4\x01\xf5\x01\xfc\x01\xfd\x01\x04\x03\x05\x03\x0c\x03\r\x03'
array('H', [30, 30, 31, 31, 32, 32, 33, 33])

[13:45] C:\pywk\clp>flavio.py 2046 2050
'\xf4\x7f\xf5\x7f\xfc\x7f\xfd\x7f\x04\x81\x05\x81\x0c\x81\r\x81'
array('H', [2046, 2046, 2047, 2047, 2048, 2048, 2049, 2049])

[13:45] C:\pywk\clp>flavio.py 4092 4096
'\xe4\xff\xe5\xff\xec\xff\xed\xff\xf4\xff\xf5\xff\xfc\xff\xfd\xff'
array('H', [4092, 4092, 4093, 4093, 4094, 4094, 4095, 4095])

From your diagram it looks to me like your data is little-endian (least significant
data in the first byte) so I did it that way.

Actually at 240 hz, I doubt you'd have any problem with speed doing it any which way,
so this is premature optimization, and just to explain the technique, in case it's interesting.
But if you had a HUGE long string of sample data, samptoval should go fairly fast I would think.

You could also use struct to convert two bytes to an unsigned 16-bit number, and then use that number
to look up the value as flavio.convert[num] instead of making two numbers and doing the combination
yourself.





Regards,
Bengt Richter

 

[Flavio codeco coelho]

struct.unpack returns a tuple of values represented by a string(the
output of the read command) packed according to the format specified
by ">BB"
In this forma string, ">" stands for big Endian representation and "B"
stands for unsigned char.
If anyone has a better suggestion, speack up!
oof! I started this thread knowing next to nothing about this stuff,
It seem that I am finally getting the idea! )
cheers,
Flávio

Ok Folks,

Thanks for all the help!

Thanks to you and especially to Chris Liechti (pyserial),

I managed to finish the Data aquisition module for the Dataq DI-151RS.

you can check it out here:

http://www.procc.fiocruz.br:8080/procc/Members/flavio/codepy/dataq/

It implements the basic operational functionality, fancier stuff
should be added by the user.

enjoy!

Flávio Codeco Coelho

P.S.: I'll be happy to support other hardware from Dataq if people
give me access to other devices.

 

[Miki]

Hello Flávio,

My problem is to how to recover my reading from these bytes, since
pyserial gives me a character (string) from each byte... I dont know
how to throw away the unneeded bits and concatenate the remaining
bits to form a number...

See the array module.
Also http://tebeka.bizhat.com/Software/bitter.py

HTH.
Miki