Non-POSIX parity (mark/space) with Python-Serial on Linux.

[mlenz]

I'm working on a project where I need to communicate with some devices via modem which have the possibility of using MARK and SPACE parity. These are not defined by POSIX and therefore are not directly supported under Linux.

I've found the following discussion on the topic:

http://www.lothosoft.ch/thomas/libmip/markspaceparity.php

and I have been trying to use this information (since the TERMIOS module is available) to proceed with the communication.

I was able to use minicom to determine that the first device I started testing with uses 7M1 but cannot figure out how to implement the solution described by the author above.

Any pointers would be greatly appreciated.

 

[Matthew Lenz]

I should also note that I am aware of the following discussion on the newsgroup:

https://groups.google.com/d/msg/comp.lang.python/1HyCqPSOf50/eQINFrrFKwoJ

However, I believe this refers to implementing the solution for 8M1 and 8S1.

 

[Nizamov Shawkat]

I'm working on a project where I need to communicate with some devices via modem which have the possibility of using MARK and SPACE parity.  Theseare not defined by POSIX and therefore are not directly supported under Linux.

I've found the following discussion on the topic:

http://www.lothosoft.ch/thomas/libmip/markspaceparity.php

and I have been trying to use this information (since the TERMIOS module is available) to proceed with the communication.

I was able to use minicom to determine that the first device I started testing with uses 7M1 but cannot figure out how to implement the solution described by the author above.

"The modes 7M1 (7 data bits, MARK parity, 1 stop bit) and 7S1 (7 data
bits, SPACE parity, 1 stop bit) can easily be emulated using 8N1 (0
data bits, NO parity, 1 stop bit) and setting the 8th data bit to 1
resp. 0. This is relatively simple to implement and cannot be
distinguished by the receiver."

It means that 7M1 === 8N1. Set 8N1 mode on your side and 7M1 on the
other side. I really do not understand what is the reason to have
dedicated 7M1 or 7S1 mode - it is no different from regular 8 bit mode
from the hardware point of view. From the software point of view it is
just the matter of the definition of the highest bit. In other words,
7M1/7S1 are two complementary subsets of a single 8N1 set.

HTH

 

[Matthew Lenz]

Using 8N1 under minicom with this device resulted in garbled text when once connected. Connection using 7M1 resulted in the correct text. So there must be something else that needs to be done in my python program correct?

 

[Matthew Lenz]

Using 8N1 under minicom with this device resulted in garbled text when once connected. Connection using 7M1 resulted in the correct text. So there must be something else that needs to be done in my python program correct?

 

[Chris Angelico]

Using 8N1 under minicom with this device resulted in garbled text when once connected.  Connection using 7M1 resulted in the correct text.  So there must be something else that needs to be done in my python program correct?

Using 8N1 when it's really 7M1 means you have the high bit set on
every byte. I don't know if there's an easy way to do this fast in
Python, but what you need to do is mask them all off... I'm not sure
if there's a better way, but this ought to work:

string = "".join(chr(ord(x)&0x7f) for x in string)

In Python 3, iterating over a 'bytes' string produces integers, so
omit the ord() call. Other than that, code is not tested.

ChrisA

 

[David Riley]

Using 8N1 under minicom with this device resulted in garbled text when once connected. Connection using 7M1 resulted in the correct text. So there must be something else that needs to be done in my python program correct?

Under minicom in 8N1, it's going to look garbled because the high bit will always be set. Minicom will try to spit out those characters anyway, which will print out whatever extended ASCII garbage your terminal supports in the 0x80-0xFF range. Programmatically, though, you can strip off the high bit when you're receiving it in Python.

"Space" parity, on the other hand, should look normal under Minicom because the high bit will always be low, giving you standard 7-bit ASCII.

- Dave

 

[Matthew Lenz]

Ahh. Ok. So how would I go about doing that with python? I think in perl (sorry for the naughty word) I could use the tr// (translate) but is there a quick way to do so with python? Is it going to be necessary to convert commands I SEND to the device or only convert what I receive?

 

[Matthew Lenz]

Ahh. Ok. So how would I go about doing that with python? I think in perl (sorry for the naughty word) I could use the tr// (translate) but is there a quick way to do so with python? Is it going to be necessary to convert commands I SEND to the device or only convert what I receive?

 

[David Riley]

Ahh. Ok. So how would I go about doing that with python? I think in perl (sorry for the naughty word) I could use the tr// (translate) but is there a quick way to do so with python? Is it going to be necessary to convert commands I SEND to the device or only convert what I receive?

The high-level overview is that you'll want to OR in 0x80 on transmit, and AND 0x7F on receive (thus inserting the high bit when you send it out and removing it when you receive). If you wanted to be extra-sure you're receiving things correctly, you should also check to see if your value ANDed with 0x80 is not zero.

In Python 2.x, as mentioned, when you iterate over a string, you get a bunch of tiny one-character strings, which you then need to convert into numbers with ord() and back into strings with chr() when you re-concatenate it. ord() and chr() may be familiar to you from Perl. For example, you could do this on reception:

----

# However you get your data out of serial
received_str = receive_my_string()

ord_str = [ord(x) for x in received_str]

# An exception may be extreme in this case, but hey. Note that
# the filter() function actually returns a list of all the
# characters that don't have the high bit set, so you could
# actually use that if you wanted to.
if filter((lambda x: x < 0x80), ord_str):
raise IOError("Received character without mark parity")

return "".join([chr(x & 0x7F) for x in received_str])

----

In Python 3.x, iterating over a bytes array (which is, hopefully, what your serial interface returns) will give you a bunch of ints, so you shouldn't need to do the conversions:


----

# However you get your data out of serial
received_bytes = receive_my_string()

# In Python 3.x, filter() returns an iterator, which is generally a
# better thing to return, but doesn't test as nicely for a bool.
for b in received_bytes:
if b < 0x80:
raise IOError("Received character without mark parity")

# None of this "".join() nonsense with byte arrays!
return bytes([(x & 0x7F) for x in received_bytes])


----


There are surely more efficient ways to do what I've typed up there, but those should work pretty well for whatever you're looking to do. For sending, you pretty much want to swap (x & 0x7F) with (x | 0x80) to insert that high bit.


- Dave

 

[gene heskett]

Under minicom in 8N1, it's going to look garbled because the high bit
will always be set. Minicom will try to spit out those characters
anyway, which will print out whatever extended ASCII garbage your
terminal supports in the 0x80-0xFF range. Programmatically, though, you
can strip off the high bit when you're receiving it in Python.

I have been using 8n1 in minicom for years, never ever had such a problem.
In fact, I don't even know if I can set the path to mark parity as it is so
rarely used. E or O as error detectors are much more commonly used.

Example copy/paste from minicom, talking to a trs-80 Color Computer 3
running a shell under nitros9, which is a bit like unix. I am asking it
for the settings of its own output path, .1=stdout:

{t2|07}/DD/NITROS9/dw3install/6309L2/SCRIPTS:tmode .1
/t2
upc=00 bso=01 dlo=00 eko=01 alf=01 nul=00 pau=01 pag=18
bsp=08 del=18 eor=0D eof=1B rpr=09 dup=01 psc=17 int=03
qut=05 bse=08 ovf=07 par=01 bau=06 xon=00 xof=00
{t2|07}/DD/NITROS9/dw3install/6309L2/SCRIPTS:

And that is 9600 baud 8n1 on both ends. Ascii is normally 7 bit and will
have a low 8th bit if fed normal ascii data, so how is the 8th bit getting
set other than purposely setting 7M1 on the other end of the cable?

"Space" parity, on the other hand, should look normal under Minicom
because the high bit will always be low, giving you standard 7-bit
ASCII.
Yes.

- Dave

Cheers, Gene
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: <http://coyoteden.dyndns-free.com:85/gene>
Everything is controlled by a small evil group to which, unfortunately,
no one we know belongs.

 

[David Riley]

And that is 9600 baud 8n1 on both ends. Ascii is normally 7 bit and will
have a low 8th bit if fed normal ascii data, so how is the 8th bit getting
set other than purposely setting 7M1 on the other end of the cable?

That's what I thought the OP was doing; it sounds like he's trying to receive 7M1 in Minicom using 8N1 on the terminal and getting garbled data because the high bit is set (because the other end is sending 7M1). I never meant to imply that 8N1 would give garbled data if both ends were set to it; indeed, that's pretty much standard communications settings for short cables in low to moderate noise environments. If anyone else read it that way, that's not what I meant.

- Dave

 

[Matthew Lenz]

Thanks, this will be a great help.

Just wanted to confirm that you meant to use [ .. for x in ord_str] in the example conversion? Got a TypeError using the received_str.

 

[Matthew Lenz]

Thanks, this will be a great help.

Just wanted to confirm that you meant to use [ .. for x in ord_str] in the example conversion? Got a TypeError using the received_str.

 

[David Riley]

Thanks, this will be a great help.

Just wanted to confirm that you meant to use [ .. for x in ord_str] in the example conversion? Got a TypeError using the received_str.

Yes, I probably should have double-checked that. ord_str is indeed what I meant.


- Dave

 

[MRAB]

Ahh. Ok. So how would I go about doing that with python? I think in
perl (sorry for the naughty word) I could use the tr// (translate)
but is there a quick way to do so with python? Is it going to be
necessary to convert commands I SEND to the device or only convert
what I receive?

Python strings have a .translate method:

# Example in Python 2
import string

# From top bit set...
from_chars = "".join(chr(c | 0x80) for c in range(0x7F))

# ...to top bit clear.
to_chars = "".join(chr(c) for c in range(0x7F))

# Build the translation table.
force_clear = string.maketrans(from_chars, to_chars)

s = "\x41\xC1"
print s
print s.translate(force_clear)

 

[gene heskett]

That's what I thought the OP was doing; it sounds like he's trying to
receive 7M1 in Minicom using 8N1 on the terminal and getting garbled
data because the high bit is set (because the other end is sending
7M1). I never meant to imply that 8N1 would give garbled data if both
ends were set to it; indeed, that's pretty much standard communications
settings for short cables in low to moderate noise environments. If
anyone else read it that way, that's not what I meant.

- Dave

I think that getting the other end off 7M1 was what I was saying. Trying
to attack the bad data after capture by writing code always seems extremely
masochistic to me.

The amount of miss-understanding that seems to pervade rs-232
communications is mind boggling at times. The tech itself is so old it is
being forgotten!

Cheers, Gene
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: <http://coyoteden.dyndns-free.com:85/gene>
Whatever occurs from love is always beyond good and evil.
-- Friedrich Nietzsche

 

[Matthew Lenz]

Another thing I noticed is that the & and | appear to give the same result as adding or subtracting 128 from the ordinal value. I'm assuming that isn't coincidence.

 

[Matthew Lenz]

Another thing I noticed is that the & and | appear to give the same result as adding or subtracting 128 from the ordinal value. I'm assuming that isn't coincidence.

 

[Grant Edwards]

Another thing I noticed is that the & and | appear to give the same
result as adding or subtracting 128 from the ordinal value.

Nope, that's only true for some values.

If we're limiting ourselves to byte values, then we're talking
modulo-256 arithmetic, so:

128 + 128 = 0
128 | 128 = 128

0 - 128 = 128
0 & 0x7f = 0


What's is true is that adding 128 is actullay the same as subtracting
128, and both are the same as exclusive-or 128 (v ^ 128):

0
128

I'm assuming that isn't coincidence.

Well, the weighting of the high-order bit in an 8-bit wide binary
number is 128, if that's what you're getting at...