Is it possible to use python to get True Full Duplex on a Serial port?

[Hendrik van Rooyen]

In the past, on this group, I have made statements that said that on Linux,
the serial port handling somehow does not allow transmitting and receiving at
the same time, and nobody contradicted me.

I am running into the self same issue again.

What I normally do is to open the port like this:

port = open("/dev/ttyS0","r+b",0)

and then I unblock it with:

def unblock(f):
"""Given file 'f', sets its unblock flag to true."""

fcntl.fcntl(f.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

Then I can write a loop that uses a try-except to see if there are characters
available, and that examines a queue to see if there is something to
transmit, to give the appearance of full duplex functionality.

What I would really like is to have two threads - one that does blocking input
waiting for a character, and one that examines an output queue and transmits
the stuff it finds.

When I try to do this, it does not seem to work - as far as I can see, it is
as if the underlying implementation is somehow single threaded - if it is
waiting for a received character, it waits until something comes in before it
will transmit anything. So if you are talking to a device that does not
respond, the whole thing freezes up waiting for a character that never comes,
and nothing is transmitted either, despite the call to
port.write(somestring). The write blocks, and everything stops, waiting for
the receipt to finish.

Is there a way to get full duplex, so that the transmit and receive are
independent of each other?

Or are we stuck with a disk-like model that forces a sequence on reads and
writes?

- Hendrik

 

[Diez B. Roggisch]

In the past, on this group, I have made statements that said that on Linux,
the serial port handling somehow does not allow transmitting and receiving at
the same time, and nobody contradicted me.

I am running into the self same issue again.

What I normally do is to open the port like this:

port = open("/dev/ttyS0","r+b",0)

How about using pyserial? With that, I never had any problems accessing
the the serial ports, and AFAIK no duplex-problems as well. And I
seriously doubt that these are a python-related problem - python only
has a very thin, direct layer above the posix-calls, and doesn't do
anything that would explain your observed behavior. The GIL is not the
issue here either - it won't interfer with blocking IO.

Diez

 

[greg]

port = open("/dev/ttyS0","r+b",0)

What I would really like is to have two threads - one that does blocking input
waiting for a character, and one that examines an output queue and transmits
the stuff it finds.

You can't read and write with the same stdio file object
at the same time. Odd things tend to happen if you try.

You need to open *two* file objects, one for reading
and one for writing:

fr = open("/dev/ttyS0","rb",0)
fw = open("/dev/ttyS0","wb",0)

and give fr to the reading thread and fw to the
writing thread.

You could also try avoiding file objects altogether
and use the raw system calls in the os module. Since
you're not using any buffering, there's little reason
to use the stdio layer. If you do that, you should be
able to use the same file descriptor for reading and
writing without any trouble.

 

[Hendrik van Rooyen]

How about using pyserial? With that, I never had any problems accessing
the the serial ports, and AFAIK no duplex-problems as well. And I
seriously doubt that these are a python-related problem - python only
has a very thin, direct layer above the posix-calls, and doesn't do
anything that would explain your observed behavior. The GIL is not the
issue here either - it won't interfer with blocking IO.

I will have a look at pyserial - have never used it before.

I agree that it is probably not a Python issue, and that the GIL is
irelevant - I was hoping that someone had already travelled the road and
could give me a signpost.

In the meantime I have had another idea which I have also not tried yet,
namely to do independent opens for reading and writing, to give me two file
instances instead of one, and to try with that. I have no idea if it would
make any difference, or even work at all.

My normal stuff works, but I do not like it as it is essentially busy looping
with short sleeps in between. In the eBox, it uses most of the processor just
to move a few bytes of I/O in and out between the serial port and the TCP/IP,
and struggles to do that better than five times a second, while the message
time on the 115200 baud port is only about 2 milliseconds.

- Hendrik

 

[Hendrik van Rooyen]

You can't read and write with the same stdio file object
at the same time. Odd things tend to happen if you try.

You need to open *two* file objects, one for reading
and one for writing:

fr = open("/dev/ttyS0","rb",0)
fw = open("/dev/ttyS0","wb",0)

and give fr to the reading thread and fw to the
writing thread.

Does this actually work without somehow falling foul of the fact that
the /dev/ttyS0 is only one thing? - I know that there is no physical reason
for not being able to go in and out at the same time - in my embedded stuff I
do that routinely - but that is all assembler running on bare metal so it is
under my own control.

You could also try avoiding file objects altogether
and use the raw system calls in the os module. Since
you're not using any buffering, there's little reason
to use the stdio layer. If you do that, you should be
able to use the same file descriptor for reading and
writing without any trouble.

Thanks for this - I now have my weekend cut out for me...

- Hendrik

 

[exarkun]

I will have a look at pyserial - have never used it before.

I agree that it is probably not a Python issue, and that the GIL is
irelevant - I was hoping that someone had already travelled the road
and
could give me a signpost.

In the meantime I have had another idea which I have also not tried
yet,
namely to do independent opens for reading and writing, to give me two
file
instances instead of one, and to try with that. I have no idea if it
would
make any difference, or even work at all.

My normal stuff works, but I do not like it as it is essentially busy
looping
with short sleeps in between. In the eBox, it uses most of the
processor just
to move a few bytes of I/O in and out between the serial port and the
TCP/IP,
and struggles to do that better than five times a second, while the
message
time on the 115200 baud port is only about 2 milliseconds.

One strategy you might employ to get rid of the busy looping is to use
Twisted and its serial port support. This also addresses the full-
duplex issue you've raised.

Jean-Paul

 

[Diez B. Roggisch]

Does this actually work without somehow falling foul of the fact that
the /dev/ttyS0 is only one thing? - I know that there is no physical reason
for not being able to go in and out at the same time - in my embedded stuff I
do that routinely - but that is all assembler running on bare metal so it is
under my own control.


Thanks for this - I now have my weekend cut out for me...

You should *really* just use pyserial. No hassle, instant satisfaction.

Diez

 

[exarkun]

There are no such full-dulex issues.

There was a perceived issues. Obviously it's possible to do full-duplex
with Linux's serial port support (and all the other major platforms too,
as far as I know), as long as you know how. Twisted makes the how a
lot simpler.

Jean-Paul

 

[Terry Reedy]

You can't read and write with the same stdio file object
at the same time. Odd things tend to happen if you try.

I believe the C standard specifies that the behavior of mixed reads and
writes is undefined without intervening seek and/or flush, even if the
seek is ignored (as it is on some Unix systems). With two threads, the
timing is undetermined.

 

[Hendrik van Rooyen]

One strategy you might employ to get rid of the busy looping is to use
Twisted and its serial port support. This also addresses the full-
duplex issue you've raised.

I know - vaguely - about twisted and I have been dancing around the fire, not
really ready to put the time in to understand it properly. Looks like the
time has come though - my weekend is really going to hell.

I am now going to make a loopback connector and start playing.

Thanks to everybody for the feedback.

- Hendrik

 

[Hendrik van Rooyen]

That should work (and shouldn't make any difference)


What platform are you using? I suppose it's possible that
there's something broken in the serial driver for that
particular hardware.

Your experience seems to be exactly the opposite to mine - you are saying it
should "just work" and I am seeing half duplex functionality.

I have seen this on my development machine which is a dual processor of some
gigs running SuSe Linux 10.3, as well as on the other end of a the scale -
the eBox (a 400MHz 486 without floating point with 128 Mb of memory) running
Slackware.

Maybe it is in the way I set the port up, because that is the common thing.
What I do is this:

reterror = os.system('stty -F /dev/ttyS0 sane 115200 cread clocal raw -echo')

It does not seem to make a difference if I do this before or after opening the
port.

Any comments from a Linux Guru?

- Hendrik

 

[Hendrik van Rooyen]

You should *really* just use pyserial. No hassle, instant satisfaction.

I have downloaded and had a quick look, and I see it is based on the
standard library's serial.Serial class - another battery that I have not used
before. And I see that serial.Serial looks like it uses os. calls, which is
one of the things Greg mentioned. Curioser and Curioser.

There was one thing I saw in a quick read of pyserial that I did not like as
I cannot understand why it is done - if a timeout is set to less than a
tenth of a second, then it is changed to be a tenth. - In a polling protocol
that will limit you to poll only ten terminals a second, or less, and is a
very long time if a message takes only a couple of millis to send.

I am getting there - this time around I want to kill this problem dead because
I seem to keep doing something wrong somewhere and I want to understand what
it is and stop doing it.

- Hendrik

 

[Hendrik van Rooyen]

There are no such full-dulex issues.

I will put an example together as soon as I have finished reading and
answering the mail - maybe I am crazy and chasing angels.

- Hendrik

 

[Hendrik van Rooyen]

8<---------------------------------------------------------------

Doh! It didn't even occur to me that somebody would use python
"file" objects for serial ports, and I completely overlooked
the fact that the OP was doing that.

In short: don't do that -- it just messes things up

*grin*

All right that makes me feel better - you were so adamant that there is no
problem that I was starting to doubt my sanity. - So I hereby cancel the
promise I have just made to put an example together. - It is no longer
needed.

Do not use Python file objects. Use the underlying file
descriptors: os.open(), os.read(), os.write(). That will
almost certainly solve your problems.

If you want examples of os.xxxxx() usage, below is the
PosixSerial.py module that I use for Linux-only applications.
For cross-platform work, use pyserial (whose Posix support is
based on the code below).

8< ---------------------------------PosixSerial.py----------------------------

Thanks that looks, on first inspection, similar to the serialposix.py module
in the stdlib, but less cluttered.

- Hendrik

 

[Hendrik van Rooyen]

I believe the C standard specifies that the behavior of mixed reads and
writes is undefined without intervening seek and/or flush, even if the
seek is ignored (as it is on some Unix systems). With two threads, the
timing is undetermined.

For a serial port, flush on write makes some sense, but seek is complete
nonsense because it is undefined, and besides- the point you try to seek to
may never come around. So the message I am getting loud and clear is that
the basic thing I am doing wrong is to use the ordinary python open() instead
of os.open().

As for the timing in two threads - Yes you are right, but there is not a lot
one can do about it - The right solution depends to a large extent on what
you are doing - for instance, if you are writing a polling protocol (such as
Burroughs poll-select, or Uniscope), then you want a loop that transmits
something, and waits for an answer or time out. This is essentially half
duplex, and in a high level language the natural structure to write this is
in one thread. On the other hand, if you are writing a sliding window type
protocol that is capable of pouring stuff into a link asynchronously from
both ends, then the natural way to do it is to use two threads - one to
handle incoming stuff, and the other to squirt out the data that must go out.
If, as is true in my case, the source of outgoing data and the sink for
incoming data is a TCP/IP socket, then one can accomplish this with blocking
I/O quite efficiently, provided you have a third thread looking after overall
timing Issues. For such a case, the timing is essentially determined by the
flow of the data (provided of course that you can keep up with the link
speed). When one introduces another variable into the equation, namely the
requirement to do a transmission at least every n milliseconds, (a feel-good
keepalive) then you need a time out on the sources, so that you can either do
a transmission or raise an alarm because a reporting period was missed. So
then you are back at a loop waiting for input or timeout, and doing a
transmission afterwards. Only now there are two of them, facing in opposite
directions.

I think this sort of thing is better written at a lower level where one has
access to the interrupts from the ports, as well as a timer interrupt to
handle timing and timeout issues. But that is a lot of work, so I make do
with python.

- Hendrik

 

[greg]

I believe the C standard specifies that the behavior of mixed reads and
writes is undefined without intervening seek and/or flush, even if the
seek is ignored (as it is on some Unix systems). With two threads, the
timing is undetermined.

It's also possible that the stdio object is being locked
while one of the threads is using it, which would also
account for the observed half-duplex behaviour.

Another good reason to steer clear of file objects!

 

[Michael Ströder]

In the past, on this group, I have made statements that said that on Linux,
the serial port handling somehow does not allow transmitting and receiving at
the same time, and nobody contradicted me.

Despite all the good comments here by other skilled people I'd recommend to
determine whether the transmission line to the devices accessed support full
duplex.

My knowledge is a bit rusty on this topic. But I vaguely remember having to
deal with symmetric two-wire connections (RS-485) which were definitely
limited to half-duplex by the wire. So the PC hardware was a normal serial
port with the usual UART hardware device but the transmission protocols were
limited to half-duplex.

Ciao, Michael.

 

[Hendrik van Rooyen]

Despite all the good comments here by other skilled people I'd recommend to
determine whether the transmission line to the devices accessed support
full duplex.

My knowledge is a bit rusty on this topic. But I vaguely remember having to
deal with symmetric two-wire connections (RS-485) which were definitely
limited to half-duplex by the wire. So the PC hardware was a normal serial
port with the usual UART hardware device but the transmission protocols
were limited to half-duplex.

You raise a good point, that is probably not well known amongst the youngsters
here, as simple serial multidropping has gone out of fashion.

There is nothing wrong with your memory as far as RS-485 goes - you have
to "turn the line around", same as for *shudder* Burroughs TDI (Two Wire
Direct Interface). Otherwise, if two or more parties talk at once you have
cacophony. An RS-422 link is to some extent worse, as it is capable of full
duplex, but the slaves cannot hear each other, so they have to listen and
play very nicely with the master.

This instance Is not one of those, thank heaven - I am on both sides of the
link - once in the eBox in python, and on the other side there is just one
Dallas chip - a fast (30 Mhz single cycle) 8051 lookalike that I programmed
in assembler. It is a thing that does discrete I/O that we have made for a
customer. The link in between is just RS-232 receive and transmit without
hardware flow control or anything fancy. This is why I was so certain that
there was something wrong in my python part, because I could use the second
port on the Dallas to do monitoring, by spewing stuff out into Hyper
Terminal.

- Hendrik

 

[Hendrik van Rooyen]

Are you using python file operations open/read/write or OS
file-descriptor operations os.open/os.read/os.write?

The former - that seems to be the source of my trouble.

I have now written a little test that uses serial.Serial and it works a treat.

I am still confused about pyserial and serial - I found serial in my
distribution library, (on the SuSe machine, not on the 2.5 in Slackware) but
I had to download pyserial. I see that you were the the original author.
Thank you for letting this stuff loose in the wild.

- Hendrik

 

[John Nagle]

Absolutely false.

All standard PC serial ports are full-duplex devices.

Here's a program I wrote which uses "pyserial" to drive Baudot teletypes
as full-duplex devices.

https://sourceforge.net/projects/baudotrss/

This uses an input thread and an output thread. It reads RSS feeds and
prints them on antique Teletype machines. (Reuters RSS feeds produce
a classic news ticker. Twitter RSS feeds work but look silly when
hammered out on yellow paper at 45.45 baud.)

You raise a good point, that is probably not well known amongst the youngsters
here, as simple serial multidropping has gone out of fashion.

Actually, no. Dynamixel servos as used on the latest Bioloid robots
are multidrop serial RS-485. But outside the embedded world, nobody uses
that stuff any more. (Embedded is going Ethernet; it's overkill but
works fine and is now cheap.)

John Nagle