Is it possible to use python to get True Full Duplex on a Serial port?

Discussion in 'Python' started by Hendrik van Rooyen, Aug 14, 2009.

  1. In the past, on this group, I have made statements that said that on Linux,
    the serial port handling somehow does not allow transmitting and receiving at
    the same time, and nobody contradicted me.

    I am running into the self same issue again.

    What I normally do is to open the port like this:

    port = open("/dev/ttyS0","r+b",0)

    and then I unblock it with:

    def unblock(f):
    """Given file 'f', sets its unblock flag to true."""

    fcntl.fcntl(f.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

    Then I can write a loop that uses a try-except to see if there are characters
    available, and that examines a queue to see if there is something to
    transmit, to give the appearance of full duplex functionality.

    What I would really like is to have two threads - one that does blocking input
    waiting for a character, and one that examines an output queue and transmits
    the stuff it finds.

    When I try to do this, it does not seem to work - as far as I can see, it is
    as if the underlying implementation is somehow single threaded - if it is
    waiting for a received character, it waits until something comes in before it
    will transmit anything. So if you are talking to a device that does not
    respond, the whole thing freezes up waiting for a character that never comes,
    and nothing is transmitted either, despite the call to
    port.write(somestring). The write blocks, and everything stops, waiting for
    the receipt to finish.

    Is there a way to get full duplex, so that the transmit and receive are
    independent of each other?

    Or are we stuck with a disk-like model that forces a sequence on reads and
    writes?

    - Hendrik
     
    Hendrik van Rooyen, Aug 14, 2009
    #1
    1. Advertising

  2. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    Hendrik van Rooyen schrieb:
    > In the past, on this group, I have made statements that said that on Linux,
    > the serial port handling somehow does not allow transmitting and receiving at
    > the same time, and nobody contradicted me.
    >
    > I am running into the self same issue again.
    >
    > What I normally do is to open the port like this:
    >
    > port = open("/dev/ttyS0","r+b",0)


    How about using pyserial? With that, I never had any problems accessing
    the the serial ports, and AFAIK no duplex-problems as well. And I
    seriously doubt that these are a python-related problem - python only
    has a very thin, direct layer above the posix-calls, and doesn't do
    anything that would explain your observed behavior. The GIL is not the
    issue here either - it won't interfer with blocking IO.

    Diez
     
    Diez B. Roggisch, Aug 14, 2009
    #2
    1. Advertising

  3. Hendrik van Rooyen

    greg Guest

    Re: Is it possible to use python to get True Full Duplex on a Serialport?

    Hendrik van Rooyen wrote:

    > port = open("/dev/ttyS0","r+b",0)
    >
    > What I would really like is to have two threads - one that does blocking input
    > waiting for a character, and one that examines an output queue and transmits
    > the stuff it finds.


    You can't read and write with the same stdio file object
    at the same time. Odd things tend to happen if you try.

    You need to open *two* file objects, one for reading
    and one for writing:

    fr = open("/dev/ttyS0","rb",0)
    fw = open("/dev/ttyS0","wb",0)

    and give fr to the reading thread and fw to the
    writing thread.

    You could also try avoiding file objects altogether
    and use the raw system calls in the os module. Since
    you're not using any buffering, there's little reason
    to use the stdio layer. If you do that, you should be
    able to use the same file descriptor for reading and
    writing without any trouble.

    --
    Greg
     
    greg, Aug 14, 2009
    #3
  4. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 12:54:32 Diez B. Roggisch wrote:

    >
    > How about using pyserial? With that, I never had any problems accessing
    > the the serial ports, and AFAIK no duplex-problems as well. And I
    > seriously doubt that these are a python-related problem - python only
    > has a very thin, direct layer above the posix-calls, and doesn't do
    > anything that would explain your observed behavior. The GIL is not the
    > issue here either - it won't interfer with blocking IO.


    I will have a look at pyserial - have never used it before.

    I agree that it is probably not a Python issue, and that the GIL is
    irelevant - I was hoping that someone had already travelled the road and
    could give me a signpost.

    In the meantime I have had another idea which I have also not tried yet,
    namely to do independent opens for reading and writing, to give me two file
    instances instead of one, and to try with that. I have no idea if it would
    make any difference, or even work at all.

    My normal stuff works, but I do not like it as it is essentially busy looping
    with short sleeps in between. In the eBox, it uses most of the processor just
    to move a few bytes of I/O in and out between the serial port and the TCP/IP,
    and struggles to do that better than five times a second, while the message
    time on the 115200 baud port is only about 2 milliseconds.

    - Hendrik
     
    Hendrik van Rooyen, Aug 14, 2009
    #4
  5. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 14:13:46 greg wrote:

    > You can't read and write with the same stdio file object
    > at the same time. Odd things tend to happen if you try.
    >
    > You need to open *two* file objects, one for reading
    > and one for writing:
    >
    > fr = open("/dev/ttyS0","rb",0)
    > fw = open("/dev/ttyS0","wb",0)
    >
    > and give fr to the reading thread and fw to the
    > writing thread.


    Does this actually work without somehow falling foul of the fact that
    the /dev/ttyS0 is only one thing? - I know that there is no physical reason
    for not being able to go in and out at the same time - in my embedded stuff I
    do that routinely - but that is all assembler running on bare metal so it is
    under my own control.

    >
    > You could also try avoiding file objects altogether
    > and use the raw system calls in the os module. Since
    > you're not using any buffering, there's little reason
    > to use the stdio layer. If you do that, you should be
    > able to use the same file descriptor for reading and
    > writing without any trouble.


    Thanks for this - I now have my weekend cut out for me...

    - Hendrik
     
    Hendrik van Rooyen, Aug 14, 2009
    #5
  6. Hendrik van Rooyen

    Guest

    Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On 01:38 pm, wrote:
    >On Friday 14 August 2009 12:54:32 Diez B. Roggisch wrote:
    >>
    >>How about using pyserial? With that, I never had any problems
    >>accessing
    >>the the serial ports, and AFAIK no duplex-problems as well. And I
    >>seriously doubt that these are a python-related problem - python only
    >>has a very thin, direct layer above the posix-calls, and doesn't do
    >>anything that would explain your observed behavior. The GIL is not the
    >>issue here either - it won't interfer with blocking IO.

    >
    >I will have a look at pyserial - have never used it before.
    >
    >I agree that it is probably not a Python issue, and that the GIL is
    >irelevant - I was hoping that someone had already travelled the road
    >and
    >could give me a signpost.
    >
    >In the meantime I have had another idea which I have also not tried
    >yet,
    >namely to do independent opens for reading and writing, to give me two
    >file
    >instances instead of one, and to try with that. I have no idea if it
    >would
    >make any difference, or even work at all.
    >
    >My normal stuff works, but I do not like it as it is essentially busy
    >looping
    >with short sleeps in between. In the eBox, it uses most of the
    >processor just
    >to move a few bytes of I/O in and out between the serial port and the
    >TCP/IP,
    >and struggles to do that better than five times a second, while the
    >message
    >time on the 115200 baud port is only about 2 milliseconds.


    One strategy you might employ to get rid of the busy looping is to use
    Twisted and its serial port support. This also addresses the full-
    duplex issue you've raised.

    Jean-Paul
     
    , Aug 14, 2009
    #6
  7. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    Hendrik van Rooyen schrieb:
    > On Friday 14 August 2009 14:13:46 greg wrote:
    >
    >> You can't read and write with the same stdio file object
    >> at the same time. Odd things tend to happen if you try.
    >>
    >> You need to open *two* file objects, one for reading
    >> and one for writing:
    >>
    >> fr = open("/dev/ttyS0","rb",0)
    >> fw = open("/dev/ttyS0","wb",0)
    >>
    >> and give fr to the reading thread and fw to the
    >> writing thread.

    >
    > Does this actually work without somehow falling foul of the fact that
    > the /dev/ttyS0 is only one thing? - I know that there is no physical reason
    > for not being able to go in and out at the same time - in my embedded stuff I
    > do that routinely - but that is all assembler running on bare metal so it is
    > under my own control.
    >
    >> You could also try avoiding file objects altogether
    >> and use the raw system calls in the os module. Since
    >> you're not using any buffering, there's little reason
    >> to use the stdio layer. If you do that, you should be
    >> able to use the same file descriptor for reading and
    >> writing without any trouble.

    >
    > Thanks for this - I now have my weekend cut out for me...


    You should *really* just use pyserial. No hassle, instant satisfaction.

    Diez
     
    Diez B. Roggisch, Aug 14, 2009
    #7
  8. Hendrik van Rooyen

    Guest

    Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On 02:19 pm, invalid@invalid wrote:
    >On 2009-08-14, <>
    >wrote:
    >>One strategy you might employ to get rid of the busy looping
    >>is to use Twisted and its serial port support. This also
    >>addresses the full- duplex issue you've raised.

    >
    >There are no such full-dulex issues.


    There was a perceived issues. Obviously it's possible to do full-duplex
    with Linux's serial port support (and all the other major platforms too,
    as far as I know), as long as you know how. :) Twisted makes the how a
    lot simpler.

    Jean-Paul
     
    , Aug 14, 2009
    #8
  9. Hendrik van Rooyen

    Terry Reedy Guest

    Re: Is it possible to use python to get True Full Duplex on a Serialport?

    greg wrote:

    > You can't read and write with the same stdio file object
    > at the same time. Odd things tend to happen if you try.


    I believe the C standard specifies that the behavior of mixed reads and
    writes is undefined without intervening seek and/or flush, even if the
    seek is ignored (as it is on some Unix systems). With two threads, the
    timing is undetermined.
     
    Terry Reedy, Aug 15, 2009
    #9
  10. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 15:58:37 wrote:

    > One strategy you might employ to get rid of the busy looping is to use
    > Twisted and its serial port support. This also addresses the full-
    > duplex issue you've raised.


    I know - vaguely - about twisted and I have been dancing around the fire, not
    really ready to put the time in to understand it properly. Looks like the
    time has come though - my weekend is really going to hell.

    I am now going to make a loopback connector and start playing.

    Thanks to everybody for the feedback.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #10
  11. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 16:19:04 Grant Edwards wrote:
    > On 2009-08-14, Hendrik van Rooyen <> wrote:
    > > In the meantime I have had another idea which I have also not tried yet,
    > > namely to do independent opens for reading and writing, to give me two
    > > file instances instead of one, and to try with that. I have no idea if
    > > it would make any difference, or even work at all.

    >
    > That should work (and shouldn't make any difference)
    >
    > > My normal stuff works, but I do not like it as it is
    > > essentially busy looping with short sleeps in between. In the
    > > eBox, it uses most of the processor just to move a few bytes
    > > of I/O in and out between the serial port and the TCP/IP, and
    > > struggles to do that better than five times a second, while
    > > the message time on the 115200 baud port is only about 2
    > > milliseconds.

    >
    > What platform are you using? I suppose it's possible that
    > there's something broken in the serial driver for that
    > particular hardware.


    Your experience seems to be exactly the opposite to mine - you are saying it
    should "just work" and I am seeing half duplex functionality.

    I have seen this on my development machine which is a dual processor of some
    gigs running SuSe Linux 10.3, as well as on the other end of a the scale -
    the eBox (a 400MHz 486 without floating point with 128 Mb of memory) running
    Slackware.

    Maybe it is in the way I set the port up, because that is the common thing.
    What I do is this:

    reterror = os.system('stty -F /dev/ttyS0 sane 115200 cread clocal raw -echo')

    It does not seem to make a difference if I do this before or after opening the
    port.

    Any comments from a Linux Guru?

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #11
  12. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 16:03:22 Diez B. Roggisch wrote:

    > You should *really* just use pyserial. No hassle, instant satisfaction.


    :) I have downloaded and had a quick look, and I see it is based on the
    standard library's serial.Serial class - another battery that I have not used
    before. And I see that serial.Serial looks like it uses os. calls, which is
    one of the things Greg mentioned. Curioser and Curioser.

    There was one thing I saw in a quick read of pyserial that I did not like as
    I cannot understand why it is done - if a timeout is set to less than a
    tenth of a second, then it is changed to be a tenth. - In a polling protocol
    that will limit you to poll only ten terminals a second, or less, and is a
    very long time if a message takes only a couple of millis to send.

    I am getting there - this time around I want to kill this problem dead because
    I seem to keep doing something wrong somewhere and I want to understand what
    it is and stop doing it.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #12
  13. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 16:19:36 Grant Edwards wrote:
    > On 2009-08-14, <> wrote:
    > > One strategy you might employ to get rid of the busy looping
    > > is to use Twisted and its serial port support. This also
    > > addresses the full- duplex issue you've raised.

    >
    > There are no such full-dulex issues.


    I will put an example together as soon as I have finished reading and
    answering the mail - maybe I am crazy and chasing angels.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #13
  14. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Friday 14 August 2009 16:28:26 Grant Edwards wrote:
    > On 2009-08-14, greg <> wrote:
    > > Hendrik van Rooyen wrote:

    8<---------------------------------------------------------------

    > Doh! It didn't even occur to me that somebody would use python
    > "file" objects for serial ports, and I completely overlooked
    > the fact that the OP was doing that.
    >
    > In short: don't do that -- it just messes things up


    *grin*

    All right that makes me feel better - you were so adamant that there is no
    problem that I was starting to doubt my sanity. - So I hereby cancel the
    promise I have just made to put an example together. - It is no longer
    needed.

    >
    > Do not use Python file objects. Use the underlying file
    > descriptors: os.open(), os.read(), os.write(). That will
    > almost certainly solve your problems.
    >
    > If you want examples of os.xxxxx() usage, below is the
    > PosixSerial.py module that I use for Linux-only applications.
    > For cross-platform work, use pyserial (whose Posix support is
    > based on the code below).
    >


    8< ---------------------------------PosixSerial.py----------------------------

    Thanks that looks, on first inspection, similar to the serialposix.py module
    in the stdlib, but less cluttered.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #14
  15. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Saturday 15 August 2009 04:03:42 Terry Reedy wrote:
    > greg wrote:
    > > You can't read and write with the same stdio file object
    > > at the same time. Odd things tend to happen if you try.

    >
    > I believe the C standard specifies that the behavior of mixed reads and
    > writes is undefined without intervening seek and/or flush, even if the
    > seek is ignored (as it is on some Unix systems). With two threads, the
    > timing is undetermined.


    For a serial port, flush on write makes some sense, but seek is complete
    nonsense because it is undefined, and besides- the point you try to seek to
    may never come around. So the message I am getting loud and clear is that
    the basic thing I am doing wrong is to use the ordinary python open() instead
    of os.open().

    As for the timing in two threads - Yes you are right, but there is not a lot
    one can do about it - The right solution depends to a large extent on what
    you are doing - for instance, if you are writing a polling protocol (such as
    Burroughs poll-select, or Uniscope), then you want a loop that transmits
    something, and waits for an answer or time out. This is essentially half
    duplex, and in a high level language the natural structure to write this is
    in one thread. On the other hand, if you are writing a sliding window type
    protocol that is capable of pouring stuff into a link asynchronously from
    both ends, then the natural way to do it is to use two threads - one to
    handle incoming stuff, and the other to squirt out the data that must go out.
    If, as is true in my case, the source of outgoing data and the sink for
    incoming data is a TCP/IP socket, then one can accomplish this with blocking
    I/O quite efficiently, provided you have a third thread looking after overall
    timing Issues. For such a case, the timing is essentially determined by the
    flow of the data (provided of course that you can keep up with the link
    speed). When one introduces another variable into the equation, namely the
    requirement to do a transmission at least every n milliseconds, (a feel-good
    keepalive) then you need a time out on the sources, so that you can either do
    a transmission or raise an alarm because a reporting period was missed. So
    then you are back at a loop waiting for input or timeout, and doing a
    transmission afterwards. Only now there are two of them, facing in opposite
    directions.

    I think this sort of thing is better written at a lower level where one has
    access to the interrupts from the ports, as well as a timer interrupt to
    handle timing and timeout issues. But that is a lot of work, so I make do
    with python.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #15
  16. Hendrik van Rooyen

    greg Guest

    Re: Is it possible to use python to get True Full Duplex on a Serialport?

    Terry Reedy wrote:

    > I believe the C standard specifies that the behavior of mixed reads and
    > writes is undefined without intervening seek and/or flush, even if the
    > seek is ignored (as it is on some Unix systems). With two threads, the
    > timing is undetermined.


    It's also possible that the stdio object is being locked
    while one of the threads is using it, which would also
    account for the observed half-duplex behaviour.

    Another good reason to steer clear of file objects!

    --
    Greg
     
    greg, Aug 15, 2009
    #16
  17. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    Hendrik van Rooyen wrote:
    > In the past, on this group, I have made statements that said that on Linux,
    > the serial port handling somehow does not allow transmitting and receiving at
    > the same time, and nobody contradicted me.


    Despite all the good comments here by other skilled people I'd recommend to
    determine whether the transmission line to the devices accessed support full
    duplex.

    My knowledge is a bit rusty on this topic. But I vaguely remember having to
    deal with symmetric two-wire connections (RS-485) which were definitely
    limited to half-duplex by the wire. So the PC hardware was a normal serial
    port with the usual UART hardware device but the transmission protocols were
    limited to half-duplex.

    Ciao, Michael.
     
    Michael Ströder, Aug 15, 2009
    #17
  18. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Saturday 15 August 2009 14:40:35 Michael Ströder wrote:
    > Hendrik van Rooyen wrote:
    > > In the past, on this group, I have made statements that said that on
    > > Linux, the serial port handling somehow does not allow transmitting and
    > > receiving at the same time, and nobody contradicted me.

    >
    > Despite all the good comments here by other skilled people I'd recommend to
    > determine whether the transmission line to the devices accessed support
    > full duplex.
    >
    > My knowledge is a bit rusty on this topic. But I vaguely remember having to
    > deal with symmetric two-wire connections (RS-485) which were definitely
    > limited to half-duplex by the wire. So the PC hardware was a normal serial
    > port with the usual UART hardware device but the transmission protocols
    > were limited to half-duplex.


    You raise a good point, that is probably not well known amongst the youngsters
    here, as simple serial multidropping has gone out of fashion.

    There is nothing wrong with your memory as far as RS-485 goes - you have
    to "turn the line around", same as for *shudder* Burroughs TDI (Two Wire
    Direct Interface). Otherwise, if two or more parties talk at once you have
    cacophony. An RS-422 link is to some extent worse, as it is capable of full
    duplex, but the slaves cannot hear each other, so they have to listen and
    play very nicely with the master.

    This instance Is not one of those, thank heaven - I am on both sides of the
    link - once in the eBox in python, and on the other side there is just one
    Dallas chip - a fast (30 Mhz single cycle) 8051 lookalike that I programmed
    in assembler. It is a thing that does discrete I/O that we have made for a
    customer. The link in between is just RS-232 receive and transmit without
    hardware flow control or anything fancy. This is why I was so certain that
    there was something wrong in my python part, because I could use the second
    port on the Dallas to do monitoring, by spewing stuff out into Hyper
    Terminal.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #18
  19. Re: Is it possible to use python to get True Full Duplex on a Serialport?

    On Saturday 15 August 2009 16:25:03 Grant Edwards wrote:

    >
    > Are you using python file operations open/read/write or OS
    > file-descriptor operations os.open/os.read/os.write?


    The former - that seems to be the source of my trouble.

    I have now written a little test that uses serial.Serial and it works a treat.

    I am still confused about pyserial and serial - I found serial in my
    distribution library, (on the SuSe machine, not on the 2.5 in Slackware) but
    I had to download pyserial. I see that you were the the original author.
    Thank you for letting this stuff loose in the wild.

    - Hendrik
     
    Hendrik van Rooyen, Aug 15, 2009
    #19
  20. Hendrik van Rooyen

    John Nagle Guest

    Re: Is it possible to use python to get True Full Duplex on a Serialport?

    Hendrik van Rooyen wrote:
    > On Saturday 15 August 2009 14:40:35 Michael Ströder wrote:
    >> Hendrik van Rooyen wrote:
    >>> In the past, on this group, I have made statements that said that on
    >>> Linux, the serial port handling somehow does not allow transmitting and
    >>> receiving at the same time, and nobody contradicted me.


    Absolutely false.

    >> Despite all the good comments here by other skilled people I'd recommend to
    >> determine whether the transmission line to the devices accessed support
    >> full duplex.


    All standard PC serial ports are full-duplex devices.

    Here's a program I wrote which uses "pyserial" to drive Baudot teletypes
    as full-duplex devices.

    https://sourceforge.net/projects/baudotrss/

    This uses an input thread and an output thread. It reads RSS feeds and
    prints them on antique Teletype machines. (Reuters RSS feeds produce
    a classic news ticker. Twitter RSS feeds work but look silly when
    hammered out on yellow paper at 45.45 baud.)
    >
    > You raise a good point, that is probably not well known amongst the youngsters
    > here, as simple serial multidropping has gone out of fashion.


    Actually, no. Dynamixel servos as used on the latest Bioloid robots
    are multidrop serial RS-485. But outside the embedded world, nobody uses
    that stuff any more. (Embedded is going Ethernet; it's overkill but
    works fine and is now cheap.)

    John Nagle
     
    John Nagle, Aug 16, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. cyberco

    Splitting a full-duplex socket

    cyberco, Dec 22, 2004, in forum: Java
    Replies:
    31
    Views:
    6,886
  2. Stanislav Tsukrov

    MIDP 2.0 SocketConnection - full duplex

    Stanislav Tsukrov, Mar 5, 2006, in forum: Java
    Replies:
    0
    Views:
    442
    Stanislav Tsukrov
    Mar 5, 2006
  3. Replies:
    3
    Views:
    695
    Greg Ewing
    Mar 31, 2005
  4. Replies:
    2
    Views:
    509
  5. bdb112
    Replies:
    45
    Views:
    1,351
    jazbees
    Apr 29, 2009
Loading...

Share This Page