pySerial Windows write problem

[Bob Greschke]

We have some equipment that communicates at 57600 baud RS232. The path from
the PC is USB to a Phillips USB hub, then off of that a TUSB3410 USB/Serial
converter. The driver for the 3410 chip creates a "normal" comm port (like
COM3). There is a C++ program that has no problem talking to the equipment,
but two Python programs have trouble. One is a big, scary, controller
program, and the other, since that is the one I'm having trouble with, is a
while() loop.

The while loop program seems to be able to open and close the serial port
all day. It also seems to be able to open, send "blahblahblah", and close
the port for the same length of time. But writing anything to the port
quickly (not always right away...like maybe 5-6 iterations through the
loop?) causes the

win32file.CloseHandle(self.hComPort)

statement in the 'def close(self)' function of the serialwin32.py file of
the pySerial package to hang for a few seconds, then return. The next
open() attempt fails with the 'could not open port: (995, 'CreateFile', 'The
I/O operation has been aborted because of either a thread exit or an
application request.') exception.

The failure mode with the large program is it opens the port successfully
completes a bunch of back and forth talking, then hangs for a few seconds
when the port is closed, and then can't reopen the port. It never seems to
fail in the middle of a bunch of reads and writes, but only when/after the
port is closed. It can also run for the better part of an hour with no
problems then all of a sudden... Like I said, the C++ program never fails.

I've looked at the settings in the C++ program, and in the serialwin32.py
file and tried to set them the same as best I can, but no luck. Digging
into the win32all stuff gets me lost quite quickly. I see all of the
settings, but have no idea what to try. This is all on WinXP, Python 2.4.1,
the latest pySerial and win32all.

Does this ring any bells with anyone?

Thanks!

Bob

 

[bob]

I forgot to mention that once the Python program(s) fail, THEN the C++
program also fails to opne the port, and the equipment has to be
power-cycled to get things to work again.

Bob

 

[Peter Hansen]

> But writing anything to the port
quickly (not always right away...like maybe 5-6 iterations through the
loop?) causes the

win32file.CloseHandle(self.hComPort)

statement in the 'def close(self)' function of the serialwin32.py file of
the pySerial package to hang for a few seconds, then return.

Are you certain it is this line that is blocking, and not the preceding
line which is a call to SetCommTimeouts()? How did you prove which line
it is? (I don't have an answer to the problem, just wanted to be sure
of the information...)

-Peter

 

[Bob Greschke]

Are you certain it is this line that is blocking, and not the preceding
line which is a call to SetCommTimeouts()? How did you prove which line
it is? (I don't have an answer to the problem, just wanted to be sure of
the information...)

-Peter

Hi!

I had the SetCommTimeouts line commented out. I just have a couple of
prints before and after the CloseHandle line.

I didn't write the C++ program, but it looks like it never closes the serial
port. It opens it when it starts up, then keeps it open until it quits.
Tsk tsk tsk. Sloppy. Maybe they did that to cover up this problem.

Bob

 

[Peter Hansen]

I didn't write the C++ program, but it looks like it never closes the serial
port. It opens it when it starts up, then keeps it open until it quits.
Tsk tsk tsk. Sloppy. Maybe they did that to cover up this problem.

Actually, I'm curious why you don't do the same. I'd call it very
unusual (in my experience) to have a program open and close a serial
port repeatedly. Among other things, this means that the DSR/DTR lines
are toggling high and low repeatedly, and that alone could cause
undesirable behaviour in certain devices.

In none of my own serial-based programs (perhaps a few dozen such to
date) have I ever opened and closed a port other than at startup and
shutdown (just as your C++ program does). Unless you've got a good
reason to do otherwise, if this solves your problem it's certainly the
most direct approach to do so.

-Peter

 

[Bob Greschke]

Actually, I'm curious why you don't do the same. I'd call it very unusual
(in my experience) to have a program open and close a serial port
repeatedly. Among other things, this means that the DSR/DTR lines are
toggling high and low repeatedly, and that alone could cause undesirable
behaviour in certain devices.

I guess I could. It's just that not releasing the port/not KNOWING that the
port has been closed at predictible times is our chief complaint about the
C++ program. As an aside, when I left work I left a version of the
while-loop program running, opening the port, writing to the equipment,
getting a response, and closing the port. It was up to about 180 sucessful
cycles (in a row -- it will stop if it fails). I think it's a hardware
problem.

In none of my own serial-based programs (perhaps a few dozen such to date)
have I ever opened and closed a port other than at startup and shutdown
(just as your C++ program does). Unless you've got a good reason to do
otherwise, if this solves your problem it's certainly the most direct
approach to do so.

One of the serial ports (there are actually two) is used to read some NMEA
sentences from a GPS. It is only rarely read. If it is also opened when
the program starts and kept open would you just dump the buffer and then
read to get the most current info? What happens when the buffer fills up?
The "main" port is just commands sent, responses received kind of traffic.

I'd write a C extension to do the serial stuff, but I don't know a thing
about Windows programming.

Thanks!

 

[Neil Benn]

I guess I could. It's just that not releasing the port/not KNOWING that the
port has been closed at predictible times is our chief complaint about the
C++ program. As an aside, when I left work I left a version of the
while-loop program running, opening the port, writing to the equipment,
getting a response, and closing the port. It was up to about 180 sucessful
cycles (in a row -- it will stop if it fails). I think it's a hardware
problem.

Hmm, keep the port open. One bad thing that can happen is that if you
don;t keep the port open then another program running on the box and nip
in and claim ownership of the port. I agree with Peter, I never
relinquish the port in code unless I have a reason to (ie shutdown or
chnaging the COM port I'm using). I doubt that it is a hardware problem
on your device as the RS232 tandard (I prefer to call it a rumour)
doesn't have any kind of RRP. Even if you are running RTS/CTS or
XON/XOFF then you again shouldn't have a problem becuase once the data
is sent.received then the lines should be back to normal. If you wish
to test this and am not happy with writing C code to test it then maybe
you could try it with something that doens;t use C code such as Java
(with the javax.comm or rxtx extensions)?

One of the serial ports (there are actually two) is used to read some NMEA
sentences from a GPS. It is only rarely read. If it is also opened when
the program starts and kept open would you just dump the buffer and then
read to get the most current info? What happens when the buffer fills up?
The "main" port is just commands sent, responses received kind of traffic.

PySerial doesn;t have any kind of event firing to notify you when data
is available. The way I get round this is to have a loop polling (in a
seperate thread) to see if any data is available (it's a method on the
interface), then read all the data in and fire this off to my
listeners/observers with the read data. That way your buffers will not
fill up. I would also do that on your other port as well to give you a
common way of receiving data. I did this and then started downloading
random stuff of wiki and sending the data from one serial port to
another in huge chunks with no problems.

I'd write a C extension to do the serial stuff, but I don't know a thing
about Windows programming.

Thanks!

I wouldn't bother as well - PySerial works fine for me, as I mentioned
the only thing I don;t like is no evetn firing when data is availabel
but you can code around taht if you wish to.

Cheers,

Neil

--

Neil Benn
Senior Automation Engineer
Cenix BioScience
BioInnovations Zentrum
Tatzberg 47
D-01307
Dresden
Germany

Tel : +49 (0)351 4173 154
e-mail : (e-mail address removed)
Cenix Website : http://www.cenix-bioscience.com

 

[Peter Hansen]

One of the serial ports (there are actually two) is used to read some NMEA
sentences from a GPS. It is only rarely read. If it is also opened when
the program starts and kept open would you just dump the buffer and then
read to get the most current info? What happens when the buffer fills up?
The "main" port is just commands sent, responses received kind of traffic.

Generally, yes, you just dump data received up to the point you are
about to send a new request (if this is a request/response type of
thing). PySerial has a .flush() method of some kind I believe, or you
can just loop as long as inWaiting() isn't False.

If you aren't using hardware or software handshaking, then the buffer
filling up is a non-issue for you or the sender. If you're using
handshaking, then you do have to keep the buffer from filling. Pretty
much all my interesting code runs the PySerial stuff in a separate
thread, reading whenever data is available, and (this is a
simplification) posting it to a Queue which the main thread can read
from as required. In that scenario, you could just have a flag that
causes the receive thread to stop posting stuff to the Queue where you
currently have close the port to prevent data being seen.

I don't recall: is NMEA 0183 asynchronous? Messages can be sent by the
GPS even when you didn't ask for one explicitly? If so, then it's
possible there are problems even with your current approach: what if you
open the port halfway through a packet, and receive only the last few
bytes? If it's synchronous (data sent only in response to your
requests), then none of this is an issue since there will be no traffic
unless you ask for it, so the serial port will be idle between uses.

-Peter

 

[Peter Hansen]

PySerial doesn;t have any kind of event firing to notify you when data
is available. The way I get round this is to have a loop polling (in a
seperate thread) to see if any data is available (it's a method on the
interface), then read all the data in and fire this off to my
listeners/observers with the read data.

On that note, I've got a preliminary version of something I call "Bent"
(think "slightly Twisted") which is focused for now on providing an
asynchronous version of PySerial which is designed around the
event-driven model instead of its current polled scheme.

It shares a number of features with the Twisted-based PySerial port
(also done by Chris Liechti, for the Twisted team, judging by the code
comments) but doesn't require one to adopt the whole Twisted framework.
Basically it provides you with the equivalent of one "reactor" per
thread for PySerial stuff. On Win32 only for now. There are other
limitations too. Not ready for prime time.

On the plus side, it's been letting me convert my serial stuff to be
fully event-driven and with the resulting much lower latencies and
better CPU efficiency, while keeping my code more "traditional" instead
of having to force it entirely into the Twisted point of view.

Just an FYI. And, obviously, to hear back from those interested. I
don't know if this is something that should be contributed to the
PySerial project (it's more of a rewrite than an add-on), or set up as a
new project, or passed around quietly behind the scenes for a while.
No docs yet, no contrived examples (but various pieces of working code
in real daily use). Did I mention it wasn't ready for prime time?

If you've ever been annoyed having to do .read(1) to get characters one
at a time in PySerial, or .read(100) with a timeout, or whatever, this
might be of interest to you...

-Peter

 

[phil]

I use PySerial in a 16 line data collection system
with LOTS of threads, and yes am frustrated by read().
This sounds excellent, keep us updated.

BTW, haven't done any event driven Python except Tkinter.
Would this a class library which would let you
define an event and a handler?

Do you have a one line code example?

Thanks.

 

[Dennis Lee Bieber]

One of the serial ports (there are actually two) is used to read some NMEA
sentences from a GPS. It is only rarely read. If it is also opened when
the program starts and kept open would you just dump the buffer and then
read to get the most current info? What happens when the buffer fills up?
The "main" port is just commands sent, responses received kind of traffic.

Two ports in use?

Best advice I could come up with is to create two threads (one
per port) to handle the I/O, and use Queue to transfer data in/out of
the I/O threads to the main process.

For the GPS side (as I recall, those send messages continuously,
at something like a 1-2second rate) use a global flag to indicate if
data is desired or not; not => read&dump, desired => read&queue to main.
The command port probably should read&queue everything. Main process
would be a loop retrieving data from the queue (use one queue, and
package the data with a source Identifier: ("GPS", data-string), ("CMD",
command-data)

I'd write a C extension to do the serial stuff, but I don't know a thing
about Windows programming.

I had to do serial stuff on a W98 laptop in C++... It was NOT
pretty (and basically came down to the same thing -- running the I/O in
a thread and signaling the main process when data had arrived, using
very low-level w32 os calls). The only "easy" serial port programming on
Windows is the VB MSCOMM control, which does have an event handler (ONE
handler, requiring long case/if logic to determine if it is input/output
or error).

--

 

[Dennis Lee Bieber]

I don't recall: is NMEA 0183 asynchronous? Messages can be sent by the

Asynchronous, messages at something like 1sec intervals (may
depend on how many variations of messages are being sent).

open the port halfway through a packet, and receive only the last few
bytes? If it's synchronous (data sent only in response to your

GPS messages have a recognizable start; something like $GPxxx

One would have to scan for the start of message, and then make
sure to read through to the end...

--

 

[Peter Hansen]

I use PySerial in a 16 line data collection system
with LOTS of threads, and yes am frustrated by read().
This sounds excellent, keep us updated.

BTW, haven't done any event driven Python except Tkinter.
Would this a class library which would let you
define an event and a handler?

Roughly speaking, yes. The primary parent class happens to be called
Handler, and is a threading.Thread subclass which in its run() method
basically sits in a win32 WaitForMultipleObjects() call for various
things to happen, then calls handler routines based on which event fires.

Do you have a one line code example?

One line? No way... this isn't Perl. ;-)


from bent.serial import Driver

class MyDriver(Driver):
def __init__(self, port, baud=9600):
Driver.__init__(self, port, baud=baud, name='iodriver')
self.start() # start the thread

def handleSerialRead(self, data):
print 'read %r' % data

def handleSerialDsr(self, level):
print 'DSR', level

def handleSerialBreak(self):
print 'break detected, ho hum'


Usage for this would be simply: d = MyDriver('COM3') and then sit back
and watch the, uh, fireworks... or at least the print statements.

Anything interesting represents a much more sophisticated subclass which
parses the data as it arrives, of course, and at least in my case then
calls a handlePacket() routine where the fun really begins.

-Peter

 

[bob]

Thanks heaps for the help, guys!

I bit the bullit and am now leaving the serial ports open all of the
time.
I still think it is a driver/hardware problem. When the ports go South
even
a reboot of the laptop the program is running on won't fix it. The
hardware
is all brand new, so it should have some bugs in it. Shouldn't it?

Anyway, here's a short page about the program, just for fun

www.greschke.com/unlinked/pocus01.htm

Thanks!

Bob