is there enough information?

[Gerard Flanagan]

### I smell Java burning...

+1 QOTW

"Mistah Kurtz - he dead."

(http://en.wikiquote.org/wiki/Heart_of_Darkness)

;-]

 

[Dennis Lee Bieber]

It runs on my machine in 2.5. First change was to run it
indefinitely, and it's doing so. Still not conclusive evidence that
it's free of deadlocks, nor is that I don't see any by
inspection. ...But it's running.

You can probably (after stripping out all the commented code) figure
out how to convert it to work with the "with" statement... For the most
part, the acquirestep() and releasenext() become the entry/exit
functions.

Second, thanks for picking better names for the identifiers. Mine
always suck.

I just picked names that seemed to represent what the pieces
were really doing.

Third. StepLock.acquirestep can use notifyAll: wake all of them up,
and all but one go back to sleep. However, the current implementation
wakes them up in order wait was called, so you do save some time here,
(Even though the docs say otherwise). The while self._step!= step is
suspicious, but the alternative is an -sequence- of locks, the penalty
is only large for high step lengths, and it's not exactly polling-- so
maybe that's just the purist in me.

The order of the wake up is not, I believe, guaranteed by the
implementation -- if it keeps waiting tasks on a FIFO queue, yes, it
would be in the order originally entered… but it could just be a list on
which "random.choice()" is called <G>.

I'll agree that notifyAll() might be more efficient, in that
those waits that DO NOT find the step level is the one they want would
NOT need to do a notify() before returning to the wait() -- they'd just
go directly back to wait() which frees the condition lock, and the next
one that had been notified (by the All) would get its chance. I wanted
to ensure that no wait could have missed a wakeup. [call it paranoia --
and my first real work with Condition in Python]

Fourth: wait is a private member, unless you're enabling the
functionality of waiting for the next change in step by another
thread. notifyAll isn't used anywhere. condition is also private, I
believe.

If you look at the documentation for threading.Condition, you
will see that I have supplied methods duplicating all the base methods
of Condition. I'd originally tried to have StepLock inherit from
Condition, but the source code reveals that Condition is not a class,
directly, but a utility function that returns an instance of a "private"
_Condition class… If I'd been able to inherit, only those methods that
needed custom action would have been defined. So, on that basis,
StepLock is still a Condition lock, just with a few added methods to
control the step level.

Fifth: My webreader gives me this information: "Note: The author of
this message requested that it not be archived. This message will be
removed from Groups in 6 days (Mar 7, 12:55 am)." Curious if you
intentially have it. It could be nice to retrieve this-- same with
Stage.

History… Classical news servers used to expire articles after
some period of time (binary groups on some servers expired after one
day, text after a week to a month). When Google came along and announced
it was going to keep archives indefinitely, many people started adding a
"x-noarchive: yes" header to their usenet posts. I'm one of those.

Sixth: If I comment out fmain:worktask.slock.release() and
threadbody:worker.slock.release() there is no effect. It's bizarre.
Is it a requirement to use them?

Hand analysis of the code seems to require them -- and so would
the documentation. notify() itself does NOT release the lock the
notifying process holds.

Seventh: What is the advantage of using a Condition over an Event?

An Event only wakes everyone up -- at once. If you need to
access/modify some shared data you would still need to wrap that in a
lock. Furthermore, the event stays "set" until some thread clears it --
you could have a race condition if multiple threads wake up, then try to
clear the event -- especially if one thread clears, does something, and
sets it before the next even runs; and that later thread (remember, it
was just woke up, so thinks it needs to clear the event itself… and does
clear the /second/ set, before any threads come to wait for it.

A Condition automatically locks; even notifyAll() might signal
all threads but only one at a time will get the related lock and be
allowed to proceed. With notifyAll() a simple release() would let the
next one that had been notified proceed, but threads that hit a wait()
after the notifyAll are not included.

You could look on a Condition using notifyAll as an Event where
the threads that had been waiting are still forced into sequential turns
on a lock.

A silly, not quite correct simile: visualize a two-lane street
broken by a narrow one-lane bridge.

A cluster of cars (threads) arrives on each end and stop
(Event.wait). Somebody on the bridge raises a flag (sets the Event) --
and all cars try to get onto the bridge at once.

Compare to: a cluster of cars arrives on each end and stops
(Condition.wait). Somebody (bridge operator) picks a car at random and
signals (gives that car a flag, maybe) just that car to cross (notify);
when it is across, it picks the next car at random (another notify) etc.
until all cars have crossed (and the bridge operator is alone to start
the notify sequence over again)

Using notifyAll is more like: a cluster of cars arrives at each
end and stops AND shuts down their engines (Condition.wait). Somebody
raises a flag (notifyAll) and then lowers it again, and all waiting cars
start their engines, and one picked at random gets to go across; when it
is across another car with running engine is chosen to cross -- but new
cars that arrive must shut down their engines until they see a raised
flag (another notifyAll)
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Dennis Lee Bieber]

I'm not sure, but you seem to be implying that the only way to use Windows'
asynchronous I/O APIs is with threads. Actually, it is possible (and Twisted
allows you) to use these as well without writing a threaded application.

I only pointed out that, on Windows, one can not use the common
/select()/ function with files. And one rarely sees examples of coding a
Twisted-style (emphasis on style) asynchronous callback system mixing
files and network sockes using the Windows-specific API.

If using threads, the Windows asynchronous I/O isn't needed... let
the thread block until the I/O completes, then transfer the data (or a
message that the data is available) back to the main processing
thread...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[David Bolen]

I only pointed out that, on Windows, one can not use the common
/select()/ function with files. And one rarely sees examples of coding a
Twisted-style (emphasis on style) asynchronous callback system mixing
files and network sockes using the Windows-specific API.

If using threads, the Windows asynchronous I/O isn't needed... let
the thread block until the I/O completes, then transfer the data (or a
message that the data is available) back to the main processing
thread...

You're probably right that it's rare, but when needed, using the
Windows asynchronous/overlapping API can provide a better solution
than blocking threads depending on the needs at hand, and without
involving any callbacks or Twisted-style programming.

An example of mine is high performance serial port handling as part of
a custom FHSS wireless adapter with a serial port interface to the PC.
In this case, minimizing I/O latency was crucial since delays could
mean missing a broadcast timeslot (about 15ms) on the wireless
network. A serial port isn't a disk file, but certainly a "file" in
the context of Windows handles.

Early implementations used independent threads for reading/writing to
the serial port and blocking during such operations, but that turned
out to have an undesirable amount of latency, and was also difficult
to interrupt when the threads were in a blocked condition.

Instead I created a single thread that had a loop using overlapped I/O
simultaneously in each direction as well as native Windows event
objects for aborting or signaling that there was additional data to be
written (the pending read I/O handled the read case). The main loop
was just a WaitForMultipleObjects to handle any of the I/O completion
indications, requests for more I/O or aborts. It was very high
performing (low latency) with low CPU usage - measurably better than a
multi-threaded version.

Communication with the rest of the application was through a
thread-safe bi-directional buffer object, also using native Win32
event objects. It worked similar to a queue, but by using the native
event objects I didn't have the performance inefficiencies for reads
with timeouts of the Python objects. The underlying Python primitives
don't have the timeout capability built in, so reads with timeouts get
implemented through checks for data interspersed with increasing
sleeps, which adds unnecessary latency.

Anyway, it worked extremely well, and was a much better fit for my
needs than a multi-threaded version with blocking I/O, without it
having to be Twisted-style.

-- David

 

[castironpi]

You're probably right that it's rare, but when needed, using the
Windows asynchronous/overlapping API can provide a better solution
than blocking threads depending on the needs at hand, and without
involving any callbacks or Twisted-style programming.

An example of mine is high performance serial port handling as part of
a custom FHSS wireless adapter with a serial port interface to the PC.
In this case, minimizing I/O latency was crucial since delays could
mean missing a broadcast timeslot (about 15ms) on the wireless
network.  A serial port isn't a disk file, but certainly a "file" in
the context of Windows handles.

Early implementations used independent threads for reading/writing to
the serial port and blocking during such operations, but that turned
out to have an undesirable amount of latency, and was also difficult
to interrupt when the threads were in a blocked condition.

Instead I created a single thread that had a loop using overlapped I/O
simultaneously in each direction as well as native Windows event
objects for aborting or signaling that there was additional data to be
written (the pending read I/O handled the read case).  The main loop
was just a WaitForMultipleObjects to handle any of the I/O completion
indications, requests for more I/O or aborts.  It was very high
performing (low latency) with low CPU usage - measurably better than a
multi-threaded version.

Communication with the rest of the application was through a
thread-safe bi-directional buffer object, also using native Win32
event objects.  It worked similar to a queue, but by using the native
event objects I didn't have the performance inefficiencies for reads
with timeouts of the Python objects.  The underlying Python primitives
don't have the timeout capability built in, so reads with timeouts get
implemented through checks for data interspersed with increasing
sleeps, which adds unnecessary latency.

Anyway, it worked extremely well, and was a much better fit for my
needs than a multi-threaded version with blocking I/O, without it
having to be Twisted-style.

-- David- Hide quoted text -

- Show quoted text -

How does it work? From reading threading.py, _Condition.wait()
acquires self.lock() too many times-- that is, once to call wait
("cannot wait on un-aquired lock"), and once after--- are "all
waiters" waiting back at self.acquire, just to make it to
self.notify... and only one at a time at that!? Don't waiters have to
release before another waiter can go?

 

[castironpi]

How does it work?  From reading threading.py, _Condition.wait()

acquires self.lock() too many times-- that is, once to call wait
("cannot wait on un-aquired lock"), and once after--- are "all
waiters" waiting back at self.acquire, just to make it to
self.notify... and only one at a time at that!?  Don't waiters have to
release before another waiter can go?

It's not like Condition().acquire() appends self._lock to
self._waiters *before* delegationing! So... how?