Protecting against callbacks queuing up?

[Esben von Buchwald]

Hello

I'm using Python for S60 1.9.7 on my Nokia phone.

I've made a program that gets input from an accelerometer sensor, and
then calculates some stuff and displays the result.

The sensor framework API does a callback to a function defined by me,
every time new data is available.

The problem is, that sometimes, the calculations may take longer than
others, but if the callback is called before the first calculation is
done, it seems like it's queuing up all the callbacks, and execute them
sequentially, afterwards.

What I need, is to simply ignore the callback, if the previous call
hasn't finished it's operations before it's called again.

I thought that this code would do the trick, but it obviously doesn't
help at all, and i can't understand why...

def doCallback(self):
if self.process_busy==False:
self.process_busy=True
self.data_callback()
self.process_busy=False


doCallback is defined as a callback function for the accelerometer
instance, and data_callback does some calculations and show them on the
display of the phone.

What to do? Thanks...

 

[MRAB]

Hello

I'm using Python for S60 1.9.7 on my Nokia phone.

I've made a program that gets input from an accelerometer sensor, and
then calculates some stuff and displays the result.

The sensor framework API does a callback to a function defined by me,
every time new data is available.

The problem is, that sometimes, the calculations may take longer than
others, but if the callback is called before the first calculation is
done, it seems like it's queuing up all the callbacks, and execute them
sequentially, afterwards.

What I need, is to simply ignore the callback, if the previous call
hasn't finished it's operations before it's called again.

I thought that this code would do the trick, but it obviously doesn't
help at all, and i can't understand why...

def doCallback(self):
if self.process_busy==False:
self.process_busy=True
self.data_callback()
self.process_busy=False


doCallback is defined as a callback function for the accelerometer
instance, and data_callback does some calculations and show them on the
display of the phone.

If it is, in fact, queuing the callbacks then that means that it's just
recording that the callback needs to be called, but doesn't actually do
it while a previous one is in effect, ie it'll wait for the current call
to return before doing the next one.

What to do? Thanks...

If you want to try multithreading you could make the callback just
trigger the actual action in another thread. The other thread could wait
for an event, perform the calculation, then clear any pending events
before waiting again.

 

[Dennis Lee Bieber]

I thought that this code would do the trick, but it obviously doesn't
help at all, and i can't understand why...

def doCallback(self):
if self.process_busy==False:
self.process_busy=True
self.data_callback()
self.process_busy=False

It doesn't because it is sequential operation internally; I suspect
the whatever mechanism activates the callback requires any prior
callback to return first, hence it may be queuing the data and calls the
callback only after the callback has returned.

If the Python on the phone(?) supports the threading library,
putting the number cruncher into a thread might help. Uh... since I
presume the number crunching relies upon data from the accelerometer,
you'll probably need to pass that data to the thread -- you wouldn't
want a later callback changing data values in the middle of a
computation, would you?...

{pseudocode}
def worker():
while True:
data = parameterQueue.get()
do_computations_with_data
nextEvent.set()

def callback():
if nextEvent.isSet():
nextEvent.clear()
parameterQueue.put( (current, accelerometer, data) )

It is a perverse use of threading.Event(), since nothing actually
waits for it to /be/ set. In fact, you could actually reverse the
conditional, clear() and set() calls for the same effect. I've used them
to indicate that the worker is "ready" for the next accelerometer event,
and the callback will just return if the worker is not ready.

 

[Hendrik van Rooyen]

Hello

I'm using Python for S60 1.9.7 on my Nokia phone.

I've made a program that gets input from an accelerometer sensor, and
then calculates some stuff and displays the result.

The sensor framework API does a callback to a function defined by me,
every time new data is available.

The problem is, that sometimes, the calculations may take longer than
others, but if the callback is called before the first calculation is
done, it seems like it's queuing up all the callbacks, and execute them
sequentially, afterwards.

What I need, is to simply ignore the callback, if the previous call
hasn't finished it's operations before it's called again.

I thought that this code would do the trick, but it obviously doesn't
help at all, and i can't understand why...

def doCallback(self):
if self.process_busy==False:
self.process_busy=True
self.data_callback()
self.process_busy=False

see if there is an "after" method somewhere.

What you have to do is to "break the link" between the callback
and the processing. Your code above is all in the callback "thread", despite
the fact that you call another function to do the processing.

So if you replace the data_callback() with an after call, it should work as
you expect.

HTH - Hendrik

 

[Esben von Buchwald]

see if there is an "after" method somewhere.

What you have to do is to "break the link" between the callback
and the processing. Your code above is all in the callback "thread", despite
the fact that you call another function to do the processing.

So if you replace the data_callback() with an after call, it should work as
you expect.

HTH - Hendrik

Thanks

I'm new to python, what is an after function and an after call? Couldn't
find excact answer on google...? Do you have a link to some docs?

 

[ryles]

I thought that this code would do the trick, but it obviously doesn't
help at all, and i can't understand why...

     def doCallback(self):
         if self.process_busy==False:
             self.process_busy=True
             self.data_callback()
             self.process_busy=False

doCallback is defined as a callback function for the accelerometer
instance, and data_callback does some calculations and show them on the
display of the phone.

What to do? Thanks...

As Dennis pointed out, it sounds like doCallback() is called serially.
That is, you can think of doCallback() as being called in a loop,
rather than entered multiple times simultaneously. Your 'busy' flag is
never actually checked until after it has already been reset to False.

If the data accessed by the callback as a unique copy and has a
timestamp then a simple approach which avoids multithreading would be
to have your callback ensure that any 'old' data is simply ignored.
This will allow it to quickly catch up to a 'new' event.

 

[Dennis Lee Bieber]

I'm new to python, what is an after function and an after call? Couldn't
find excact answer on google...? Do you have a link to some docs?

They would be particular to whatever GUI/event library is in use.
They aren't part of Python itself.

 

[Hendrik van Rooyen]

Hendrik van Rooyen wrote:

8< -------------- some stuff about an "after" call --------------------------

I'm new to python, what is an after function and an after call? Couldn't
find excact answer on google...? Do you have a link to some docs?

The after call is common in GUI stuff.

Here is a link to nice Tkinter manual - it is the first link if you google
for: tkinter "new mexico tech" shipman

www.nmt.edu/tcc/help/pubs/tkinter/

What after does is that it is like a normal call, except that it is not done
immediately, but after some settable time, hence the "after".

- Hendrik

 

[Esben von Buchwald]

They would be particular to whatever GUI/event library is in use.
They aren't part of Python itself.

This is how the accelerometer is accessed
http://pys60.garage.maemo.org/doc/s60/node59.html

I found this called "after"...
http://pys60.garage.maemo.org/doc/s60/node12.html

would that be usable?

If so, how?

 

[Hendrik van Rooyen]

This is how the accelerometer is accessed
http://pys60.garage.maemo.org/doc/s60/node59.html

I found this called "after"...
http://pys60.garage.maemo.org/doc/s60/node12.html

would that be usable? Probably

If so, how?

This is a guess, for your device, but I suspect
something along these lines:

t = Ao_timer()

cb = t.after(100,thing_that_does_the_work(with_its_arguments))

Lots of assumptions here - the 100 should give you a tenth of a second...
Don't know what the arguments look like, and if you need to pass an instance
(like self) - that would depend on where you are calling it from.

Play and see

You should also be able to cancel the callback like this:

t.cancel(cb)

If you do it before the time out

- Hendrik

 

[Dennis Lee Bieber]

This is how the accelerometer is accessed
http://pys60.garage.maemo.org/doc/s60/node59.html

I found this called "after"...
http://pys60.garage.maemo.org/doc/s60/node12.html

would that be usable?

If so, how?

I suspect the timer is not usable...

However...

def doCallback(self):
if self.process_busy==False:
self.process_busy=True
self.data_callback()
self.process_busy=False

Based on the documentation... I'd suggest...

def doCallback(self):
self.accelerometer.stop_listening()
self.data_callback()
self.accelerometer.start_listening()

with appropriate changes to whatever reference you use for the
accelerometer would solve the problem... IE, when the callback is
triggered, you STOP the accelerometer action, compute results, and then
restart the accelerometer..

 

[Esben von Buchwald]

This is a guess, for your device, but I suspect
something along these lines:

t = Ao_timer()

cb = t.after(100,thing_that_does_the_work(with_its_arguments))

Lots of assumptions here - the 100 should give you a tenth of a second...
Don't know what the arguments look like, and if you need to pass an instance
(like self) - that would depend on where you are calling it from.

Play and see

You should also be able to cancel the callback like this:

t.cancel(cb)

If you do it before the time out

- Hendrik

I don't really get it...

I can see that I should put a t.after(...) around the function that does
the work, when calling it. That delays each call for the given period. I
just tried it out, the console keeps saying
-------------------------------------------------------------------------
Traceback (most recent call last):
File "c:\resource\python25\python25.zip\sensorfw.py", line 499, in
data_cb
File "c:\resource\python25\python25.zip\sensorfw.py", line 160, in
custom_cb
File "e:\python\r3s_contextdata.py", line 79, in acc_filter
self.acc_callback()
File "e:\python\r3s_contextdata.py", line 85, in acc_callback
self.doCallback()
File "e:\python\r3s_contextdata.py", line 47, in doCallback
self.at.after(0.05,self.data_callback)
RuntimeError: Timer pending - cancel first

-------------------------------------------------------------------------

It seems like i should cancel the current counting timer, when a new
call comes in - but how to determine when to cancel etc?

I'm kind of new to all this async python stuff.

 

[Esben von Buchwald]

Based on the documentation... I'd suggest...

def doCallback(self):
self.accelerometer.stop_listening()
self.data_callback()
self.accelerometer.start_listening()

with appropriate changes to whatever reference you use for the
accelerometer would solve the problem... IE, when the callback is
triggered, you STOP the accelerometer action, compute results, and then
restart the accelerometer..

I just tried that - it made the application lock up, leaving the sensors
unusable until next reboot of the phone.

I don't think these start/stop calls are supposed to be called as often
as it does the callback...

 

[Dennis Lee Bieber]

I don't think these start/stop calls are supposed to be called as often
as it does the callback...

Well, http://pys60.garage.maemo.org/doc/s60/node58.html does state:

doc> Open and Listen
doc> Function signature: start_listening()
doc>
doc> Opens the sensor channel and start listening. Returns True on
success and False on failure.
doc>
doc> Stop and Close
doc> Function signature: stop_listening()
doc>
doc> Stop listening to the open channel and close the channel. To start
receiving updates again the start_listening method can be called on the
same sensor object.

The only other thing I could suggest is exactly what is done on:
http://pys60.garage.maemo.org/doc/s60/node59.html

Initialize a counter value to 0, then increment it in the callback,
only doing REAL work every n calls.


def doCallback(self):
if self.count % 35 == 0: #doc says 35 hits/second, so this
self.data_callback() #will run one once per second
self.count += 1

You'll still get that slew of backlogged callbacks that built up
while doing the real processing, but unless self.data_callback() takes
more time than the "35" covers, most of the callbacks will just come in
and exit with an increment.

 

[Dennis Lee Bieber]

Traceback (most recent call last):
File "c:\resource\python25\python25.zip\sensorfw.py", line 499, in
data_cb
File "c:\resource\python25\python25.zip\sensorfw.py", line 160, in
custom_cb
File "e:\python\r3s_contextdata.py", line 79, in acc_filter
self.acc_callback()
File "e:\python\r3s_contextdata.py", line 85, in acc_callback
self.doCallback()
File "e:\python\r3s_contextdata.py", line 47, in doCallback
self.at.after(0.05,self.data_callback)
RuntimeError: Timer pending - cancel first

Don't set the .after() call IN doCallback, set it at the end of
self.data_callback (and in the initialization)

I suspect nothing in the API is designed to slow down the rate at
which the accelerometer queues events. You have to filter them yourself.

Either the example I pseudo-coded in the other message (may appear
before or after this one), or...


def __init__(self...)
....
self.at.after(0.05, self.data_callback)
#note -- if events trigger at 35/sec, that means you get one
#near 0.03 (if that is in fractional seconds); may need a
#longer time if processing takes a while... 0.10 or more

def doCallback(self):
self.data = (self.at.x, self.at.y, self.at.z)
#save most recent values

def data_callback(self):
x, y, z = self.data #copy current values so no chance
#of callback changing parts in the middle
#do stuff with x, y, z
self.at.after(0.05, self.data_callback)


Perhaps you should examine
http://pys60.garage.maemo.org/doc/s60/node55.html

You could set an event filter that does the skip n items counting
(from the other message)

The init would set count to 0; the event() method would do the

if self.count % SKIPSPAN == 0:
self.callback()
self.count += 1

 

[Hendrik van Rooyen]

I don't really get it...

I can see that I should put a t.after(...) around the function that does
the work, when calling it. That delays each call for the given period. I
just tried it out, the console keeps saying
-------------------------------------------------------------------------
Traceback (most recent call last):
File "c:\resource\python25\python25.zip\sensorfw.py", line 499, in
data_cb
File "c:\resource\python25\python25.zip\sensorfw.py", line 160, in
custom_cb
File "e:\python\r3s_contextdata.py", line 79, in acc_filter
self.acc_callback()
File "e:\python\r3s_contextdata.py", line 85, in acc_callback
self.doCallback()
File "e:\python\r3s_contextdata.py", line 47, in doCallback
self.at.after(0.05,self.data_callback)
RuntimeError: Timer pending - cancel first

Hmm - now I am really starting to fly by the seat of my pants - but it looks
as if your problem is that your routine is basically being called faster than
what it can do the processing.

So what I would try is to define a global (you should really acquire a lock,
but forget that for the moment) So lets call this thing running_calculation.

Then you set this in the routine just before the after call, and reset it just
before exiting the calculation routine, and if it is set when you get into
the first routine, then you just exit, without doing the after. This should
have the effect of skipping some calls, to make it all manageable. Then you
should (hopefully) not have the duplication that it looks to me like you are
now having.

Bye the way, make the time short - like 1 - it could also be that the time is
now so long before you start calculating, that you do not have a snowball's
hope in hell to be finished before the next value comes in.

- Hendrik

 

[Esben von Buchwald]

The only other thing I could suggest is exactly what is done on:
http://pys60.garage.maemo.org/doc/s60/node59.html

Initialize a counter value to 0, then increment it in the callback,
only doing REAL work every n calls.


def doCallback(self):
if self.count % 35 == 0: #doc says 35 hits/second, so this
self.data_callback() #will run one once per second
self.count += 1

You'll still get that slew of backlogged callbacks that built up
while doing the real processing, but unless self.data_callback() takes
more time than the "35" covers, most of the callbacks will just come in
and exit with an increment.

Of course I can do that.


But it'll only make a noticable delay EVERY time the user moves, and not
prevent the build up of calls if it doesn't finish within the 35 callbacks.

The point is that I don't know in advance, how long the call will take.
It depends on the amount of data i load and process during the call.
I only know when the calculations have finished, and when they are
called, and think there might be some way to block further callbacks
until the first one returns?

 

[Esben von Buchwald]

Hmm - now I am really starting to fly by the seat of my pants - but it looks
as if your problem is that your routine is basically being called faster than
what it can do the processing.

So what I would try is to define a global (you should really acquire a lock,
but forget that for the moment) So lets call this thing running_calculation.

Then you set this in the routine just before the after call, and reset it just
before exiting the calculation routine, and if it is set when you get into
the first routine, then you just exit, without doing the after. This should
have the effect of skipping some calls, to make it all manageable. Then you
should (hopefully) not have the duplication that it looks to me like you are
now having.

Bye the way, make the time short - like 1 - it could also be that the time is
now so long before you start calculating, that you do not have a snowball's
hope in hell to be finished before the next value comes in.

OK, now things starts to make sense.

You tell me to do something like this?


def doCallback(self):
if self.process_busy==False:
self.process_busy=True
self.at.after(0.01,self.data_callback)
def doneCallback(self):
self.process_busy=False


And the after command will cause python to spawn a new thread, which
runs self.data_callback, which measn that the doCallback will return
immediately, so the next callback from accelerometer can be processed?

And when the self.data_callback() finishes, i'd make it call the
doneCallback() to release my busy flag, right?

I'm gonna try this tomorrow

 

[MRAB]

Of course I can do that.


But it'll only make a noticable delay EVERY time the user moves, and not
prevent the build up of calls if it doesn't finish within the 35 callbacks.

The point is that I don't know in advance, how long the call will take.
It depends on the amount of data i load and process during the call.
I only know when the calculations have finished, and when they are
called, and think there might be some way to block further callbacks
until the first one returns?

You could record when your callback finishes (is about to return) and
then ignore any callback that happens too soon after that:

def doCallback(self):
now = time.time()
if now - self.last_call >= MIN_TIME:
self.data_callback()
now = time.time()
self.last_call = now

 

[MRAB]

You could record when your callback finishes (is about to return) and
then ignore any callback that happens too soon after that:

def doCallback(self):
now = time.time()
if now - self.last_call >= MIN_TIME:
self.data_callback()
now = time.time()
self.last_call = now

After a little more thought, I think that should be:

def doCallback(self):
if time.time() - self.last_call >= MIN_TIME:
self.data_callback()
self.last_call = time.time()