Turning a callback function into a generator

[Kirk McDonald]

Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

For the sake of argument, assume the iteration is something more
interesting than this which relies on the callback mechanism. The
function is an existing interface, and I cannot change it.

I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?

-Kirk McDonald

 

[Peter Otten]

Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

For the sake of argument, assume the iteration is something more
interesting than this which relies on the callback mechanism. The
function is an existing interface, and I cannot change it.

I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?

I don't think there is a robust solution, see

http://groups.google.com/group/comp...read/0ce55373f128aa4e/1d27a78ca6408134?&hl=en
aka http://mail.python.org/pipermail/python-list/2003-December/197726.html

http://groups.google.com/group/comp...read/5793298a1ce93292/9230ddcc55d84694?&hl=en
aka http://mail.python.org/pipermail/python-list/2004-September/239025.html

Peter

 

[cmdrrickhunter]

Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

Which object is immutable? the callback or the function? If its the
callback then

def func(callback):
for i in numbers:
yield callback(i)

If the function is immutable, then its a bit harder. The callback has
to be able to do the processing. You can't use an iterator here
because call stack gets in the way. You could store the information
being passed to the callback in a list, then iterate over the list
afterwards. Or you could have the callback be able to handle all of
the work at once.

What do you intend to use this for? Python has a lot of options and
you may not be using the best one for the problem

 

[Lawrence D'Oliveiro]

I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?

Run it in a separate thread/process?

 

[Kirk McDonald]

Kirk McDonald wrote:

Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

Which object is immutable? the callback or the function? If its the
callback then

def func(callback):
for i in numbers:
yield callback(i)

If the function is immutable, then its a bit harder. The callback has
to be able to do the processing. You can't use an iterator here
because call stack gets in the way. You could store the information
being passed to the callback in a list, then iterate over the list
afterwards. Or you could have the callback be able to handle all of
the work at once.

What do you intend to use this for? Python has a lot of options and
you may not be using the best one for the problem

It is the function that is immutable.

I am writing a library for the D programming language that is not
totally unlike Boost.Python:

http://dsource.org/projects/pyd/wiki

It is still in the fairly early stages, although the basic function and
class wrapping do work.

This particular functionality is required to wrap D's basic iteration
protocol, the opApply function:

http://www.digitalmars.com/d/statement.html#foreach

opApply works using a callback, as described. Because D does not (yet)
have anything like Python's "yield", wrapping it with Python's iteration
interface is turning out to be dreadfully annoying.

-Kirk McDonald

 

[Alex Martelli]

Run it in a separate thread/process?

Sounds best to me. Specifically, given (e.g.) something like the OP's

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

we might have a wrapper such as:

import thread
import Queue

def wrap_cb_into_gen(func):
q = Queue.Queue(1) # maximum size of 1
def callback(item):
q.put(item)
all_done_sentinel = object()
def thread_skeleton():
func(callback)
q.put(all_done_sentinel)
thread.start_new_thread(thread_skeleton, ())
while True:
item = q.get()
if item is all_done_sentinel: break
yield item


Of course, there are lighter-weight options than a length-1 Queue for
the purpose of synchronizing these two threads, but I always tend to
prefer Queue (for its simplicity, generality, solidity) to other
synchronization structures and architectures, unless there is some very
strong reason to do otherwise in a specific case. Also, I normally use
module threading rather than the lower-level module thread -- here,
clearly, either one will do just fine.


Alex

 

[Kirk McDonald]

Run it in a separate thread/process?

Sounds best to me. Specifically, given (e.g.) something like the OP's

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

we might have a wrapper such as:

import thread
import Queue

def wrap_cb_into_gen(func):
q = Queue.Queue(1) # maximum size of 1
def callback(item):
q.put(item)
all_done_sentinel = object()
def thread_skeleton():
func(callback)
q.put(all_done_sentinel)
thread.start_new_thread(thread_skeleton, ())
while True:
item = q.get()
if item is all_done_sentinel: break
yield item


Of course, there are lighter-weight options than a length-1 Queue for
the purpose of synchronizing these two threads, but I always tend to
prefer Queue (for its simplicity, generality, solidity) to other
synchronization structures and architectures, unless there is some very
strong reason to do otherwise in a specific case. Also, I normally use
module threading rather than the lower-level module thread -- here,
clearly, either one will do just fine.


Alex

Threads are probably a weak point of mine. I have little experience with
them, and so I am inclined to paranoia while using them. What
implications does this have for "func"? In my case, it is a function in
an extension module. Does it have to be thread safe? If it isn't, what
limitations does this impose on the use of this wrapper?

All that aside, I read the previous threads but it took until just now
to understand how this works. I'll probably have a go at
implementing this in D, as that will be more convenient for the library...

-Kirk McDonald

 

[Alex Martelli]

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

...
Threads are probably a weak point of mine. I have little experience with
them, and so I am inclined to paranoia while using them. What

Paranoia is the correct state of mind to start from when threads are
involved.

implications does this have for "func"? In my case, it is a function in
an extension module. Does it have to be thread safe? If it isn't, what
limitations does this impose on the use of this wrapper?

The 'func' you gave as an example IS threadsafe (if callback is): it
does not access any global object that might be used by other threads at
the same time. If the real func *DOES* access (or, even worse, modify)
global objects -- or more generally objects that might be modified by
other threads -- then you must try to make things safe again by adding
locks (with a strong risk of deadlocking, of course).

This problem is basically (to a large extent) tied to your very specs:
you want the whole callstack up to the point where 'func' calls
'callback' to be essentially ``frozen'' while somehow the 'callback'
(the only point in which you allow intervention in the code, since you
forbid alterations of 'func'!) magically hands over the item to a yield
statement which provides the item to calling-code. Now calling code can
do anything it wants (to globals or any other objects that, for all
you've told us, 'func' might be accessing or altering too) -- and this
must not damage func's behavior.

If there are no other threads in your process, beyond the main one (on
which the generator-wrapper is called) and the specialized one which the
generator-wrapper created (to run func in), then the issues are pretty
much the same as if the generator-wrapper was able to avoid using
threads -- it's just slightly less deterministic where control may shift
between the two threads, but I believe (without being able to prove it)
that this is unlikely to matter... the key issue remains ensuring that
the *caller* of the wrapper, and func itself, don't tread on each
other's toes, regarding all that's "suspended" on the stack while the
caller does whatever it wants with the just-yielded item.

If there ARE other threads in your process, and func is not threadsafe
with respect to them, then of course you should not be calling it
(wrapped or not) without precautions in terms of locking &c -- the
wrapper doesn't matter.

All that aside, I read the previous threads but it took until just now
to understand how this works. I'll probably have a go at
implementing this in D, as that will be more convenient for the library...

OK, sounds good. D looks interesting (I found Walter Bright's short
presentation of it at Google quite engaging), and I might like to give
it a try if it was as handy to interface to Python as C++ (or Java or C#
or Objective C) already are.


Alex