Callbacks to generators

[Dave Benjamin]

Is there a straightforward way to create a generator from a function that
takes a callback? For instance, I have a function called "process":

def process(text, on_token):
...

For each token that "process" finds in "text", it creates a token object,
"token", and calls "on_token(token)".

Now, suppose I wanted to create a generator based on this function. I tried
to do the following:

def process_iter(text):
def on_token(token):
yield token
process(text, on_token)

However, rather than create a generator as I had hoped, this function
doesn't return anything. I guess it creates a bunch of singleton generators,
one per token, and throws them away. In any case, is there a way to do what
I am trying to do here without resorting to threads?

The reason I am trying to do this is that I'd still like to be able to use
"process" in code for Python 2.1, which does not support generators. I'd
like to avoid implementing the entire thing twice or basing the callback
version on the generator version.

Thanks,
Dave

 

[Peter Otten]

Is there a straightforward way to create a generator from a function that
takes a callback? For instance, I have a function called "process":

Last time I asked there wasn't.

The reason I am trying to do this is that I'd still like to be able to use
"process" in code for Python 2.1, which does not support generators. I'd
like to avoid implementing the entire thing twice or basing the callback
version on the generator version.

When you know beforehand that your code should play nice with both the
visitor and the iterator pattern, you can use an iterator class instead of
a generator, e. g:

try:
iter
except NameError:
# prepare for the poor man's generator...
class StopIteration(Exception):
pass

def iter(seq):
try:
seq.__iter__
except AttributeError:
pass
else:
return seq.__iter__()
# test for as many special cases as you like, e. g.
if isinstance(seq, type([])):
return ListIterator(seq)
raise TypeError

hasGenerators = 0
else:
hasGenerators = 1

# this is clumsy...
class ListIterator:
""" Example of an iterator class """
def __init__(self, seq):
self.seq = seq
self.index = 0

def __iter__(self):
return self

def next(self):
index = self.index
self.index += 1
try:
return self.seq[index]
except IndexError:
raise StopIteration

def visit(seq, visitor):
it = iter(seq)
try:
while 1:
visitor(it.next())
except StopIteration:
pass

# ... but leaves client code unaffected:
if hasGenerators:
print "using generator"
for item in range(3):
print 2 * item,
print

def printItem(item):
print 2 * item,

print "using visitor"
visit(range(3), printItem)
print

Peter

 

[Terry Reedy]

Is there a straightforward way to create a generator from a function that
takes a callback? For instance, I have a function called "process":

This sentence is not clear *to me*, as written, so I respond to how I
interpret it, in light of process_iter below.

def process(text, on_token):
...
For each token that "process" finds in "text", it creates a token object,
"token", and calls "on_token(token)".

I think your fundamental problem is that callbacks usually have an
item-to-process parameter, as with on_token above, whereas iterator.next
methods do not. So you would have to use a token communicating method that
both types of responders can use.

Now, suppose I wanted to create a generator based on this function. I tried
to do the following:

def process_iter(text):
def on_token(token):
yield token
process(text, on_token)

This on_token is a generator function. When called, it returns a generator
whose next method would yield token if it were to be called, but which
never is called.

However, rather than create a generator as I had hoped, this function

Not sure whether 'this function' means process_iter, which creates a
generator function on_token and passes it to process(), or on_token itself,
which does create a generator each time it is called within process.

doesn't return anything. I guess it creates a bunch of singleton generators,
one per token, and throws them away.

Correct. I *think* what you want is something more like

#select one on following depending on Python version

def on_token(): # modified traditional call_back
token = <get token from token_stash>
return <some func of token>

def on_token_gen(): # generator version for 2.2+
while 1:
token = <get token from token_stash>
yield <some func of token>
on_token = on_token_gen().next

process(text, on_token)

where process puts token in token_stash before calling on_token()

In any case, is there a way to do what
I am trying to do here without resorting to threads?

Does above help?

Terry J. Reedy

 

[Humpty Dumpty]

def process(text, on_token):

...

For each token that "process" finds in "text", it creates a token object,
"token", and calls "on_token(token)".

Now, suppose I wanted to create a generator based on this function. I tried
to do the following:

def process_iter(text):
def on_token(token):
yield token
process(text, on_token)

However, rather than create a generator as I had hoped, this function
doesn't return anything. I guess it creates a bunch of singleton generators,
one per token, and throws them away. In any case, is there a way to do what
I am trying to do here without resorting to threads?

Something weird here. I hven't used generators much, but seems to me that:

1) you maybe don't need them:

def process_iter(text):
def on_token(token):
return token
process(text, on_token)

2) you need some sort of while loop so the fnction doesn't return after the
first yield:

def process_iter(text):
def on_token(token):
while 1:
yield token
process(text, on_token)

I'm curious to understand if neither of those are true, why...

Oliver

 

[Terry Reedy]

Something weird here. I hven't used generators much, but seems to me that:

1) you maybe don't need them:

def process_iter(text):
def on_token(token):
return token
process(text, on_token)

True. And one cannot use them for 2.1. I expect OP wants to use iterators
in 2.2+ to avoid repeated function call overhead. The main problem is
writing code that works both ways.

2) you need some sort of while loop so the fnction doesn't return after the
first yield:

True. however...

def process_iter(text):
def on_token(token):
while 1:
yield token

This is garbled and repeats OP's mistake of making token a param of the
generator function. A generator function is called once and its parameter
is used to set up the generator whose next method is what gets called
repeatedly. See my previous post for a possibly workable example.

Terry J. Reedy

 

[Terry Reedy]

#select one on following depending on Python version

def on_token(): # modified traditional call_back
token = <get token from token_stash>
return <some func of token>

def on_token_gen(): # generator version for 2.2+
while 1:
token = <get token from token_stash>
yield <some func of token>
on_token = on_token_gen().next

process(text, on_token)

where process puts token in token_stash before calling on_token()

If doing lots of callbacks, repeating anything like the above for each
would get pretty tedious. So after getting the pattern correct, if the
above does indeed work, I might try to factor out the common portion and
write a callback factory function that inserts the variable code into the
version-appropriate template, compiles it, and returns the
version-apprieate callback.

Terry J. Reedy