Coroutines: unexpected behaviour

[Jérôme Mainka]

Hello,

I try to experiment with coroutines and I don't understand why this
snippet doesn't work as expected... In python 2.5 and python 2.6 I get
the following output:

0
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<generator object at 0x7e43f8> ignored

The TypeError exception comes from the pprint instruction...

If i replace the main part with:

==
p1 = dump()
p2 = sort(p1)
for item in my_list: p2.send(item)
==

it works as expected.

I don't understand what is goind wrong. Has someone an explanation for
this issue?

Thanks,

Jérôme


===
from functools import wraps
from pprint import pprint
import random

def coroutine(f):
@wraps(f)
def start(*args, **kwargs):
res = f(*args, **kwargs)
res.next()
return res
return start

@coroutine
def sort(target):
l = []

try:
while True:
l.append((yield))
except GeneratorExit:
l.sort()
for item in l:
target.send(item)

@coroutine
def dump():
while True:
pprint((yield))

if __name__ == "__main__":
my_list = range(100)
random.shuffle(my_list)

p = sort(dump())

for item in my_list:
p.send(item)

 

[Thomas Jollans]

Hello,

I try to experiment with coroutines and I don't understand why this
snippet doesn't work as expected... In python 2.5 and python 2.6 I get
the following output:

0
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<generator object at 0x7e43f8> ignored

The TypeError exception comes from the pprint instruction...

If i replace the main part with:

==
p1 = dump()
p2 = sort(p1)
for item in my_list: p2.send(item)
==

it works as expected.

What a strange problem. I hope somebody else can shed more light on what
is going on

Anyway, it appears (to me) to be a scoping issue: if you move
"from pprint import pprint" into the dump() function body, it works. I
don't understand why this changes anything: pprint was in the global
namespace all along, and it's present and functioning exactly *once*,
after which it's None. It's still there, otherwise there'd be a
NameError, but it was set to None without any line of code doing this
explicitly as far as I can see.

Another VERY strange thing is, of course, your "fix": renaming "p" to
"p2" helps. "p", "p1", "p3", and "yyy" are just some examples of
variable names that don't work.

As for Python versions: I changed the code to work with Python 3 and get
the same odd behaviour.

I don't understand what is goind wrong. Has someone an explanation for
this issue?

Thanks,

Jérôme


===
from functools import wraps
from pprint import pprint
import random

def coroutine(f):
@wraps(f)
def start(*args, **kwargs):
res = f(*args, **kwargs)
res.next()
return res
return start

@coroutine
def sort(target):
l = []

try:
while True:
l.append((yield))
except GeneratorExit:
l.sort()
for item in l:
target.send(item)

@coroutine
def dump():
while True:
pprint((yield))

if __name__ == "__main__":
my_list = range(100)
random.shuffle(my_list)

p = sort(dump())

for item in my_list:
p.send(item)

 

[Steven D'Aprano]

Hello,

I try to experiment with coroutines and I don't understand why this
snippet doesn't work as expected... In python 2.5 and python 2.6 I get
the following output:

0
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<generator object at 0x7e43f8> ignored

I changed the while loop in dump() to print the value of pprint:

while True:
print type(pprint), pprint
pprint((yield))


and sure enough I get this output:

[steve@sylar ~]$ python2.6 test_coroutine.py
<type 'function'> <function pprint at 0xb7edc534>
0
<type 'NoneType'> None
Exception TypeError: "'NoneType' object is not callable" in <generator
object sort at 0xb7ee5c84> ignored


BUT then I wrapped the __main__ code in a function:


def f():
my_list = range(100)
random.shuffle(my_list)
p = sort(dump())
for item in my_list:
p.send(item)

if __name__ == "__main__":
f()


and the problem went away:

[steve@sylar ~]$ python2.6 test_coroutine.py
<type 'function'> <function pprint at 0xb7eb44fc>
0
<type 'function'> <function pprint at 0xb7eb44fc>
1
<type 'function'> <function pprint at 0xb7eb44fc>
2
[...]
<type 'function'> <function pprint at 0xb7eb44fc>
99
<type 'function'> <function pprint at 0xb7eb44fc>


How bizarre is that?


I have to say that your code is horribly opaque and unclear to me. Maybe
that's just because I'm not familiar with coroutines, but trying to
follow the program flow is giving me a headache. It's like being back in
1977 trying to follow GOTOs around the code, only without line numbers.
But if I have understood it correctly, I think the generator function
coroutine(sort) is being resumed by both the send() method and the next()
method. I draw your attention to this comment from the PEP that
introduced coroutines:

Because yield will often be returning None, you should always check for
this case. Don't just use its value in expressions unless you're sure
that the send() method will be the only method used resume your generator
function.
[end quote]

http://docs.python.org/release/2.5/whatsnew/pep-342.html


So I wonder if that has something to do with it?

 

[Jérôme Mainka]

How bizarre is that? Sure...


I have to say that your code is horribly opaque and unclear to me.

Welcome to the coroutines world

This is mainly a pipeline where each function suspends execution
waiting for data (yield), and feeding other functions (the sort
function's target argument) with data (.send())

-> sort() -> dump()

Because yield will often be returning None, you should always check for
this case. Don't just use its value in expressions unless you're sure
that the send() method will be the only method used resume your generator
function.

This remark deals with constructions like:

value = (yield i)

For an excellent introduction to coroutines in python, I recommend:

http://www.dabeaz.com/coroutines/


Thanks,

Jérôme

 

[Thomas Jollans]

Heureka! I think I've found the answer.

Have a look at this:

###
from itertools import izip

def gen():
print globals()
try:
while True:
yield 'things'
except GeneratorExit:
print globals()

if __name__ == "__main__":
g = gen()
g.next()
###
% python genexit.py
{'izip': <type 'itertools.izip'>, 'g': <generator object at
0x7f76c7340878>, '__builtins__': <module '__builtin__' (built-in)>,
'__file__': 'genexit.py', 'gen': <function gen at 0x7f76c7327848>,
'__name__': '__main__', '__doc__': None}
{'izip': None, 'g': None, '__builtins__': <module '__builtin__'
(built-in)>, '__file__': 'genexit.py', 'gen': <function gen at
0x7f76c7327848>, '__name__': '__main__', '__doc__': None}
###

Note that izip (the thing I imported, could just as well have been
pprint) is None the second time I print globals(). why?

This second print happens when GeneratorExit is raised. Just like all
occurences of dump.send(..) in the OP's code are underneath
"except GeneratorExit:"
When does GeneratorExit happen? Simple: When the generator is destroyed.
And when is it destroyed? *looks at the end of the file*
Since it's a global variable, it's destroyed when the module is
destroyed. Python sees the end of the script, and del's the module,
which means all the items in the module's __dict__ are del'd one-by-one
and, apparently, to avoide causing havoc and mayhem, replaced by None.
Fair enough.

So why did the "solutions" we found work?

(1) changing the variable name:
I expect Python del's the globals in some particular order, and this
order is probably influenced by the key in the globals() dict. One key
happens to be deleted before pprint (then it works, pprint is still
there), another one happens to come after pprint. For all I know, it
might not even be deterministic.

(2) moving pprint into the dump() function
As long as there's a reference to the dump generator around, there will
be a reference to pprint around. sort() has a local reference to
target == dump(), so no problem there.

(3) wrapping the code in a function
As the generator is now a local object, it gets destroyed with the
locals, when the function exits. The module gets destroyed afterwards.


What should you do to fix it then?

If you really want to keep the except GeneratorExit: approach, make sure
you exit it manually. Though really, you should do something like
p.send(None) at the end, and check for that in the generator: recieving
None would mean "we're done here, do post processing!"

-- Thomas

 

[Carl Banks]

Hello,

I try to experiment with coroutines and I don't understand why this
snippet doesn't work as expected... In python 2.5 and python 2.6 I get
the following output:

0
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<generator object at 0x7e43f8> ignored

The TypeError exception comes from the pprint instruction...

If i replace the main part with:

==
p1 = dump()
p2 = sort(p1)
for item in my_list: p2.send(item)
==

it works as expected.

I don't understand what is goind wrong. Has someone an explanation for
this issue?

Thanks,

Jérôme

===
from functools import wraps
from pprint import pprint
import random

def coroutine(f):
    @wraps(f)
    def start(*args, **kwargs):
        res = f(*args, **kwargs)
        res.next()
        return res
    return start

@coroutine
def sort(target):
    l = []

    try:
        while True:
            l.append((yield))
    except GeneratorExit:
        l.sort()
        for item in l:
            target.send(item)

@coroutine
def dump():
    while True:
        pprint((yield))

if __name__ == "__main__":
    my_list = range(100)
    random.shuffle(my_list)

    p = sort(dump())

    for item in my_list:
        p.send(item)

Tricky, but pretty simple once you know what happens.

What's happening here is that the GeneratorExit exception isn't raised
until the generator is destroyed (because how does the generator know
there are no more sends coming?). That only happens when the __main__
module is being destroyed.

Well, when destroying __main__, Python overwrites all its attributes
with None, one-by-one, in arbitrary order. In the first scenario,
"pprint" was being set to None before "p" was, thus by the time the
generator got about to running, the pprint was None. In the second
scenario, "p2" was being set to None before "pprint", so that pprint
still pointed at the appropriate function when the generator began
executing.

The solution is to explicity close the generator after the loop, this
signaling it to run before __main__ is destroyed.

p.close()


I suggest, if you intend to use this kind of thing in real code (and I
would not recommend that) that you get in a habit of explicitly
closing the generator after the last send(), even when you don't think
you have to.

IMHO, coroutines are the one time during the PEP-era that Python can
be accused of feature creep. All other changes seemed driven by
thoughtful analysis, this one seemed like it was, "OMG that would be
totally cool". PEP 342 doesn't give any compelling use cases (it only
gives examples of "cool things you can do with coroutines"), no
discussion on how in improves the language. Suffice to say that,
thanks to this example, I'm more averse to using them than I was
before.


Carl Banks

 

[Jérôme Mainka]

I suggest, if you intend to use this kind of thing in real code (and I
would not recommend that) that you get in a habit of explicitly
closing the generator after the last send(), even when you don't think
you have to.

Very clear explanation. Thanks for the investigation...

IMHO, coroutines are the one time during the PEP-era that Python can
be accused of feature creep.  All other changes seemed driven by
thoughtful analysis, this one seemed like it was, "OMG that would be
totally cool".  PEP 342 doesn't give any compelling use cases (it only
gives examples of "cool things you can do with coroutines"), no
discussion on how in improves the language.  Suffice to say that,
thanks to this example, I'm more averse to using them than I was
before.

The wonderful David Beazley's course [1] piqued my curiosity. But,
indeed, this is a hard piece difficult to master.


Thanks again,

Jérôme

[1] http://www.dabeaz.com/coroutines/