Odd closure issue for generators

[Brian Quinlan]

This is from Python built from the py3k branch:.... print(q())
....
11
12
13
14
15.... print(q())
....
15
15
15
15
15
Looking at the implementation, I see why this happens:.... print(id(q.__closure__[0]))
....
3847792
3847792
3847792
3847792
3847792
But it seems very odd to me and it can lead to some problems that are a
real pain in the ass to debug.

Cheers,
Brian

 

[Carl Banks]

This is from Python built from the py3k branch:
 >>> c = (lambda : i for i in range(11, 16))
 >>> for q in c:
...     print(q())
...
11
12
13
14
15
 >>> # This is expected
 >>> c = (lambda : i for i in range(11, 16))
 >>> d = list(c)
 >>> for q in d:
...     print(q())
...
15
15
15
15
15
 >>> # I was very surprised

Looking at the implementation, I see why this happens:
 >>> c = (lambda : i for i in range(11, 16))
 >>> for q in c:
...     print(id(q.__closure__[0]))
...
3847792
3847792
3847792
3847792
3847792
 >>> # The same closure is used by every lambda

But it seems very odd to me and it can lead to some problems that are a
real pain in the ass to debug.

It's really the only sane way to handle it, odd though it may seem in
this narrow case. In Python nested functions have to be able to
reference the current value of a variable because of use cases like
this:

def func():
def printx():
print x
x = 1
printx()
x = 2
printx()

Referencing a nonlocal variable always uses the current (or last)
value of that variable, not the value it had when the nested function
was defined.


The way to handle the issue you are seeing is to create a new scope
with a variable the remains at the value you want to close upon:

create_const_function(value):
def func():
return value
c = (create_const_function(i) for i in range(11, 16))

Or you can do it the slacker way and use a default argument:

c = (lambda i=i: i for i in range(11, 16))


Carl Banks

 

[Michele Simionato]

It's really the only sane way to handle it, odd though it may seem in
this narrow case.  In Python nested functions have to be able to
reference the current value of a variable because of use cases like
this:

def func():
    def printx():
        print x
    x = 1
    printx()
    x = 2
    printx()

Referencing a nonlocal variable always uses the current (or last)
value of that variable, not the value it had when the nested function
was defined.

This is not a good argument. Counter example: Scheme works
exactly like Python

(define (func)
(define x 1)
(define (printx)
(display x) (newline))
(printx)
(set! x 2)
(printx)) ;; prints 1 and 2

but it is still possible to have a list comprehension working
as the OP wants:

(list-of (lambda () i) (i in (range 11 16)))

Actually, in Scheme one would have to fight to define
a list comprehension (more in general loops) working as
in Python: the natural definition works as the OP wants. See
http://www.artima.com/weblogs/viewpost.jsp?thread=251156 and the
comments below for the details.

 

[Brian Quinlan]

Scott David Daniels wrote:
[snipped]

When you evaluate a lambda expression, the default args are evaluated,
but the expression inside the lambda body is not. When you apply that
evaluated lambda expression, the expression inside the lambda body is
is evaluated and returned.

But that's not really the issue. I knew that the lambda was not
evaluated but thought each generator expression got its own context
rather than sharing one.

Taken in isolation, having one context per expression has more
compelling semantics but it is inconsistent with the obvious
transliteration of the generator into a loop.

Cheers,
Brian

 

[Dave Angel]

<snip>

The way to handle the issue you are seeing is to create a new scope
with a variable the remains at the value you want to close upon:

create_const_function(value):
def func():
return value
c =create_const_function(i) for i in range(11, 16))

Or you can do it the slacker way and use a default argument:

c =lambda i=i: i for i in range(11, 16))


Carl Banks

I agree with most of what you say, but I think you missed the last line
of the function:

create_const_function(value):
def func():
return value

return func

 

[Aahz]

Actually, in Scheme one would have to fight to define
a list comprehension (more in general loops) working as
in Python: the natural definition works as the OP wants. See
http://www.artima.com/weblogs/viewpost.jsp?thread=3D251156 and the
comments below for the details.

This URL isn't working for me, gives 500.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Given that C++ has pointers and typecasts, it's really hard to have a
serious conversation about type safety with a C++ programmer and keep a
straight face. It's kind of like having a guy who juggles chainsaws
wearing body armor arguing with a guy who juggles rubber chickens wearing
a T-shirt about who's in more danger." --Roy Smith, c.l.py, 2004.05.23

 

[Ned Deily]

Scott David Daniels wrote:
[snipped]

When you evaluate a lambda expression, the default args are evaluated,
but the expression inside the lambda body is not. When you apply that
evaluated lambda expression, the expression inside the lambda body is
is evaluated and returned.

But that's not really the issue. I knew that the lambda was not
evaluated but thought each generator expression got its own context
rather than sharing one.

Each? Maybe that's a source of confusion. There is only one generator
expression in your example.
[<function <lambda> at 0x119348>, <function <lambda> at 0x119390>,
<function <lambda> at 0x1193d8>, <function <lambda> at 0x119420>,
<function <lambda> at 0x119468>]

 

[Lawrence D'Oliveiro]

... print(q())
...
15
15
15
15
15
Try

... print q()
...
11
12
13
14
15

 

[Michele Simionato]

This URL isn't working for me, gives 500.

This happens sometimes with Artima. Usually it is just a matter of
waiting a few minutes.

 

[Michele Simionato]

This URL isn't working for me, gives 500.

Anyway, the point is that to explain Python behavior with closures in
list/generator comprehension it is not enough to invoke
late bindings (Scheme has late bindings too but list comprehension
works differently). The crux is in the behavior of the for loop:
in Python there is a single scope and the loop variable is
*mutated* at each iteration, whereas in Scheme (or Haskell or any
other functional language) a new scope is generated at each
iteration and there is actually a new loop variable at each iteration:
no mutation is involved. Common Lisp works like Python. It is a design
decision which at the end comes down to personal preference and
different
languages make different choices with no clear cut winner (I
personally
prefer the more functional way).

 

[Lawrence D'Oliveiro]

In message

The crux is in the behavior of the for loop:
in Python there is a single scope and the loop variable is
*mutated* at each iteration, whereas in Scheme (or Haskell or any
other functional language) a new scope is generated at each
iteration and there is actually a new loop variable at each iteration:
no mutation is involved.

I think it's a bad design decision to have the loop index be a variable that
can be assigned to in the loop.

 

[Gabriel Genellina]

This URL isn't working for me, gives 500.

Mmm, the URL ends with: thread, an equals sign, and the number 251156
If you see =3D -- that's the "=" encoded as quoted-printable...

 

[Michele Simionato]

Mmm, the URL ends with: thread, an equals sign, and the number 251156
If you see =3D -- that's the "=" encoded as quoted-printable...

Actually this is the right URL:

http://www.artima.com/weblogs/viewpost.jsp?thread=251156

 

[Aahz]

In message


I think it's a bad design decision to have the loop index be a variable
that can be assigned to in the loop.

Why?
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Given that C++ has pointers and typecasts, it's really hard to have a
serious conversation about type safety with a C++ programmer and keep a
straight face. It's kind of like having a guy who juggles chainsaws
wearing body armor arguing with a guy who juggles rubber chickens wearing
a T-shirt about who's in more danger." --Roy Smith, c.l.py, 2004.05.23

 

[Brian Quinlan]

Scott David Daniels wrote:
[snipped]

When you evaluate a lambda expression, the default args are evaluated,
but the expression inside the lambda body is not. When you apply that
evaluated lambda expression, the expression inside the lambda body is
is evaluated and returned.

But that's not really the issue. I knew that the lambda was not
evaluated but thought each generator expression got its own context
rather than sharing one.

Each? Maybe that's a source of confusion. There is only one generator
expression in your example.
[<function <lambda> at 0x119348>, <function <lambda> at 0x119390>,
<function <lambda> at 0x1193d8>, <function <lambda> at 0x119420>,
<function <lambda> at 0x119468>]

Sorry, I wasn't as precise as I should have been.

If you consider this example:
(<expr> for x in y)

I thought that every time that <expr> was evaluated, it would be done in
a new closure with x bound to the value of x at the time that the
closure was created.

Instead, a new closure is created for the entire generator expression
and x is updated inside that closure.

Cheers,
Brian

 

[Terry Reedy]

Sorry, I wasn't as precise as I should have been.

If you consider this example:
(<expr> for x in y)

I thought that every time that <expr> was evaluated, it would be done in
a new closure with x bound to the value of x at the time that the
closure was created.

Instead, a new closure is created for the entire generator expression
and x is updated inside that closure.

Thanks you for explaining your confusion. Knowing what sort of
other-language-baggage people are being mislead by can only help in
explaining Python. But here is my question. In Python,

g = (<expr> for x in iterable)

is essentially an abbreviation for, and means the same as

def _(it):
for x in it:
yield <expr>
g = _(iterable)
del _

Are there language in which a similar construct has an essentially
different meaning?

Terry Jan Reedy

 

[Michele Simionato]

Thanks you for explaining your confusion.  Knowing what sort of
other-language-baggage people are being mislead by can only help in
explaining Python.  But here is my question.  In Python,

g = (<expr> for x in iterable)

is essentially an abbreviation for, and means the same as

def _(it):
   for x in it:
     yield <expr>
g = _(iterable)
del _

Are there language in which a similar construct has an essentially
different meaning?

Terry Jan Reedy

Yes, most functional languages have the concept of streams.
You can even define a stream-comprehension that looks like
Python generator comprehension but it is an essentially different
thing. See for instance

http://www.artima.com/weblogs/viewpost.jsp?thread=251159

 

[Terry Reedy]

Yes, most functional languages have the concept of streams.
You can even define a stream-comprehension that looks like
Python generator comprehension but it is an essentially different
thing. See for instance

http://www.artima.com/weblogs/viewpost.jsp?thread=251159

I read that. It seems that streams are virtual or lazy linked-lists(1).
I think, though, that comparing them to iterators is misleading. They
are iterables, but with a different iteration protocol. Python
iterables are generally reiterable just like streams.

chars = 'abc'
for c in chars: print(c,end=' ')
for c in chars: print(c,end=' ')

produces repeatable output just like your stream example. Python
*could* have given iterables .first and .rest methods instead of
..__iter__, but that works best for linked lists and awfully for arrays.

Anyway, I realize now that having generator comprehensions produce an
*iterator* rathar than an *iterable* or *generator function* is
something of an anomaly Set, dict, and list comprehensions in Python
produce iterables, of course, as does a stream comprehension in Scheme
and, I presume, comprehensions in other languages.

A generator expression could have been defined in Python to just produce
a generator function, without calling it, but the intent of the
abbreviation was for one-use situations. Multi-use gfs should be
defined with a def statement.

Terry Jan Reedy

(1) Calling the first and rest methods 'car' and 'cdr' convinces me that
schemers really do not want scheme to be popular, but prefer it to
remain a small cult language.

 

[Aahz]

(1) Calling the first and rest methods 'car' and 'cdr' convinces me that
schemers really do not want scheme to be popular, but prefer it to
remain a small cult language.

What, you don't get a warm, fuzzy feeling from saying, "Today is the car
of the cdr of your life"?

 

[David Stanek]

This is from Python built from the py3k branch:
 >>> c = (lambda : i for i in range(11, 16))
 >>> for q in c:
...     print(q())
...
11
12
13
14
15
 >>> # This is expected
 >>> c = (lambda : i for i in range(11, 16))
 >>> d = list(c)
 >>> for q in d:
...     print(q())
...
15
15
15
15
15
 >>> # I was very surprised

You are entitled to be surprised.  Then figure out what is going on.
Hint: it is the moral equivalent of what is happening here:

   >>> c = []
   >>> for i in range(11, 16):
           c.append(lambda: i)

   >>> i = 'Surprise!'
   >>> print([f() for f in c])
   ['Surprise!', 'Surprise!', 'Surprise!', 'Surprise!', 'Surprise!']

   >>> i = 0
   >>> print([f() for f in c])
   [0, 0, 0, 0, 0]

The body of your lambda is an un-evaluated expression with a reference,
not an expression evaluated at the time of loading c.  TO get what you
expected, try this:

   >>> c = []
   >>> for i in range(11, 16):
           c.append(lambda i=i: i)

   >>> i = 'Surprise!'
   >>> print([f() for f in c])
   [11, 12, 13, 14, 15]

When you evaluate a lambda expression, the default args are evaluated,
but the expression inside the lambda body is not.  When you apply that
evaluated lambda expression, the expression inside the lambda body is
is evaluated and returned.

Getting around this can be pretty easy:

c = (lambda i=i: i for i in range(11, 16))
for q in (list(c)):
print(q())