Reasoning behind nested scope

[Andy Baker]

Hi there,

I'm learning Python at the moment and trying to grok the thinking behind
it's scoping and nesting rules.

I was googling for nested functions and found this Guido quote:
(http://www.python.org/search/hypermail/python-1993/0343.html)

"This is because nested function definitions don't have access to the
local variables of the surrounding block -- only to the globals of the
containing module. This is done so that lookup of globals doesn't
have to walk a chain of dictionaries -- as in C, there are just two
nested scopes: locals and globals (and beyond this, built-ins).
Therefore, nested functions have only a limited use. This was a
deliberate decision, based upon experience with languages allowing
arbitraries nesting such as Pascal and both Algols -- code with too
many nested scopes is about as readable as code with too many GOTOs.
And, of course, in Python, the "proper" way is to use an
object-oriented programming style"

This sounds reasonable to me although nested scope always struck me as more
natural and intuitive ('Don't be surprising')

I was wondering how the balance changed in favour of nested scope? What
changed people's minds about 'readability' vs other factors? Are nested
scopes still regarded as leading to spagetti code etc?

(On a side note is there any way to call a nested function from outside the
parent? I was kind of expecting nested functions to be addressable through
dot notation like methods are but I can see why that wouldn't be quite
right. This might be a better question for the tutor list...)

Andy Baker

 

[Mel Wilson]

"This is because nested function definitions don't have access to the
local variables of the surrounding block -- only to the globals of the
containing module. This is done so that lookup of globals doesn't
have to walk a chain of dictionaries -- as in C, there are just two
nested scopes: locals and globals (and beyond this, built-ins).
Therefore, nested functions have only a limited use. This was a
deliberate decision, based upon experience with languages allowing
arbitraries nesting such as Pascal and both Algols -- code with too
many nested scopes is about as readable as code with too many GOTOs.
And, of course, in Python, the "proper" way is to use an
object-oriented programming style"

This sounds reasonable to me although nested scope always struck me as more
natural and intuitive ('Don't be surprising')

I was wondering how the balance changed in favour of nested scope? What
changed people's minds about 'readability' vs other factors? Are nested
scopes still regarded as leading to spagetti code etc?

I guess a consensus built up that the power/safety
tradeoff had given too much emphasis to safety.

My own poster child for nested scopes is in some old post
on c.l.p . I wanted to build a lot of similar labelled text
controls in wxPython, following a naming convention. I
could have written a function, but pre-nested-scope, any
change I made to the information used in building the
controls would have meant changing the function parameter
list, and all the function calls. I came up with a template
string which got parameter substituted via the '%' operator,
and then `exec`ed so as code it had full access to all the
info that was in the contemporary scope.

Post nested scope, any info the function needed from the
surrounding scope could simply be read from there. The code
was much less unusual.

I believe (without proof) that any language that won't
let me shoot myself in the foot will also prevent me doing
anything useful. Debate is heating up now about write
access to enclosing scopes. We'll see how strong my faith
really is. People will be able to write function nests
instead of classes.

(On a side note is there any way to call a nested function from outside the
parent? I was kind of expecting nested functions to be addressable through
dot notation like methods are but I can see why that wouldn't be quite
right. This might be a better question for the tutor list...)

Yup.

def f1 (a):
if a:
x = 2
else:
x = 3

def f2 (y):
return y % x

return f2

import os
b = f1 (os.name.startswith ('p'))
for c in range (10):
print b (c)



Regards. Mel.

 

[Larry Bates]

Ever since my first Fortran class some 30+ years ago
I've always liked nested scopes. If you want something
to be global, say so in your code. In Fortran we used
common blocks, in Python you use a global directive.
Making everything global can lead to many name clashes
or you must have lots of unique variable names which
lengthens code and makes reuse more difficult.

Judicious use of OOP and globals seems to be the best
approach. The more I learn/use OOP the less I seem
to require globals, but there are some times they
seem a preferable solution.

I don't believe there is any way to call nested
functions from outside their parent. If you need
something like that, make the function a class and
make the nested function a method of the class.

HTH,
Larry Bates
Syscon, Inc.

 

[Peter Otten]

(On a side note is there any way to call a nested function from outside
the parent? I was kind of expecting nested functions to be addressable
through dot notation like methods are but I can see why that wouldn't be
quite right. This might be a better question for the tutor list...)

When you are nesting functions, you don't get one function sitting inside
another function. Instead, the function creation code of the "inner"
function is executed every time to the effect that you get a new inner
function every time you call the outer one:
.... def f(): return "shoobidoo"
.... return f
....False

You wouldn't do that in cases like the above, when you get nothing in return
for the extra overhead, but somtimes nesting is useful - because of the
change in the scoping rules you now get readonly-closures:
.... def f(): return s
.... return f
....('now', 'what')

Peter

 

[Nigel Rowe]

When you are nesting functions, you don't get one function sitting inside
another function. Instead, the function creation code of the "inner"
function is executed every time to the effect that you get a new inner
function every time you call the outer one:

... def f(): return "shoobidoo"
... return f
...
False

You wouldn't do that in cases like the above, when you get nothing in
return for the extra overhead, but somtimes nesting is useful - because of
the change in the scoping rules you now get readonly-closures:

... def f(): return s
... return f
...
('now', 'what')

Peter

What work is actually done when the
'nested function creation code of the "inner" function'
is executed?

Given a quick test:-
<code>
def outer():
def inner():
pass
return inner

b1 = outer()
b2 = outer()

attrs=[a for a in dir(b1) if not a.startswith('_')]
for a, a1, a2 in zip(attrs,
[getattr(b1,a) for a in attrs],
[getattr(b2,a) for a in attrs]):
print a, a1 is a2

</code>
<result>
func_closure True
func_code True
func_defaults True
func_dict True
func_doc True
func_globals True
func_name True
</result>

it appears that all the components of the inner function are the same, which
just leaves the binding of the code object to 'inner'.

Am I missing something, or is the overhead no worse than, say, foo=self.foo,
where self.foo is a method?

 

[Peter Otten]

When you are nesting functions, you don't get one function sitting inside
another function. Instead, the function creation code of the "inner"
function is executed every time to the effect that you get a new inner
function every time you call the outer one:

... def f(): return "shoobidoo"
... return f
...
False

You wouldn't do that in cases like the above, when you get nothing in
return for the extra overhead, but somtimes nesting is useful - because
of the change in the scoping rules you now get readonly-closures:

... def f(): return s
... return f
...
('now', 'what')

Peter

What work is actually done when the
'nested function creation code of the "inner" function'
is executed?

Given a quick test:-
<code>
def outer():
def inner():
pass
return inner

b1 = outer()
b2 = outer()

attrs=[a for a in dir(b1) if not a.startswith('_')]
for a, a1, a2 in zip(attrs,
[getattr(b1,a) for a in attrs],
[getattr(b2,a) for a in attrs]):
print a, a1 is a2

</code>
<result>
func_closure True
func_code True
func_defaults True
func_dict True
func_doc True
func_globals True
func_name True
</result>

it appears that all the components of the inner function are the same,
which just leaves the binding of the code object to 'inner'.

Not the binding, a new function object is created every time outer is run,
i. e. (faked, but I did it for real above with f1 and f2)

False

Am I missing something, or is the overhead no worse than, say,
foo=self.foo, where self.foo is a method?

If it were pure Python it would rather be something like

foo = Function(func_closure=..., func_code=..., ...)

Another perspective is to look at the byte code:
.... def inner(): pass
.... return inner
.... 2 0 LOAD_CONST 1 (<code object inner at
0x4028eee0, file "<stdin>", line 2>)
3 MAKE_FUNCTION 0
6 STORE_FAST 0 (inner)

3 9 LOAD_FAST 0 (inner)
12 RETURN_VALUE
13 LOAD_CONST 0 (None)
16 RETURN_VALUE.... pass
........ return inner
.... 2 0 LOAD_GLOBAL 0 (inner)
3 RETURN_VALUE
4 LOAD_CONST 0 (None)
7 RETURN_VALUE

In the example you get three extra instructions, LOAD_CONST, MAKE_FUNCTION
and STORE_FAST (and earn the slight benefit that inner is now a local
instead of a global). A constant code object is used, meaning compilation
takes place at most once when the module is loaded. There is a dedicated
op-code, so function creation should be faster than "normal" creation of a
class instance.

Now this is all nice and dandy, but how do the two contenders perform?

$ timeit.py -s"def inner(): pass" -s"def outer(): return inner" "outer()"
1000000 loops, best of 3: 0.469 usec per loop
$ timeit.py -s"def outer():" -s" def inner(): pass" -s" return inner"
"outer()"
1000000 loops, best of 3: 1.12 usec per loop

i. e. nesting the two functions roughly doubles execution time.
However, creation of an inner function often will only take a small fraction
of the total time spent in the outer function - in the end it's just a
matter of style.

I use inner functions only when they depend on the local context because I
think it nicely discriminates closures from helpers. I tend to omit
implicit parameters even for helper funtions, therefore nesting them would
gain me nothing.

Peter

 

[Nigel Rowe]

In the example you get three extra instructions, LOAD_CONST, MAKE_FUNCTION
and STORE_FAST (and earn the slight benefit that inner is now a local
instead of a global). A constant code object is used, meaning compilation
takes place at most once when the module is loaded. There is a dedicated
op-code, so function creation should be faster than "normal" creation of a
class instance.

Now this is all nice and dandy, but how do the two contenders perform?

$ timeit.py -s"def inner(): pass" -s"def outer(): return inner" "outer()"
1000000 loops, best of 3: 0.469 usec per loop
$ timeit.py -s"def outer():" -s" def inner(): pass" -s" return inner"
"outer()"
1000000 loops, best of 3: 1.12 usec per loop

i. e. nesting the two functions roughly doubles execution time.
However, creation of an inner function often will only take a small
fraction of the total time spent in the outer function - in the end it's
just a matter of style.

I use inner functions only when they depend on the local context because I
think it nicely discriminates closures from helpers. I tend to omit
implicit parameters even for helper funtions, therefore nesting them would
gain me nothing.

Peter

Thanks Peter, I didn't know about timeit.py (it's now on my $PATH).

I don't think I'm going to worry about the overhead (except in deeply nested
loops). From some quick testing it looks like the overhead is about the
same as 1.5 times
a=b[10:20] # where b="the quick brown jox jumps over the lazy dog"
and less than 1/2 of
a=math.sin(0)

So if the nested code is easier to read, in it goes.