Need help with Python scoping rules

[kj]

I have many years of programming experience, and a few languages,
under my belt, but still Python scoping rules remain mysterious to
me. (In fact, Python's scoping behavior is the main reason I gave
up several earlier attempts to learn Python.)

Here's a toy example illustrating what I mean. It's a simplification
of a real-life coding situation, in which I need to initialize a
"private" class variable by using a recursive helper function.

class Demo(object):
def fact(n):
if n < 2:
return 1
else:
return n * fact(n - 1)

_classvar = fact(5)

This code fails to compile, with the error "global name 'fact' not
defined".

Scanning the Python Language Reference page I see nothing that
would suggest to me a discussion of Python's scoping rules, let
alone anything that would help me solve the specific problem above.
I'm sure it's in there, *somewhere*, but it's anyone's guess where.

Likewise, my book, Beazley's "Python: Essential Reference" is of
no help. I don't doubt that the answer to my question "is in there,
*somewhere*", but finding it (without reading the book sequentially
from page 1) is a tall order.

All that's left is trial-and-error, the worst kind of programming.
And still I can't find the right way to do this... I've tried
every variant of this code that I can imagine, including decorating
fact with @staticmethod or @classmethod, etc., etc. (In the latter
case, I get cryptic errors saying that the staticmethod or classmethod
object is not callable. Or something like that.)

Needless to say, I'm pretty beat by this point. Any help would be
appreciated.

Thanks,

kynn

 

[Diez B. Roggisch]

I have many years of programming experience, and a few languages,
under my belt, but still Python scoping rules remain mysterious to
me. (In fact, Python's scoping behavior is the main reason I gave
up several earlier attempts to learn Python.)

Here's a toy example illustrating what I mean. It's a simplification
of a real-life coding situation, in which I need to initialize a
"private" class variable by using a recursive helper function.

class Demo(object):
def fact(n):
if n < 2:
return 1
else:
return n * fact(n - 1)

_classvar = fact(5)

This code fails to compile, with the error "global name 'fact' not
defined".

Scanning the Python Language Reference page I see nothing that
would suggest to me a discussion of Python's scoping rules, let
alone anything that would help me solve the specific problem above.
I'm sure it's in there, *somewhere*, but it's anyone's guess where.

Likewise, my book, Beazley's "Python: Essential Reference" is of
no help. I don't doubt that the answer to my question "is in there,
*somewhere*", but finding it (without reading the book sequentially
from page 1) is a tall order.

All that's left is trial-and-error, the worst kind of programming.
And still I can't find the right way to do this... I've tried
every variant of this code that I can imagine, including decorating
fact with @staticmethod or @classmethod, etc., etc. (In the latter
case, I get cryptic errors saying that the staticmethod or classmethod
object is not callable. Or something like that.)

Classes are not scopes.

So the above doesn't work because name resolution inside functions/methods
looks for local variables first, then for the *global* scope. There is no
class-scope-lookup.

If you want anything else, you need to qualify it with a full name,
Demo.fact. Which isn't working for your example because "Demo" isn't
available yet.

But if you insist on the above methodology, you can do this:


class Demo(object):

def fact(n):
def inner(n):
if n < 2:
return 1
else:
return n * inner(n - 1)
return inner(n)

_classvar = fact(5)

This makes inner a *local* variable, which is found.

Diez

 

[Jean-Michel Pichavant]

I have many years of programming experience, and a few languages,
under my belt, but still Python scoping rules remain mysterious to
me. (In fact, Python's scoping behavior is the main reason I gave
up several earlier attempts to learn Python.)

Here's a toy example illustrating what I mean. It's a simplification
of a real-life coding situation, in which I need to initialize a
"private" class variable by using a recursive helper function.

class Demo(object):
def fact(n):
if n < 2:
return 1
else:
return n * fact(n - 1)

_classvar = fact(5)

This code fails to compile, with the error "global name 'fact' not
defined".

[snip]

fact is defined within the Demo class, so to access it, you'll need to
prefix it with Demo:

_classvar = Demo.fact(5).

The problem is, Demo will raise a NameError exception.
The solution is pretty simple, cause your fact function seems to be
unrelated to the Demo class :

def _fact(n):
# some code

class Demo(object):
_classvar = _fact(5)


It may happen that you *want* your fact within the Demo, in you example
it's not that obvious, but I think Java would allow such following patern:

class Color:
def __init__(self, r, g,b):
pass
BLACK = Color(0,0,0)

It make sens from a design point of view to put BLACK in the Color
namespace. But I don't think it's possible with python.
You could trick the system using inheritance:

class Chrome:
def __init__(self, r,g,b)
pass
Putting all your code in the Chrome class then:

class Color(Chrome):
BLACK = Chrome(0,0,0)

I'm not sure this is satisfying.

JM

 

[Jean-Michel Pichavant]

Diez B. Roggisch wrote

Classes are not scopes.

Too bad, could have been handy.

JM

 

[Diez B. Roggisch]

Diez B. Roggisch wrote
Too bad, could have been handy.

Nope. Because then a lot of people would write something like this:


class Foo(object):


def bar(self):
bar() # note the missing self.


And this would lead to errors because self was missing from the call
to "bar".

And you'd need a disambiguation for methodname/global-name-clashes. The
global-statement would work, but then code could break when all of a
sudden a subclass defines a method that hitherto was only known as global.
So via subclassing, you introduce *arbitray* and hard to debug errors.

No. I'm certain, not a good idea.

Diez

 

[Dave Angel]

You are trying to run code in a class that does not exist yet.


def Demo():
def fact(n):
if n < 2:
return 1
else:
return n * fact(n - 1)
return type("Demo", (object,), {"fact": staticmethod(fact), "_classvar":
fact(5)})
Demo = Demo()

d = Demo()
print d._classvar # prints 120
print d.fact(7) # prints 5040
print Demo # prints <class '__main__.Demo'>

In all these messages, something I haven't seen pointed out is that
fact() has no self argument. Seems to me that makes it a staticmethod,
so it should be declared that way. But you can't call a static method
from the class scope, since the class hasn't been completed yet. That's
the part I consider a problem, not all the rest. I've seen the same
problem in Forth, where 'smudge' is used to prevent a definition from
accidentally calling itself. But I don't recall whether that was in the
standard, or just vendor's helpful additions.

Anyway, my first choice is to move the static method out of the class.
Second choice workaround follows, moving the initialization of the class
variable to after the class, when it can properly use the class name.:

class Demo(object):
@staticmethod
def fact(n):
if n < 2:
return 1
else:
return n * Demo.fact(n - 1)

Demo._classvar = Demo.fact(5)

print Demo._classvar
xx = Demo()
print xx.fact(6)

DaveA

 

[Ulrich Eckhardt]

class Color:
...

setattrib(Color, "BLACK", Color(0,0,0))

Apart from it being "setattr" and not "setattrib", a simple

Color.BLACK = Color(0,0,0)

should have done the job here. However, what I had in mind was this:

class Color:
_colors = [ ("BLACK", (0,0,0)),
("WHITE", (1,1,1))
]

def __str__(self):
# try to locate a name
for name, rgb in Color._colors:
if self.rgb==rgb:
return name
# no name found, just format as a triplet
return "(%s, %s, %s)" % self.rgb

# add symbolic names
for name, rgb in Color._colors:
setattr(Colors, name, Color(rgb))


....which I got as suggestion on my question how to model C enumeration
lookalikes.

Uli

 

[kj]

Classes are not scopes.

This looks to me like a major wart, on two counts.

First, one of the goals of OO is encapsulation, not only at the
level of instances, but also at the level of classes. Your comment
suggests that Python does not fully support class-level encapsulation.

Second, my example shows that Python puts some peculiar restrictions
on recursion. Recursion! One of the central concepts in the theory
of functions! This is shown most clearly by the following elaboration
of my original example:

class Demo(object):
def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)

def fact_iter(n):
ret = 1
for i in range(1, n + 1):
ret *= i
return ret

classvar1 = fact_iter(5)
classvar2 = fact_rec(5)

This code won't compile as shown, but it does compile if the last
line (the call to the recursive fact_rec) is commented out. There
is no justification for discriminating against recursive functions
in this context. Recursive functions should be OK wherever functions
are OK. I can't think of any other language that allows recursion
but not anywhere.

Is there any good reason (from the point of view of Python's overall
design) for not fixing this?

kynn

 

[Carl Banks]

This looks to me like a major wart, on two counts.

Class statements *are* scopes, they are just not accessible from
scopes nested within them.

First, one of the goals of OO is encapsulation, not only at the
level of instances, but also at the level of classes.  Your comment
suggests that Python does not fully support class-level encapsulation.

I can't answer this, I don't even know what you are talking about.
The OO notion of encapsulation that I'm familar means that an object
can seal off access to private data. This has nothing to do with
class statement scoping.

Second, my example shows that Python puts some peculiar restrictions
on recursion.  Recursion!  One of the central concepts in the theory
of functions!  This is shown most clearly by the following elaboration
of my original example:

class Demo(object):
    def fact_rec(n):
        if n < 2:
            return 1
        else:
            return n * fact_rec(n - 1)

    def fact_iter(n):
        ret = 1
        for i in range(1, n + 1):
            ret *= i
        return ret

    classvar1 = fact_iter(5)
    classvar2 = fact_rec(5)

This code won't compile as shown, but it does compile if the last
line (the call to the recursive fact_rec) is commented out.  There
is no justification for discriminating against recursive functions
in this context.  Recursive functions should be OK wherever functions
are OK.  I can't think of any other language that allows recursion
but not anywhere.

Inside class statements are not the place for this kind of thing.
Define functions like fact_rec outside the class statement.

Is there any good reason (from the point of view of Python's overall
design) for not fixing this?

I suspect that no reason, however profound, would satisfy you.
Therefore I will merely answer your question without fanfare, and you
can take it as you will.

1. One of the key aspects of Python's design is that attributes must
be accessed explicitly with dot notation. Accessing class scopes from
nested functions would (seemingly) allow access to class attributes
without the dotted notation. Therefore it is not allowed.

2. This is considered more important that your ability to define
recursive functions in the class statement.


Carl Banks

 

[Dave Angel]

class Color:
...

setattrib(Color, "BLACK", Color(0,0,0))

Uli

Or instead of setattrib,

Color.BLACK = Color(0,0,0)

DaveA

 

[7stud]

In all these messages, something I haven't seen pointed out is that
fact() has no self argument.  

An "argument" is something that is specified in the the function
call. I assume you are trying to state something like, "fact() is not
defined with a parameter variable named self". However, that has
never been a requirement in python:


class A(object):

def fact(n):
print n

fact("hello")


a = A()
a.fact()

--output:--
hello
<__main__.A object at 0x7faf0>

 

[kj]

kj wrote:

Who says?

Python itself: it already offers a limited form of class encapsulation
(e.g. class variables). It would be nice if it went all the way
and gave classes their own bona fide scope. (But I hasten to add:
I *still* don't understand the Python scope model, and not for lack
of trying. I've only succeeded in finding fragments of this model
explained here and there, like pottery shards: a bit lost in a
tutorial, or some comment in a newsgroup thread, etc. Maybe the
full, authoritative documentation of Python's scope model got lost
somewhere, and will be found by archaeologists in the 25th century...)

Anyway, you could be right (I am not capable to judge it) and Python
should change on this issue but from what I gathered, Pythons OO is
inspired by the following:
- Don't repeat yourself
- Containing code into logical units makes it easier to understand and
maintain.

....except, apparently, when that code is a recursive function, in
which case one's out of luck.

<cut>
It is also one of the best ways to shoot yourself in the foot...

If recursion is so evil, and Python so intent in saving us from
shooting ourselves in the foot, why does it allow recursion at all?

kynn

 

[kj]

I can't answer this, I don't even know what you are talking about.

Yes, you do. As I said in another post, Python offers some degree
of class-level encapsulation (e.g. class variables). But it is
limited by the fact that these class-encapsulated elements can't
always be accessed from within the class itself, which is kind of
silly.

1. One of the key aspects of Python's design is that attributes must
be accessed explicitly with dot notation. Accessing class scopes from
nested functions would (seemingly) allow access to class attributes
without the dotted notation. Therefore it is not allowed.

It would be trivial to define a keyword (e.g. this, or if you
prefer, __this__), valid only within a class statement, and that
the interpreter would recognize as "the current class", even before
this class is full defined.

kynn

 

[Dave Angel]

An "argument" is something that is specified in the the function
call. I assume you are trying to state something like, "fact() is not
defined with a parameter variable named self". However, that has
never been a requirement in python:


class A(object):

def fact(n):
print n

fact("hello")


a =()
a.fact()

--output:--
hello
<__main__.A object at 0x7faf0>

You're good at nitpicking. I concede the distinction between argument
and formal parameter. And self is a convention, not a requirement.
But the fact is that the method as written would never have worked, when
called from outside the class, since you'd have to call it with either
the class name or an instance, and in either case, the method was then
trying to do arithmetic on one of those.

Thanks for diluting my point. The OP is chasing the wrong problem. Who
cares whether a class initializer can call a method, if the method
doesn't meet its original requirements, to be callable outside the class?

And the arguments about how recursion is restricted are ridiculous.
Nothing wrong with a method calling itself, once it's got a proper
signature. You just have to make the call agree with the signature.
The problem is only that the method may not be actually called until the
class definition is complete.

DaveA

 

[Mel]

Is there any good reason (from the point of view of Python's overall
design) for not fixing this?

Python is not a compiled language, in the sense that a compiler can go back
and forth over the program, filling in the details that make the program
runnable. Python is an interpreted language in a sense that makes a `class`
statement an executable statement, with a start time and an end time.
Before the start time, the affected class doesn't exist. After the start
time, the class does exist. In between, while the `class` statement is
executing, it's best to make no promises so as not to constrain present and
future implementations.


Mel.

 

[Carl Banks]

Yes, you do.  As I said in another post, Python offers some degree
of class-level encapsulation (e.g. class variables).  But it is
limited by the fact that these class-encapsulated elements can't
always be accessed from within the class itself, which is kind of
silly.

Nope, you're wrong. Class variables are accessible wherever a class
exists. The apparent silliness of this is because you are confusing
classes with class statements.

You know that the class isn't created until the class statement exits,
don't you?

Yeah, it's a little surprising that you can't access class scope from
a function, but that has nothing to do with encapsulation.

It would be trivial to define a keyword (e.g. this, or if you
prefer, __this__), valid only within a class statement, and that
the interpreter would recognize as "the current class", even before
this class is full defined.

Your solution to this "problem" is to add a keyword to Python.
Laughable. (BTW, it's not trivial.)


Carl Banks

 

[kj]

In all these messages, something I haven't seen pointed out is that
fact() has no self argument. Seems to me that makes it a staticmethod,
so it should be declared that way.

No, the fact() function here represents an internal "helper"
function. It is meant to be called only once to help initialize
a class variable that would be inconvenient to initialize otherwise;
this helper function is not meant to be called from outside the
class statement. Granted, in the example I gave, the "helper"
function (factorial) is a bit silly, but that was just intended as
a simple and familiar example of a recursive function. The actual
function that motivated this post would be considerably more
difficult to explain and would have obscured the point of the post.

kynn

 

[Dave Angel]

This looks to me like a major wart, on two counts.

First, one of the goals of OO is encapsulation, not only at the
level of instances, but also at the level of classes. Your comment
suggests that Python does not fully support class-level encapsulation.

Second, my example shows that Python puts some peculiar restrictions
on recursion. Recursion! One of the central concepts in the theory
of functions! This is shown most clearly by the following elaboration
of my original example:

class Demo(object):
def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)

Why not fix the call? The correct way to call a method is with either
the class name or an instance object, depending on whether it's a
instance method or not. By default, a method is unbound, and its first
argument is by convention called self. If you want to recurse, then add
self in the appropriate places.

As you have it, neither of the methods can be called outside the class:

class Demo(object):
def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)
....

obj = Demo()
print obj.fact_rec(5)

gives error:

Traceback (most recent call last):
File "M:\Programming\Python\sources\dummy\stuff2.py", line 20, in <module>
print obj.fact_rec(5)
TypeError: fact_rec() takes exactly 1 argument (2 given)

To fix it, you need to either change the signature (add in 'self'
argument before the n argument) or do some form of decorator to the
function. If I assume you never intended this method to use 'self'
(ie. it's a static method), then declare it so. And call it accordingly.


class Demo(object):
@staticmethod
def fact(n):
if n < 2:
return 1
else:
return n * Demo.fact(n - 1)


print Demo.fact(6)

On the other hand, if I assume you're just giving a simple example of
what's really intended to be a normal method (called with an instance,
that it uses), then you'd change it this way.

class Demo2(object):
def fact(self, n):
if n<2:
return 1
else:
return n * self.fact(n-1)

obj = Demo2()
print obj.fact(5)

Now the only real restriction, as opposed to all these red-herrings, is
that the class may not be used before it's complete. That only comes to
play when class attributes (non-instance "variables") are defined in
terms of class methods. So if you have such attributes to define, move
them outside of the class, perhaps immediately after it.


class Demo(object):
@staticmethod
def fact(n):
if n < 2:
return 1
else:
return n * Demo.fact(n - 1)
Demo._classvar = Demo.fact(5)

def fact_iter(n):
ret = 1
for i in range(1, n + 1):
ret *= i
return ret

classvar1 = fact_iter(5)
classvar2 = fact_rec(5)

This code won't compile as shown,

Sure it will compile. It just won't run. You get the error after
compiling the function when *calling* it from the classvar* line. But
at that time the class is not defined, and not everything is ready for use.

You can probably work around this by replacing the staticmethod
decorator with an equivalent function call:.

class Demo9(object):
def fact(n):
if n < 2:
return 1
else:
return n * Demo.fact(n - 1)

_classvar = fact(5)
fact = staticmethod(fact)

print Demo9._classvar
xx = Demo9()
print xx.fact(6)
print Demo9.fact(8)

 

[kj]

Thanks for diluting my point. The OP is chasing the wrong problem. Who
cares whether a class initializer can call a method, if the method
doesn't meet its original requirements, to be callable outside the class?

And the arguments about how recursion is restricted are ridiculous.
Nothing wrong with a method calling itself, once it's got a proper
signature. You just have to make the call agree with the signature.
The problem is only that the method may not be actually called until the
class definition is complete.

As I described at length in another reply, the function in question
is not intended to be "callable outside the class". And yes,
recursive functions in Python *are* restricted in ways that
non-recursive functions aren't. The examples I've posted prove
this point unambiguously.

At this point the only defense for this restriction is to claim
that it is somehow negligible. But I disagree. It's easy to come
up with equally negligible, and equally indefensible, restrictions
to the language would be found quite unacceptable by most users.
E.g. suppose that Python's specification prohibited the use of
upper case X in an identifier. That's certainly a negligible
restriction: it is not harder at all to write great software without
the services of the letter X than with them. Still, I think most
users would find this restriction infuriatingly idiotic. Which
pretty much captures what I feel about Python's prohibition of
recursive functions within class statements.

kynn

 

[kj]

Nope, you're wrong. Class variables are accessible wherever a class
exists. The apparent silliness of this is because you are confusing
classes with class statements.

Repeating an earlier example (though I've switched the order of
the two internal functions):

class Demo(object):
def fact_iter(n):
ret = 1
for i in range(1, n + 1):
ret *= i
return ret

def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)

classvar1 = fact_iter(5)
classvar2 = fact_rec(5)


In the initialization of classvar1, fact_iter is invoked without
any problem even though the class is not yet created: no need to
qualify it with the name of the class. This is completely inconsistent
with the requirement that fact_rec be so qualified when invoked
within fact_rec.

kynn