Importing variables non-deterministic?

[tmellman]

I have this file that has this:

from struct_global import y
from struct_global import x

and this usage (a few lines later in a class method definition in the same file):

if y == x:

If I stop there in pdb, I can observe this:

(Pdb) p x
66
(Pdb) p y
-1
(Pdb) from struct_global import y
(Pdb) p y
62

How did my first import get hidden? Is there any way to see where a variable resolves to?

 

[Steven D'Aprano]

I have this file that has this:

from struct_global import y
from struct_global import x

The import of x appears to be irrelevant, so I'll ignore it.

Here you import y from the struct_global module. This creates a new name
y in the current module. y is bound to the same object that
struct_global.y happens to be at the moment of the import, but y is *not*
a reference to the *name* struct_global.y.

This means that the two names, "y in current module" and "y in
struct_global module" point to the same object, but they aren't two
aliases for the same variable. They are different variables that just
happen to share the same value (an object).

If the object mutates (which ints cannot do), then both "y"s will see the
change (naturally, since both refer to the same object). But if either
name is rebound to a different object, the other name is unaffected.

We can see the same behaviour locally, without an import, by doing this:

py> x = [1]
py> y = x # y is bound to the same object as x
py> print(y)
[1]
py> x.append(2) # mutate x in place
py> print(y)
[1, 2]
py> x = [1, 2, 3] # re-bind x to a new object
py> print(y) # y still bound to the first object
[1, 2]


"from struct_globals import y" is a binding operation, no different from
"y = something".

and this usage (a few lines later in a class method definition in the
same file):

if y == x:

If I stop there in pdb, I can observe this:

(Pdb) p x
66
(Pdb) p y
-1
(Pdb) from struct_global import y
(Pdb) p y
62

And here you re-import the name "y" from struct_global. That rebinds the
current module's "y" with whatever value struct_global.y has *now*,
rather than a second (or a minute, or an hour) earlier when the first
import took place. Obviously at some point between the first import and
the second import, struct_global.y must have been reassigned from -1 to
62.

This goes to show why global variables are considered harmful, and why
clean, modern program design tries to reduce the use of them as much as
possible. Global variables are too easily modified by, well, *anything*.
The sort of behaviour you are seeing is sometimes called "action at a
distance" -- something, anything, anywhere in your program, possibly
buried deep, deep down inside some function you might never suspect, is
changing the global variable.

How did my first import get hidden?

You reassigned to it. "from struct_global import y" is practically
syntactic sugar for this:

import struct_global
y = struct_global.y

Is there any way to see where a
variable resolves to?

You just did. You inspected the variable "y", and you saw that it is
assigned the value 62.

 

[Antoon Pardon]

Op 17-08-13 17:01, Steven D'Aprano schreef:

And here you re-import the name "y" from struct_global. That rebinds the
current module's "y" with whatever value struct_global.y has *now*,
rather than a second (or a minute, or an hour) earlier when the first
import took place. Obviously at some point between the first import and
the second import, struct_global.y must have been reassigned from -1 to
62.

This goes to show why global variables are considered harmful, and why
clean, modern program design tries to reduce the use of them as much as
possible. Global variables are too easily modified by, well, *anything*.
The sort of behaviour you are seeing is sometimes called "action at a
distance" -- something, anything, anywhere in your program, possibly
buried deep, deep down inside some function you might never suspect, is
changing the global variable.

I think you are overstating your case. Classes and functions are
variables too and in general nobody seems to have a problem with them
being global.

 

[Dave Angel]

Op 17-08-13 17:01, Steven D'Aprano schreef:

I think you are overstating your case. Classes and functions are
variables too and in general nobody seems to have a problem with them
being global.

It's global *variables* that are to be avoided. constants like clsases
and functions are fine. On the other hand, class attributes can be
variable, and thus are to be avoided when reasonable.

There *are* places where global variables make sense, such as for
caching, or counting. But those are typically hidden, so they are
global in lifetime, but not in scope.

 

[Antoon Pardon]

Op 19-08-13 09:45, Dave Angel schreef:

It's global *variables* that are to be avoided. constants like clsases
and functions are fine. On the other hand, class attributes can be
variable, and thus are to be avoided when reasonable.

Python has no constants. Classes and functions can be changed just like
any other variable. I agree that classes and function are generally
meant to be constant, but often enought so are global int variables.

And some of those that do change, only do so in the initialisation phase
and should be considered constant for the rest of the program.

My point was that Steven has no way of knowing what exactly is going on
here and so shouldn't be making such a sweeping statement.

 

[Chris Angelico]

Op 19-08-13 09:45, Dave Angel schreef:

Python has no constants. Classes and functions can be changed just like
any other variable. I agree that classes and function are generally
meant to be constant, but often enought so are global int variables.

# telnet.py
IAC = 0xFF
GA = 0xF9
WILL = 0xFB
WONT = 0xFC
DO = 0xFD
DONT = 0xFE


# connection.py
from telnet import IAC,DO,DONT


To be sure, Python won't stop me from changing the value of DONT. But
it's a constant, and its value is defined elsewhere (RFC 854). (In
this instance, an enum would probably be the better option; but this
is an example of a more general case.)

In connection.py, I don't care that I've taken a "copy" of the integer
0xFE. It's never going to change; it is a constant in the best
tradition of "named number". I could hard-code 0xFE everywhere and the
code would work *just fine*, but it'd be less readable, so I don't.

Python does have constants. It just doesn't have interpreter support
for them. Same as private members, in fact.

ChrisA

 

[Antoon Pardon]

Op 19-08-13 10:32, Chris Angelico schreef:

# telnet.py
IAC = 0xFF
GA = 0xF9
WILL = 0xFB
WONT = 0xFC
DO = 0xFD
DONT = 0xFE


# connection.py
from telnet import IAC,DO,DONT


To be sure, Python won't stop me from changing the value of DONT. But
it's a constant, and its value is defined elsewhere (RFC 854). (In
this instance, an enum would probably be the better option; but this
is an example of a more general case.)

This is irrelevant. That some context defines a constant, and that you
can use a variable with the same name as a constant in python, doesn't
contradict the statement that python (as a language) doesn't has
constants. There is nothing in the language that would prevent buggy
code from changing any of those variables. So from a python point of
views these are just global variables. Just as the struct_global.y was
in the original contribution.

 

[Chris Angelico]

This is irrelevant. That some context defines a constant, and that you
can use a variable with the same name as a constant in python, doesn't
contradict the statement that python (as a language) doesn't has
constants. There is nothing in the language that would prevent buggy
code from changing any of those variables. So from a python point of
views these are just global variables. Just as the struct_global.y was
in the original contribution.

And there's nothing preventing a program from using ctypes to
overwrite an object's refcount, thus causing a segfault. So? The issue
was regarding imports, and it's perfectly safe to import a constant,
even if the interpreter doesn't protect you from then being a total
idiot and changing it.

ChrisA

 

[Peter Otten]

And there's nothing preventing a program from using ctypes to
overwrite an object's refcount, thus causing a segfault. So? The issue
was regarding imports, and it's perfectly safe to import a constant,
even if the interpreter doesn't protect you from then being a total
idiot and changing it.

Come on, breaking a "gentlemen's agreement" by rebinding a name and using
ctypes to manipulate the internal state of the cpython imlementation are
very different things. If that proves anything I'll best you with a Python
script that controls a robot which in turn destroys the computer running the
script with a hammer

If Antoon's point is that global "constants" in Python are only constants
because the programmer thinks of them that way I'd say that's an important
addition to put Steve's "global variables are considered harmful" into
perspective.

 

[Antoon Pardon]

Op 19-08-13 11:18, Chris Angelico schreef:

And there's nothing preventing a program from using ctypes to
overwrite an object's refcount, thus causing a segfault. So?

Yes so? What is your point? Since Cpython caches small integers
there is probably nothing preventing a program from using ctypes
to make the number three behave as the number five. That doesn't
mean that the language define three and five to be equal.

A language having constants, does mean that the language can prevent
a name from being reassigned.

The issue
was regarding imports, and it's perfectly safe to import a constant,
even if the interpreter doesn't protect you from then being a total
idiot and changing it.

Python doesn't have constants, so you statement about importing a
constant doesn't make sense. The point is that python doesn't provide
the mechanism for protecting names against reassignments. So you
don't know whether the variable you think of as a constant is so
in reality. And this from a pure language definition point of view.
That you can use tools that make the interpreter no longer behave
as the language should, doesn't negate that.

 

[Dave Angel]

Op 19-08-13 11:18, Chris Angelico schreef:

Python doesn't have constants, so you statement about importing a
constant doesn't make sense. The point is that python doesn't provide
the mechanism for protecting names against reassignments. So you
don't know whether the variable you think of as a constant is so
in reality. And this from a pure language definition point of view.
That you can use tools that make the interpreter no longer behave
as the language should, doesn't negate that.

Who cares what the language "protects?" I don't know any language
whose protections can't be at least partially bypassed by clever
foot-shooters. In any case, we all know that Python doesn't protect
constants, so we're free to use the word in a friendlier way.

A Python constant is what I use as a constant. I follow Pep-8 and make
it all caps. So in any library I write struct_global.y would be a bug
or a design flaw. (And with a single character name like that, it
wouldn't be global in any case. Single character names are reserved
for play code and for short loops)

And if I subsequently change it in my calling code, it isn't a
constant any more. If I rebind the name, it's not even the same
variable any more.

That's a bug, not an exception to the rule "don't use global variables."

I also accept as a constant those values which are initialized
sufficiently early in the code that most places will only ever see the
final value. Those may be global without worry.

 

[Antoon Pardon]

Op 19-08-13 14:33, Dave Angel schreef:

Who cares what the language "protects?" I don't know any language
whose protections can't be at least partially bypassed by clever
foot-shooters.

So? A number of language designers seem to think it is still worth
while. And even as it may be possible to circumvent it, when we are
talking about whether a language has constants or not, we are
talking about this language feature. Whether you think this
feature is worth having or not, doesn't change that this is
what is meant when people talk about a language having constants
or not.

In any case, we all know that Python doesn't protect
constants, so we're free to use the word in a friendlier way.

A Python constant is what I use as a constant.

No it is not. It can be a project constant, but it is not a
python constant. As it turns out, python does have some constants
if the version is high enough, being True, False and None. So
for those values the python designers thought it valuable enough
to protect them against reassignment. Pity enough they chose a
way that didn't allow programmers to protect names they thought
important enough to do so too.

I follow Pep-8 and make
it all caps. So in any library I write struct_global.y would be a bug
or a design flaw.

No it is not. People are not obligated to follow pep-8 in their
projects and not following pep-8 doesn't imply a design flaw.

(And with a single character name like that, it
wouldn't be global in any case. Single character names are reserved
for play code and for short loops)

So? What if it is play code. You still don't know whether this person
meant this variable in his play code to be used as a constant or not.

 

[Chris Angelico]

Pity enough they chose a
way that didn't allow programmers to protect names they thought
important enough to do so too.

As of Python 3, we can redefine something that used to be a keyword,
which is even stronger than a constant.

def print(*args,print=print,**kw):
print("##",*args,**kw)

What easier way to tag all your print calls? And this is why it helps
to have nothing technically constant. You should be able to do the
same with anything, without having to go and edit someone else's code
to remove the 'const' keyword.

ChrisA

 

[Steven D'Aprano]

Op 19-08-13 09:45, Dave Angel schreef:

Python has no constants. Classes and functions can be changed just like
any other variable. I agree that classes and function are generally
meant to be constant, but often enought so are global int variables.

You are technically correct, but missing the point. If I wrote code that
went around reassigning names from one function to another function, I
would very likely soon work myself into a state of utter confusion:

def func(x):
...

# later
save_func = func
func = lambda x, y: do_stuff(x, 3*y)-4
result = something_that_calls_func()
func = save_func


Nasty, horrible code, yes? But it's nasty and horrible because "func" is
bound to a function, it would be equally nasty and horrible if it was a
data type (a string, a list, an int, a flag, ...) instead.

Since classes and functions are First Class objects in Python, naturally
if you treat them as global *variables* rather than global *constants*
you can end up with problems. The problem is not the global part alone.
Global constants, or pseudo-constant-by-convention-only, are fine.

And some of those that do change, only do so in the initialisation phase
and should be considered constant for the rest of the program.

My point was that Steven has no way of knowing what exactly is going on
here and so shouldn't be making such a sweeping statement.

On the contrary, I can read the Original Poster's description of the
problem, and then use my ability to reason to deduce what the most likely
explanation was.

Now of course I might be wrong. I don't claim infallibility or
omniscience. But I'm not an idiot, and if the OP says that a global
variable has one value at one time, and then some time later has a
different value, there are two likely possibilities:

* The variable was changed by some other piece of code, i.e. it
actually was being used as a *global variable*.

* The OP is completely confused, and (inadvertently, we hope)
reporting things which are actually not true.

Much less likely:

* Despite 15+ years of experience with Python, and reading the
documentation, I have failed to notice that, yes, the OP is
correct and importing variables is non-deterministic.

* There's a bug in Python.


Both of these are theoretically possible, but the chance of them is
somewhere between Buckley's and None.


Of course, if you have an alternate explanation for the issue the OP
reported, I would love to hear it. Maybe I have completely misunderstood
the situation.

 

[Steven D'Aprano]

Python does have constants. It just doesn't have interpreter support for
them. Same as private members, in fact.

"Constant by convention".

I wish Python had stronger support for enforcing constantness, to whit,
some way to say "you can't rebind or delete this name once it is bound".
You can do it with attributes, by use of property, or in C extensions,
but you cannot do it with top-level name bindings. It makes me terribly
sad that you can do this:

import math
math.pi = 3.0


although I can't decide whether I am less sad or more sad to see that the
behaviour of math.sin and friends doesn't depend on math.pi.

Now, of course, and I don't expect any such "constants" to be proof
against a dedicated attacker. One can work around read-only properties,
and I expect that any future "no-rebind" constants will also be capable
of being worked around. This is Python, not Haskell or Ada.

But, naming convention or no naming convention, it is still valuable to
get an exception if you accidentally rebind something that you shouldn't
rebind.

 

[Steven D'Aprano]

def func(x):
...

# later
save_func = func
func = lambda x, y: do_stuff(x, 3*y)-4 result =
something_that_calls_func()
func = save_func


Nasty, horrible code, yes? But it's nasty and horrible because "func" is
bound to a function, it would be equally nasty and horrible if it was a
data type (a string, a list, an int, a flag, ...) instead.

Oops, I left out a word. It is NOT nasty and horrible specifically
because "func" is bound to a function, it would be equally horrible if it
were a data type.

 

[Chris Angelico]

Oops, I left out a word. It is NOT nasty and horrible specifically
because "func" is bound to a function, it would be equally horrible if it
were a data type.

I was having a little trouble parsing that! Thank you for the
Iolanthe-style[1] update.

ChrisA
[1] http://math.boisestate.edu/gas/iolanthe/web_op/iol23d.html

 

[Antoon Pardon]

Op 19-08-13 17:04, Chris Angelico schreef:

As of Python 3, we can redefine something that used to be a keyword,
which is even stronger than a constant.

def print(*args,print=print,**kw):
print("##",*args,**kw)

What easier way to tag all your print calls? And this is why it helps
to have nothing technically constant. You should be able to do the
same with anything, without having to go and edit someone else's code
to remove the 'const' keyword.

But what you are showing here is shadowing, not redefintion and you can
do that even if the shadowed name would be a constant.

 

[Antoon Pardon]

Op 19-08-13 18:57, Steven D'Aprano schreef:

You are technically correct, but missing the point. If I wrote code that
went around reassigning names from one function to another function, I
would very likely soon work myself into a state of utter confusion:

def func(x):
...

# later
save_func = func
func = lambda x, y: do_stuff(x, 3*y)-4
result = something_that_calls_func()
func = save_func


Nasty, horrible code, yes? But it's nasty and horrible because "func" is
bound to a function, it would be equally nasty and horrible if it was a
data type (a string, a list, an int, a flag, ...) instead.

Since classes and functions are First Class objects in Python, naturally
if you treat them as global *variables* rather than global *constants*
you can end up with problems. The problem is not the global part alone.
Global constants, or pseudo-constant-by-convention-only, are fine.

I don't think there is a big disagreement on this, I just thought
you overstated your case as you originally worded it.

On the contrary, I can read the Original Poster's description of the
problem, and then use my ability to reason to deduce what the most likely
explanation was.

Now of course I might be wrong. I don't claim infallibility or
omniscience. But I'm not an idiot, and if the OP says that a global
variable has one value at one time, and then some time later has a
different value, there are two likely possibilities:

* The variable was changed by some other piece of code, i.e. it
actually was being used as a *global variable*.

Sure. But was that because it was intended to be used as a variable
or was the intention to use it as a constant but because of a bug
it was changed anyway? I don't think you have enough information
to discriminate between these two cases and for the possibility
of this being the latter case I found your disgression into the
harm of global variable too strongly worded.

 

[Antoon Pardon]

Op 19-08-13 19:05, Steven D'Aprano schreef:

I wish Python had stronger support for enforcing constantness, to whit,
some way to say "you can't rebind or delete this name once it is bound".
You can do it with attributes, by use of property, or in C extensions,
but you cannot do it with top-level name bindings. It makes me terribly
sad that you can do this:

import math
math.pi = 3.0


although I can't decide whether I am less sad or more sad to see that the
behaviour of math.sin and friends doesn't depend on math.pi.

Why should you expect math.sin and friends be dependant on math.pi?
AfAIR the numerical algorithms for calulating sin and friends don't
depend on (the value of) pi. So there is no reason to suspect that
altering math.pi would have any effect on the results of these
functions.