C-style static variables in Python?

[Patrick Maupin]

The cost of a try...except is *very* low -- about the same as a pass
statement:

Agreed. In the example above, if frobnicate() is a null function, the
try/except adds about 5% to execution time on my machine. If I were
really worried about execution time, I would use a closure *for this
particular example* as I mentioned elsewhere. However, the cost of
the try/except is not zero, and when I have something I prefer looking
at (the __getattr__ doesn't clutter up the main-line execution with
conditionals for stuff that only gets used once at initialization)
that is always known to be cheaper in execution, that's what I use. I
suppose some people might not like looking at the __getattr__, but
this is a memoization technique I use quite often, so I find it
idiomatic.

Regards,
Pat

 

[Terry Reedy]

Unless one wants the intialization of mongo delayed in case spam is
never called, it can go in __init__ instead.

 

[Terry Reedy]

When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:

Here is a decorator to make a function self-aware, giving it a "this"
variable that points to itself, which you could then initialize from
outside with static flags or values:

from functools import wraps

def self_aware(fn):
@wraps(fn)
def fn_(*args):
return fn(*args)
fn_.__globals__["this"] = fn_
return fn_

In 3.1, at least, the wrapper is not needed.

def self_aware(fn):
fn.__globals__["this"] = fn
return fn

Acts the same

@self_aware
def foo():
this.counter += 1
print this.counter

foo.counter = 0

Explicit and separate initialization is a pain. This should be in a
closure or class.

foo()
foo()
foo()
Prints:

1
2
3

However, either way, the __globals__ attribute *is* the globals dict,
not a copy, so one has
<function foo at 0x00F5F5D0>

Wrapping a second function would overwrite the global binding.

Terry Jan Reedy

 

[Ethan Furman]

Unless one wants the intialization of mongo delayed in case spam is
never called, it can go in __init__ instead.

As a matter of fact, I have an object that is usually not called during
it's modules use, so I put in __getattr__. Sped the modules load time
back up to pert near instantaneous.

~Ethan~

 

[Ethan Furman]

[...]

Sounds like a personal preference issue, rather than a necessary /
unnecessary issue -- after all, if you call that function a thousand
times, only once is mongo not defined... clearly the exception.

~Ethan~

Well, I think the whole discussion has basically been about personal
preference. OTOH, but if you call the function a few million times, you
might find the cost of try/except to be something that you would rather
not incur -- it might become a performance issue rather than a personal
choice issue.

The cost of a try...except is *very* low -- about the same as a pass
statement:


1.9227982449955801


Actually catching the exception, on the other hand, is quite expensive:


10.598482743564809


The heuristic I use is, if I expect the try block to raise an exception
more than about one time in ten, I change to an explicit test. In this
case, since the exception should only be raised once, and then never
again, I would use a try...except block.

That was my reasoning as well, but when I timed it for one million runs
(so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
seconds, the try...execpt block was .6; at ten million it was 5 and 6.

At those rates, personal preference takes over, at least for me.

~Ethan~

 

[Patrick Maupin]

I don't know if properties are really faster or slower then a
__getattr__, but I find them a lot cleaner if I want to delay some
calculation until needed like that.

Well, the relative speed of properties vs. __getattr__ can become
irrelevant in at least two ways:

1) If the __getattr__ only calculates the value one time and then
stuffs it into the instance dictionary, now you are really comparing
the relative speed of properties vs. lookup of an attribute in the
instance dict. If you're at all concerned about speed, I think there
is a clear winner here.

2) There is a single __getattr__ function, vs. one property for every
attribute that needs a property. In cases where you can somehow
easily compute the attribute names as well as the attribute values,
__getattr__ can be a *lot* less code than defining dozens of
properties.

But you're absolutely right that, in many cases, property is the best
way to go for readability (especially if the property is read-only and
you're using a recent enough python to use decorators).

Regards,
Pat

 

[Steven D'Aprano]

That was my reasoning as well, but when I timed it for one million runs
(so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
seconds, the try...execpt block was .6; at ten million it was 5 and 6.

Care to share your timing code? Not that I don't trust your results, but
timings are very sensitive to the exact thing you do, and I'd like to see
what that is.

 

[Ethan Furman]

Care to share your timing code? Not that I don't trust your results, but
timings are very sensitive to the exact thing you do, and I'd like to see
what that is.

Happy to do so -- if I made a mistake I'd like to know about it and learn.

It'll have to wait two days 'til I get back to work, though... I'll post
it asap.

~Ethan~

 

[John Nagle]

When coding C I have often found static local variables useful for
doing once-only run-time initializations.

If you want functions with state, use an object. That's what they're
for. Don't muck with the internal representation of functions.

John Nagle

 

[Patrick Maupin]

    If you want functions with state, use an object. That's what they're
for.  Don't muck with the internal representation of functions.

While "Don't muck with the internal representation of functions" is
excellent advice over 99% of the time, it is also true that it is
often possible, sometimes even encouraged, to have "functions with
state."

This is done without "mucking" and without explicitly declaring a
class or a class instance. See, e.g. closures and generator
functions.

Regards,
Pat

 

[Ethan Furman]

Happy to do so -- if I made a mistake I'd like to know about it and learn.

It'll have to wait two days 'til I get back to work, though... I'll post
it asap.

Well, so much for asap, but here's what I used (with one correction: in
the 'if' code I had forgotten to actually reference the missing
attribute, so the __getattr__ look up never happened; now the
try...except block is /slightly/ faster, as opposed to 20% slower).


class spam_except(object):
def __call__(self, x, y, z):
try:
mongo = self.mongo
except AttributeError:
mongo = self.mongo = 1
return

class spam_if(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = 1
return self.mongo
def __call__(self, x, y, z):
self.mongo # didn't have this line before. d'oh!
return


--> timeit.Timer('spammer(1,2,3)','from spam import spam_except;
spammer=spam_except()').timeit()
0.65764130543749388

--> timeit.Timer('spammer(1,2,3)','from spam import spam_if;
spammer=spam_if()').timeit()
0.66972877235545525

~Ethan~

 

[Patrick Maupin]

(Posted some code with a timeit...)

Well, I'm not going to debug this, but with the *original* thing you
posted, and the thing I posted, with a call and everything (more
realistic scenario), the exception version seems slower on my machine:

#!/usr/bin/env python

import timeit

def frobnicate(a,b,c,d):
pass

def heavy_lifting_at_runtime():
print 'heavy'

class spam_except(object):
def __call__(self, x, y, z):
try:
mongo = self.mongo
except AttributeError:
mongo = self.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)
se = spam_except()


class spam_if(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = heavy_lifting_at_runtime()
return self.mongo
def __call__(self, x, y, z):
return frobnicate(x, y, z, self.mongo)
si = spam_if()

tse = timeit.Timer('se(1,2,3)', "from __main__ import se")
tsi = timeit.Timer('si(1,2,3)', "from __main__ import si")

for i in range(5):
ve = tse.timeit(10000000)
vi = tsi.timeit(10000000)
print ve, vi, '%.1f' % ((ve-vi) / vi * 100)

------

heavy
heavy
5.45695090294 5.10844397545 6.8
5.43381404877 5.01345705986 8.4
5.42474508286 5.02641201019 7.9
5.40713405609 5.04178905487 7.2
5.38063693047 4.96194696426 8.4

The output indicates that the exception one is, on average, around
7.5% slower.

Regards,
Pat

 

[Albert van der Horst]

I suspect you mean

foo = make_foo()

I don't think I'd ever want to use such an obscure technique in a
program. You might want to consider using functools.wraps to make sure
that the foo function looks right.

Imagine that heavy_lifting is only ever used here and uses 4 Gbyte of core.
Suddenly deleting those function objects seems the right thing to do,
instead of an obscure technique.
(I'm not sure the Python compiler could take advantage of this,
I know I could in my Forth compiler, under circumstances.)

regards
Steve

Groetjes Albert