Help me dig my way out of nested scoping

[Brendan]

Hi everyone

I'm new to Python, so forgive me if the solution to my question should
have been obvious. I have a function, call it F(x), which asks for two
other functions as arguments, say A(x) and B(x). A and B are most
efficiently evaluated at once, since they share much of the same math,
ie, A, B = AB(x), but F wants to call them independantly (it's part of
a third party library, so I can't change this behaviour easily). My
solution is to define a wrapper function FW(x), with two nested
functions, AW(x) and BW(x), which only call AB(x) if x has changed.

To make this all clear, here is my (failed) attempt:

#------begin code ---------

from ThirdPartyLibrary import F
from MyOtherModule import AB

def FW(x):
lastX = None
aLastX = None
bLastX = None

def AW(x):
if x != lastX:
lastX = x
# ^ Here's the problem. this doesn't actually
# change FW's lastX, but creates a new, local lastX

aLastX, bLastX = AB(x)
return aLastX

def BW(x):
if x != lastX:
lastX = x
# ^ Same problem

aLastX, bLastX = AB(x)
return bLastX

#finally, call the third party function and return its result
return F(AW, BW)

#-------- end code ---------

OK, here's my problem: How do I best store and change lastX, A(lastX)
and B(lastX) in FW's scope? This seems like it should be easy, but I'm
stuck. Any help would be appreciated!

-Brendan

 

[Michael Spencer]

Hi everyone

I'm new to Python, so forgive me if the solution to my question should
have been obvious.

....
Good question. For a thorough explanation see:
http://www.python.org/dev/doc/devel/ref/naming.html

Simple version follows:

OK, here's my problem: How do I best store and change lastX, A(lastX)
and B(lastX) in FW's scope? This seems like it should be easy, but I'm
stuck. Any help would be appreciated!

Assignments (i.e., binding names to objects) are always made in the local scope
(unless you've used the 'global' declaration, which I don't think can help you
here). So, for an even simpler demonstration of the problem see: ... b = 1
... def inner():
... b += 1
... print b
... inner()
... Traceback (most recent call last):
File "<input>", line 1, in ?
File "<input>", line 6, in outer
File "<input>", line 4, in inner
UnboundLocalError: local variable 'b' referenced before assignment

The solution is not to re-bind the identifier from the enclosing scope, but
rather to mutate the object that it references. This requires a mutable object,
such as a list:
... b = [1] # bind b to a mutable object
... def inner():
... b[0] += 1
... print b[0]
... inner()
...


HTH
Michael

 

[James Stroud]

I wish I had time to dig into your specific problem because it looks
interesting. But I think you might want to look at "python generators". I
beleive there is no reason that they can't yield a function.

http://www.python.org/peps/pep-0255.html
http://docs.python.org/ref/yield.html
http://linuxgazette.net/100/pramode.html

James

Hi everyone

I'm new to Python, so forgive me if the solution to my question should
have been obvious. I have a function, call it F(x), which asks for two
other functions as arguments, say A(x) and B(x). A and B are most
efficiently evaluated at once, since they share much of the same math,
ie, A, B = AB(x), but F wants to call them independantly (it's part of
a third party library, so I can't change this behaviour easily). My
solution is to define a wrapper function FW(x), with two nested
functions, AW(x) and BW(x), which only call AB(x) if x has changed.

To make this all clear, here is my (failed) attempt:

#------begin code ---------

from ThirdPartyLibrary import F
from MyOtherModule import AB

def FW(x):
lastX = None
aLastX = None
bLastX = None

def AW(x):
if x != lastX:
lastX = x
# ^ Here's the problem. this doesn't actually
# change FW's lastX, but creates a new, local lastX

aLastX, bLastX = AB(x)
return aLastX

def BW(x):
if x != lastX:
lastX = x
# ^ Same problem

aLastX, bLastX = AB(x)
return bLastX

#finally, call the third party function and return its result
return F(AW, BW)

#-------- end code ---------

OK, here's my problem: How do I best store and change lastX, A(lastX)
and B(lastX) in FW's scope? This seems like it should be easy, but I'm
stuck. Any help would be appreciated!

-Brendan

--
James Stroud, Ph.D.
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[Ron_Adam]

Hi everyone

I'm new to Python, so forgive me if the solution to my question should
have been obvious. I have a function, call it F(x), which asks for two
other functions as arguments, say A(x) and B(x). A and B are most
efficiently evaluated at once, since they share much of the same math,
ie, A, B = AB(x), but F wants to call them independantly (it's part of
a third party library, so I can't change this behaviour easily). My
solution is to define a wrapper function FW(x), with two nested
functions, AW(x) and BW(x), which only call AB(x) if x has changed.

You have several easy choices, that would not require you modifying
your program much.

1. Use the 'global' keyword to declare lastX, aLastX, and bLastX as
globals, then all functions will have access to them.

def FW(x):
global lastX, aLastX, bLastX

2. Use function attributes, which are just names attached to the
function using a '.'.

def FW(x):
#
# Function body here
#
return F(AW, BW)

FW.lastX = None
FW.aLastX = None
FW.bLastX = None

result = FW(x)


You will need to always include the FW. in front of those names.


3. Something else, that may help is you can return more than one value
at a time. Python has this neat feature that you can have multiple
items on either side of the '=' sign.

a,b,c = 1,2,3

same as:
a=1
b=2
c=3

And it also works with return statements so you can return multiple
value.

def abc(n):
return n+1, n+2, n+3

a,b,c = abc(0)


5. Choice 5 and above is to rewrite your function as a class. Names
in class's retain their values between calls and you can access those
values the same way as accessing function attributes.


Hope this helped.

Cheers,
Ron

 

[Terry Hancock]

from ThirdPartyLibrary import F
from MyOtherModule import AB

def FW(x):
lastX = None
aLastX = None
bLastX = None

I'm pretty sure your method will work if you just specify
that these are global:

def FW(x):
global lastX = None
global aLastX = None
global bLastX = None

OTOH, I'm biased against using module-level variables
for this kind of purpose and I think things that retain
state really ought to be class instances, so I'd probably replace
AB(x) with a callable object, and define two wrappers to access it
(untested):

class AB:
def __init__(self):
self._last_x = None
self._last_a = None
self._last_b = None

def __call__(self, x):
if x == self._last_x:
return self._last_a, self._last_b
else:
self._last_a, self._last_b = self.AB(x)
return self._last_a, self._last_b

def A(self, x):
return self(x)[0]

def B(self, x):
return self(x)[1]

def AB(self, x):
"""
This is where you compute your new values when needed.
"""
# something that computes a and b
return a,b

ab = AB()

Then you actually pass the methods ab.A and ab.B to your
library routine. This will usually work, though if it somehow
insists on an actual function instead of a callable, you can always
use wrapper functions.

This also has the advantage that you *can* process more than
one case at a time (i.e. if you have two different places where you
need this function to be called and you aren't sure what order
they'll be processed (or don't want to think about it), you can
give them different instances of AB to work with, and they'll
remember their previous calls separately.

Cheers,
Terry

 

[Terry Reedy]

I have a function, call it F(x), which asks for two
other functions as arguments, say A(x) and B(x). ...

If I understand this and the rest, a third party library whose code you
cannot modify (easily) has a function F with (at least) three parameters:
A, B, and x. During its operation, F calls A(x) and B(x). Because of code
commonality for the particular A and B arg funcs you want to feed to F, you
want avoid duplication by having the first call to either to calculate both
return values.

If F calls each of A and B exactly once and always in the same order and
only for the value x you supply, the solution is pretty easy. A calls AB,
stashes the B value away where it can be retrieved, and return the A value.
B retrieves the B value and returns it. But your problem is the stash and
retrieve part. Solutions:
1. global variable (easiest) - use global declaration in A;
2. closure variable - use mutable such as 1 element list (see below);
3. instance attribute - with A and B as methods.

2 is what you tried to do, but without knowing the mutable (list or dict)
trick:

def ABwrapper():
bsave = [None]
def A(x):
aval,bval = AB(x)
bsave[0] = bval
return aval
def B(x):
return bsave[0]
return A,B

This works because A does not try to *rebind* bsave to a new object. It
only mutates the existing object.

If the order of calling changes, you need more logic. If F calls A and B
on multiple 'x's, as with, for instance, a derivative approximizer, then I
would memoize A and/or B using the recipe posted here more than once and on
the cookbook site and included in the new Python Cookbook v2 (and maybe v1,
don't have it).

Terry J. Reedy

 

[Brendan]

Thanks for the tips. Making FW a callable class (choice 5) seems to be
a good (if verbose) solution. I might just wrap my temporary values in
a list [lastX, lastA, lastB] and mutate them as Michael suggests.
Thanks to Michael especially for the explanation of the name-binding
process that's at the heart of the issue.

The other choicess are not as helpful to me for the following reasons:

choice 1: I don't want the temporary values of lastA and lastB to be
global variables in my case as they are great big numeric arrays, and
I'd like their memory to be reclaimed after FW is done.

choice 2: I tried this without success. Using Micheal's example, I
would assume you mean something like this:

def outer():
b = 1
def inner():
outer.b += 1
print outer.b
inner()
outer()

Which gives me:
AttributeError: 'function' object has no attribute 'b'

Perhaps I misapplied this method?

choice 3: I know that Python can return multiple values in one line,
but I don't think that applies here. My library function F, is looking
for two separate function arguments

 

[Brendan]

F -is- in fact an iterative optimizer that minimizes A on x (B is the
derivative of A). So yes, F will call A and B on mulitple 'x's. In
that case, it seems the mutable object trick is the way to go. Thanks.

I didn't follow your last sentence. What about the Python Cookbook?

 

[Brendan]

James Stroud Apr 3, 3:18 pm:
I think you might want to look at "python generators".

I've seen discussion of generators before, but haven't invested the
time to understand them yet. This might be a good excuse.

 

[Ron_Adam]

Thanks for the tips. Making FW a callable class (choice 5) seems to be
a good (if verbose) solution. I might just wrap my temporary values in
a list [lastX, lastA, lastB] and mutate them as Michael suggests.
Thanks to Michael especially for the explanation of the name-binding
process that's at the heart of the issue.

The other choicess are not as helpful to me for the following reasons:

choice 1: I don't want the temporary values of lastA and lastB to be
global variables in my case as they are great big numeric arrays, and
I'd like their memory to be reclaimed after FW is done.

Generally global variables should be avoided in python if you are
doing a large application. For smaller ones, they are ok, but they
are just a little slower than local variables.

You could use a classic class which is a good way to store a single
group of data. The 'del' will unbind a name from an object so the
objects can be garbage collected.

class data:
A = []
B = []

def countupdown():
for n in xrange(11):
data.A.append(n)
data.B.append(10-n)
print data.A
print data.B

countupdown()

# store data # Check out pickle module for this.

del data

choice 2: I tried this without success. Using Micheal's example, I
would assume you mean something like this:

def outer():
def inner():
outer.b += 1
print outer.b

inner()

outer.b = 1 # <-- initialize here after function of same name
outer()

# save data method here

del outer # delete outer and it's attributes

 

[Terry Reedy]

F -is- in fact an iterative optimizer that minimizes A on x (B is the
derivative of A). So yes, F will call A and B on mulitple 'x's. In
that case, it seems the mutable object trick is the way to go. Thanks.

As long as it calls A and B in that same order, rather than A several times
and B several times, then my code, fleshed out, will probably work.

I didn't follow your last sentence. What about the Python Cookbook?

It is a book that is one place to find memoizer code which allows one to
save several x,f(x) pairs at once for later reuse. But you don't seem to
need it since an optimizer should never return to the exact same x except
by accident.

Terry J. Reedy