changing local namespace of a function

B

Bo Peng

Dear list,

I have many dictionaries with the same set of keys and I would like to
write a function to calculate something based on these values. For
example, I have

a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

def fun(dict):
dict['z'] = dict['x'] + dict['y']

fun(a) and fun(b) will set z in each dictionary as the sum of x and y.

My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun. Is this
possible? The ideal code would be:

def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y

Many thanks in advance.
Bo
 
M

M.E.Farmer

Hello Bo,
Don't use dict it is a builtin ;)

Also try this stuff out in an interpreter session it is easy and fast
to get your own answers.
.... __dict__ = d
.... return __dict__
hth,
M.E.Farmer
 
M

Michael Spencer

Bo said:
Dear list,

I have many dictionaries with the same set of keys and I would like to
write a function to calculate something based on these values. For
example, I have

a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

def fun(dict):
dict['z'] = dict['x'] + dict['y']

fun(a) and fun(b) will set z in each dictionary as the sum of x and y.

My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun. Is this
possible? The ideal code would be:

def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y
As you no doubt have discovered from the docs and this group, that isn't doable
with CPython.

If you must write your functions as real functions, then you might do something
like this:
... del kw #Careful of unwanted names in locals with this approach
... z = x + y
... return locals()
...

Alternatively, you could use exec:

Michael
 
J

Jeff Shannon

Bo said:
My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun.

This sounds to me like you're trying to re-implement object orientation.

Turn all of those functions into methods on a class, and instead of
creating dictionaries that'll be passed into functions, create class
instances.

class MyClass(object):
def __init__(self, **kwargs):
for key, val in kwargs:
setattr(self, key, val)
def fun(self):
self.z = self.y + self.x

a = MyClass(x=1, y=2)
a.fun()
print a.z

Jeff Shannon
Technician/Programmer
Credit International
 
B

Bo Peng

M.E.Farmer said:
... __dict__ = d
... return __dict__
hth,


Does not work?
.... __dict__ = d
.... print locals()
.... z = x + y{'__dict__': {'y': 2, 'x': 1}, 'd': {'y': 2, 'x': 1}}
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/tmp/python-6377tax.py", line 4, in fun
NameError: global name 'x' is not defined
 
B

Bo Peng

Michael said:
As you no doubt have discovered from the docs and this group, that isn't
doable with CPython.

Too bad to know this.
... del kw #Careful of unwanted names in locals with this approach
... z = x + y
... return locals()

There are hundreds of items in the dictionary (that will be needed in
the calculation) so passing the whole dictionary is a lot better than
passing individual items.
Alternatively, you could use exec:

Yes. I thought of using exec or eval. If there are a dozen statements,

def fun(d):
exec 'z = x + y' in globals(), d

seems to be more readable than

def fun(d):
d['z'] = d['x'] + d['y']

But how severe will the performance penalty be?

Bo
 
B

Bo Peng

Jeff said:
This sounds to me like you're trying to re-implement object orientation.

I have no control over the big dictionaries. All I need to do is
processing them in situ --- that is to say, go into each map and
manipulate numbers. Parameter passing should be avoid whenever possible
since involved number of items are huge.

It is basically like

exec 'statements' in d

but a function with local dictionary d would be best.

Bo
 
M

Michael Spencer

Bo said:
Michael Spencer wrote:

>
There are hundreds of items in the dictionary (that will be needed in
the calculation) so passing the whole dictionary is a lot better than
passing individual items. ....

def fun(d):
exec 'z = x + y' in globals(), d

seems to be more readable than

def fun(d):
d['z'] = d['x'] + d['y']

But how severe will the performance penalty be?

Try it and see.

Compare it with Jeff Shannon's suggestion, and with a lazy dict-wrapper like this:
... """Lazy attribute access to dictionary keys. Will not access
... keys that are not valid attribute names!"""
... def __init__(self, mydict):
... object.__setattr__(self, "mydict",mydict)
... def __getattr__(self, attrname):
... return self.mydict[attrname]
... def __setattr__(self, attrname, value):
... self.mydict[attrname] = value
...
... ... d.z = d.x + d.y
...
 
N

Nick Coghlan

Michael said:
As you no doubt have discovered from the docs and this group, that isn't
doable with CPython.

Not entirely impossible:

Py> def f(d):
.... exec "locals().update(d)"
.... return x + y
....
Py> f(dict(x=1, y=2))
3

Due to the way 'exec' is implemented, modifications to locals() inside an exec
statement actually take effect (basically, they're freeloading on the code which
allows 'exec "x = 1"' to work properly).

This is an evil, evil hack and almost certainly not what anyone should be doing.
Also, variables created this way will be slower than normal variables due to the
way the associated code works.

Cheers,
Nick.
 
N

Nick Coghlan

Bo said:
I have no control over the big dictionaries. All I need to do is
processing them in situ --- that is to say, go into each map and
manipulate numbers. Parameter passing should be avoid whenever possible
since involved number of items are huge.

No, all parameter passing does is pass a pointer to the dictionary - it doesn't
copy the dictionary itself.
It is basically like

exec 'statements' in d

but a function with local dictionary d would be best.

Why? How do you know that the performance is unacceptable?

If you *really* want to execute a function with different locals, then you can do:

Py> def f():
.... print x + y
....
Py> exec f.func_code in dict(x=1, y=2)
3

This is unreliable though, since name binding doesn't work correctly:

Py> def f():
.... z = x + y
....
Py> d = dict(x=1, y=2)
Py> exec f.func_code in d
Py> d['z]
Py> d['z']
Traceback (most recent call last):
File "<stdin>", line 1, in ?
KeyError: 'z'
Py> exec "z = x + y" in d
Py> d['z']
3

An approach like Michael's is going to be much easier to debug, much easier to
understand later, and far more reliable all the way along. The performance is
almost certainly going to be acceptable.

Cheers,
Nick.
 
M

Michael Spencer

Nick said:
Not entirely impossible:

Py> def f(d):
... exec "locals().update(d)"
... return x + y
...
Py> f(dict(x=1, y=2))
3

Due to the way 'exec' is implemented, modifications to locals() inside
an exec statement actually take effect (basically, they're freeloading
on the code which allows 'exec "x = 1"' to work properly).

This is an evil, evil hack and almost certainly not what anyone should
be doing. Also, variables created this way will be slower than normal
variables due to the way the associated code works.

Cheers,
Nick.
Oooh - evil indeed, but thanks for the pointer.

I debated including a link to one of the 'writable locals' threads, when I
settled on not 'doable', but gambled on being probably useful rather than
certainly accurate. Just goes to show you can't get away with anything in this
NG ;-)

Cheers

Michael
 
B

Bo Peng

Thank all for your suggestions. I have tried all methods and compared
their performance.
.... def fun1(d):
.... for i in xrange(0,N):
.... d['z'] = d['x'] + d['y']
........ def fun2(d):
.... for i in xrange(0,N):
.... exec 'z = x + y' in globals(), d
........ # Note that locals() is *not* d after update() so
.... # z = x + y
.... # does not set z in d
.... def fun3(d):
.... exec "locals().update(d)"
.... for i in xrange(0,N):
.... d['z'] = x + y
........ # this makes code easier to write and read
.... class wrapdict(object):
.... """Lazy attribute access to dictionary keys. Will not access
.... keys that are not valid attribute names!"""
.... def __init__(self, mydict):
.... object.__setattr__(self, "mydict",mydict)
.... def __getattr__(self, attrname):
.... return self.mydict[attrname]
.... def __setattr__(self, attrname, value):
.... self.mydict[attrname] = value
........ def fun4(d):
.... wd = wrapdict(d)
.... for i in xrange(0,N):
.... wd.z = wd.x + wd.y
.... 3 function calls in 0.070 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.060 0.060 <string>:1(?)
1 0.010 0.010 0.070 0.070 profile:0(fun1(a))
0 0.000 0.000 profile:0(profiler)
1 0.060 0.060 0.060 0.060 python-4645vcY.py:2(fun1)

100003 function calls (3 primitive calls) in 5.890 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
100001/1 0.440 0.000 5.890 5.890 <string>:1(?)
1 0.000 0.000 5.890 5.890 profile:0(fun2(a))
0 0.000 0.000 profile:0(profiler)
1 5.450 5.450 5.890 5.890 python-46458me.py:2(fun2)

4 function calls (3 primitive calls) in 0.060 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
2/1 0.000 0.000 0.060 0.060 <string>:1(?)
1 0.000 0.000 0.060 0.060 profile:0(fun3(a))
0 0.000 0.000 profile:0(profiler)
1 0.060 0.060 0.060 0.060 python-4645Jxk.py:5(fun3)

300004 function calls in 3.910 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 3.910 3.910 <string>:1(?)
1 0.000 0.000 3.910 3.910 profile:0(fun4(a))
0 0.000 0.000 profile:0(profiler)
100000 0.530 0.000 0.530 0.000
python-4645W7q.py:10(__setattr__)
1 0.000 0.000 0.000 0.000 python-4645W7q.py:6(__init__)
200000 0.960 0.000 0.960 0.000
python-4645W7q.py:8(__getattr__)
1 2.420 2.420 3.910 3.910 python-4645jFx.py:1(fun4)

Exec is slow since compiling the string and calls to globals() use a lot
of time. The last one is most elegant but __getattr__ and __setattr__
are costly. The 'evil hack' solution is good since accessing x and y
takes no additional time.

I guess I will go with solution 3. It is evil but it is most close to my
original intention. It leads to most readable code (except for the first
line to do the magic and the last line to return result) and fastest
performance.

Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.

Bo
 
M

M.E.Farmer

I quote myself :
Also try this stuff out in an interpreter session it is easy and fast
to get your own answers.
Sorry I guess I should have added a nudge and a <.5 wink> at the end.
Sometimes misinformation is just what we need to get our own answers!
The absurdity of what you are doing led me to give you that piece of
advice.
It seems if I had just told you to RTF you would not have learned as
much.
I have been out so did not respond earlier, but I see others have given
you other useful information. There is no reason to make these local
epecially if they are large , leave them in the dictionary they come in
and acess the members as needed. I really don't see your need.
M.E.Farmer
 
B

Bo Peng

M.E.Farmer said:
I really don't see your need.

Maybe it is just my laziness. It is almost intolerable for me to write
lines and lines of code like

d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0] )

It is ugly, unreadable and error prone. If I have to use this code, I
would write

_z = func(_x + _y + _whatever['as'] + _a[0])

and use a perl script to generate the real code. (See, I am not lazy :)

Bo
 
B

Bo Peng

Maybe it is just my laziness. It is almost intolerable for me to write
lines and lines of code like

d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0] )

By the way, will 'with statement', like the one in pascal and many other
languages, be a good addition to python? For example,

with d do:
z = x + y

would be equivalent to d['z']=d['x']+d['y'] or d.z = d.x + d.y in some
other cases.

This would absolutely be the *best* solution to my problem.

Bo
 
M

M.E.Farmer

It is ugly, unreadable and error prone. If I have to use this code, I
would write
_z = func(_x + _y + _whatever['as'] + _a[0])
and use a perl script to generate the real code. (See, I am not lazy
:)
Ok laziness is an acceptable answer ;)
This is starting to make sense , you've been reading Perl's 'dense'
code and are not used to Python's explicitness.
*Explicit is better than implicit!*
It is a Zen of Python. Also I notice you say you will probably use the
'evil hack'.
Uhmmm, it is labeled evil hack for a reason! So go ahead and make life
hard and save a few keystrokes. There is a reason for the verbosity of
Python.
I really don't completly follow what you are trying to construct here
but it seems like you need to create an adapter class of some sort and
use that ( much like Jeff Shannon or Michael Spencer mentioned )
As a general rule class abstraction is your friend in Python.
It looks like ultimatly you are just using dictionary's and want to do
inplace operations on some of the data and store the results in other
parts of the dictionary... If that is the case then why not creat a few
helper functions that do exactly one thing each and use them as your
interface.

Py> def fun_helper(d):
.... d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0])
.... return d

and use them as needed.

M.E.Farmer
 
N

Nick Coghlan

Bo said:
By the way, will 'with statement', like the one in pascal and many other
languages, be a good addition to python? For example,

with d do:
z = x + y

would be equivalent to d['z']=d['x']+d['y'] or d.z = d.x + d.y in some
other cases.

This would absolutely be the *best* solution to my problem.

Bo

Guido van Rossum has stated that he wants Python to eventually have a 'with'
statement of the form:

with d:
.z = .x + .y

(The leading '.' being required to avoid ambiguity in name lookups)

Cheers,
Nick.
 
N

Nick Coghlan

Bo said:
I guess I will go with solution 3. It is evil but it is most close to my
original intention. It leads to most readable code (except for the first
line to do the magic and the last line to return result) and fastest
performance.

Thousands of programs use Python's class attribute access syntax day-in and
day-out and find the performance to be acceptable.

Premature optimization is the root of much evil - the exec + locals() hack if
very dependent on the exact Python version you are using. Expect some hard to
find bugs if you actually use the hack.

And code which relies on an evil hack in order to have the desired effect
doesn't count as readable in my book - I would be very surprised if many people
reading your code knew that updates to locals() can sometimes be made to work by
performing them inside a bare exec statement.
Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.

It sounds like the dictionaries you are working with actually have some
meaningful content that you want to manipulate.

I'd seriously suggest creating a class which is able to extract the relevant
data from the dictionaries, supports the desired manipulations, and then be
instructed to update the original dictionaries.

For example, an approach based on Michael's dictionary wrapper class

class DataAccessor(object):
# Just a different name for Michael's "wrapbigdict"
def __init__(self, data):
object.__setattr__(self, "_data", data)
def __getattr__(self, attrname):
return self._data[attrname]
def __setattr__(self, attrname, value):
self._data[attrname] = value

class DataManipulator(DataAccessor):
def __init__(self, data):
DataAccessor.__init__(self, data)
def calc_z(self):
self.z = self.x + self.y

In action:

Py> data = DataManipulator(dict(x=1, y=2))
Py> data.z
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 6, in __getattr__
KeyError: 'z'
Py> data.calc_z()
Py> data.z
3
Py> data._data
{'y': 2, 'x': 1, 'z': 3}

The class based approach also gives you access to properties, which can be used
to make that first call to 'z' automatically calculate the desired result
instead of giving a KeyError:

class EnhancedDataManipulator(DataAccessor):
def __init__(self, data):
DataAccessor.__init__(self, data)

class _utils(object):

@staticmethod
def make_prop(name, calculator):
def get(self, _name=name, _calculator=calculator):
try:
return self._data[_name]
except KeyError:
val = _calculator(self)
self._data[_name] = val
return val
def set(self, val, _name=name):
self._data[_name] = val
def delete(self, _name=name):
del self._data[_name]
return property(get, set, delete)

@staticmethod
def calc_z(self):
return self.x + self.y

z = _utils.make_prop('z', _utils.calc_z)

Note the nested _utils class isn't strictly necessary. I just like to use it to
separate out the information which the class uses to construct itself from those
which are provided to clients of the class.

Anyway, the enhanced version in action:

Py> data = EnhancedDataManipulator(dict(x=1,y=2))
Py> data.x
1
Py> data.y
2
Py> data.z
3
Py> data.x = 30
Py> data.z
3
Py> data.z = 10
Py> data.z
10
Py> del data.z
Py> data.z
32

If you want to add more calculated properties to the data manipulator, simply
define additional calculator methods, and define the attribute with make_prop.

Use a class. Giving meaningful form to a collection of attributes is what
they're for, and Python provides many features designed to make that process easier.

Trying to fake it with function namespaces is just going to make trouble for you
in the future.

Cheers,
Nick.
 
A

Alex Martelli

Bo Peng said:
M.E.Farmer said:
I really don't see your need.

Maybe it is just my laziness. It is almost intolerable for me to write
lines and lines of code like

d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0] )

It is ugly, unreadable and error prone. If I have to use this code, I
would write

_z = func(_x + _y + _whatever['as'] + _a[0])

So, what about having as the real code
d.z = func(d.x + d.y + d.whatever['as'] + d.a[0])
Doesn't seem too different from what you would write, would it?

To wrap a dictionary into an object whose attributes are mapped to items
of the dictionary is not hard: just start your function with

d = type('wrapadict', (), {})()
d.__dict__ = the_dict_to_wrap

and there you are.

Affecting *BARE* names is a completely different order of magnitude of
complication away from affecting *COMPOUND* names. Barenames are,
almost inevitably, handled with deep magic, essentially for the purposes
of can't-do-without optimization; compound names are _way_ more
malleable...


Alex
 
A

Alex Martelli

Bo Peng said:
Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.

Hmmm, you do realize that wrapdict uses a lot of indirection while my
equivalent approach, just posted, is very direct, right? To reiterate
the latter, and dress it up nicely too, it's

class wrapwell(object):
def __init__(self, somedict):
self.__dict__ = somedict

and use d=wrapwell(whateverdict) to make the wrapper. Now, reading or
writing d.x is just like reading or writing whateverdict['x'], etc etc.


Alex
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,776
Messages
2,569,602
Members
45,185
Latest member
GluceaReviews

Latest Threads

Top