class constructors: class vs. instance defaults

[Erik Johnson]

I am rather new to Python and still learning the finer points. I wanted
to have a class which has a member dictionary of other class instances, a
constructor that would initiate that dict with an empty dict if you didn't
provide one, and a member function to stick more other classes into that
dictionary.

I wrote something akin to this:

#! /usr/bin/python

#=======================================================================
class Bar:

def __init__(self, foo_d={}):
self.foo_d = foo_d

def add_foo(self, foo):
self.foo_d[foo.key] = foo
#=======================================================================


#=======================================================================
class Foo:
pass
#=======================================================================


You might be surprised to find out that all the Bar objects end up
sharing the same foo_d dictionary.
I certainly was.

from Test import *
b1 = Bar()
b2 = Bar()
dir(b1) ['__doc__', '__init__', '__module__', 'add_foo', 'foo_d']
b1.foo_d {}
b2.foo_d {}
f = Foo()
f.key = 'key'
b1.add_foo(f)
b1.foo_d

b2.foo_d

A single, shared dictionary is definitely not what I wanted or expected
and didn't see why it was happening. So... I struggled with that for a
while and eventually reasoned that the {} in the argument list to my
constructor is executed at the time of class definition, and is essentially
a reference to an instantiated dictionary object and therefor there is a
little empty dictionary sitting in memory somewhere prior to me ever
instantiating a Bar object. New Bar objects constructed without providing
their own foo_d end up sharing that one. I think this makes sense enough,
now (if I'm missing something, speak up).

So, one work-around is to do this:

#=======================================================================
class Bar:

def __init__(self, foo_d=None)
if foo_d:
self.foo_d = foo_d
else:
self.foo_d = {}
#=======================================================================


This works. I reasoned that, unlike above, the {} in this definition is
not executed until a Bar object is instantiated. But there is a little voice
in my head saying "This is clunky. There must be some smarter, more Pythonic
way to do this sort of default handling without ending up with a single
shared object" As I said, I am pretty new to Python and I am pretty pleased
with Python so far and expect to be using it more and more, (currently
porting and re-writing a lot of clunky PHP). I figured I might as well take
the time to learn this finer point now.

This example is trivial and it may seem silly to waste time considering
the 5 lines in my constructor, but this is just a pared down skeleton of the
problem I ran into and I am about to write lots of classes with lots of
members and want to call constructors with different numbers of arguments
and have different sorts of defaults (not necessarily shared, but maybe) for
the arguments not passed explicitly. And so, if my workaround above is not a
very clean one, I want to learn to do it the right way before I amplify it
1000X.

So... Yes? Is there a better way to do this sort of thing, or is that
perfectly reasonable code?

Thanks for taking the time to read my post!

-ej

 

[Michael Hoffman]

You might be surprised to find out that all the Bar objects end up
sharing the same foo_d dictionary.

I struggled with that for a
while and eventually reasoned that the {} in the argument list to my
constructor is executed at the time of class definition, and is essentially
a reference to an instantiated dictionary object and therefor there is a
little empty dictionary sitting in memory somewhere prior to me ever
instantiating a Bar object. New Bar objects constructed without providing
their own foo_d end up sharing that one. I think this makes sense enough,
now (if I'm missing something, speak up).

That's exactly right.

So, one work-around is to do this:

#=======================================================================
class Bar:

def __init__(self, foo_d=None)
if foo_d:
self.foo_d = foo_d
else:
self.foo_d = {}

To be honest that is almost exactly what I would do. Except that I would
use "if foo_d is not None" because the truth value of an empty container
(such as {}) is False. So if someone specified an empty dict subclass or
a list as the foo_d, you would end up creating a new empty dict instead
for self.foo_d.

Maybe that is a little far-fetched, but you seem to want to start with
good habits (which is very commendable), and using "if x is not None"
when you want to test whether it is None or not is a good idea.

> Is there a better way to do this sort of thing, or is that
perfectly reasonable code?

Looks reasonable enough to me.

 

[Dan Perl]

I am rather new to Python and still learning the finer points. I wanted
to have a class which has a member dictionary of other class instances, a
constructor that would initiate that dict with an empty dict if you didn't
provide one, and a member function to stick more other classes into that
dictionary.

I wrote something akin to this:

#! /usr/bin/python

#=======================================================================
class Bar:

def __init__(self, foo_d={}):
self.foo_d = foo_d

def add_foo(self, foo):
self.foo_d[foo.key] = foo
#=======================================================================

I think that a more "pythonic" way to implement your class is with a change
in the definition of your __init__:
def __init__(self, foo_d=dict({})):
This will create a separate copy for each one of your class instances
instead of the common object that you are creating in the definition of the
function.

Dan

 

[Michael Hoffman]

I think that a more "pythonic" way to implement your class is with a change
in the definition of your __init__:
def __init__(self, foo_d=dict({})):
This will create a separate copy for each one of your class instances
instead of the common object that you are creating in the definition of the
function.

>>> def x(d=dict({})): return d ....
>>> a = x()
>>> b = x()
>>> a {}
>>> b {}
>>> a[3] = 4
>>> b

{3: 4}

Taking out the {} doesn't help either.

 

[Steve Holden]

I am rather new to Python and still learning the finer points. I wanted
to have a class which has a member dictionary of other class instances, a
constructor that would initiate that dict with an empty dict if you didn't
provide one, and a member function to stick more other classes into that
dictionary.

I wrote something akin to this:

#! /usr/bin/python

#=======================================================================
class Bar:

def __init__(self, foo_d={}):
self.foo_d = foo_d

def add_foo(self, foo):
self.foo_d[foo.key] = foo
#=======================================================================
[...]


A single, shared dictionary is definitely not what I wanted or expected
and didn't see why it was happening. So... I struggled with that for a
while and eventually reasoned that the {} in the argument list to my
constructor is executed at the time of class definition, and is essentially
a reference to an instantiated dictionary object and therefor there is a
little empty dictionary sitting in memory somewhere prior to me ever
instantiating a Bar object. New Bar objects constructed without providing
their own foo_d end up sharing that one. I think this makes sense enough,
now (if I'm missing something, speak up).

So, one work-around is to do this:

#=======================================================================
class Bar:

def __init__(self, foo_d=None)
if foo_d:
self.foo_d = foo_d
else:
self.foo_d = {}
#=======================================================================

[...]

So... Yes? Is there a better way to do this sort of thing, or is that
perfectly reasonable code?

Thanks for taking the time to read my post!

-ej

Well you have managed to come up with the standard paradigm (and
successfully diagnose one of the most common beginner's conundrums along
the way) by the exercise of pure thought. I don't think there's much to
add, except possibly "congratulations"!

regards
Steve

 

[Carlos Ribeiro]

...
<snip>
...
You might be surprised to find out that all the Bar objects end up
sharing the same foo_d dictionary.

Yes, they will share. And you explanation is correct. The default
arguments are evaluated only once, when the def() is executed.

Now an explanation on what I mean when I say 'the def() is executed'.
It took me a while to really 'get' it, and I would not be surprised if
anyone ever made the same mistake.

-- When Python reads you code, it compiles it to bytecode. That's the
first step. It will always do it before execution.

-- It then executes your code -- *the entire code*. It includes class
definitions and function definitions.

-- In the case of class definitions, it really executes the class
statement body, but only once. After executing the class statement
body, it creates a class with all the symbols defined there. It's much
more flexible than a simple compilation step. When you instantiante a
object, it just reads this definition and calls the proper constructor
(__init__).

-- In the case of the def() -- or function definition -- it really
'executes the def() statement', which is not the same thing as to say
that it will execute its body. Understand? The def statement has an
effect -- it creates a function object, and associates it with the
code of the function. The default arguments are evaluated at this time
and also stored as part of the function object (you can do all types
of nice stuff with function objects, if that's something you like...
but I'll leave that for other opportunity).

-- Of course, the function body is executed only when you call it, by
referring to the function object.

That's it, in a nutshell.

class Bar:

def __init__(self, foo_d=None)
if foo_d:
self.foo_d = foo_d
else:
self.foo_d = {}

Well, it works. I don't see anything wrong with it. Youonly need to
take care with objects that are mutable. This includes lists and
dicts, but _not_ ints, floats or strings. These you can use as
defaults just fine, without fear.

.... but you can try this code also:

self.foo_d = foo_d or {}

It works, and it's short-circuiting -- the new dict will only be
constructed if foo_d is None.

This works. I reasoned that, unlike above, the {} in this definition is
not executed until a Bar object is instantiated. But there is a little voice
in my head saying "This is clunky. There must be some smarter, more Pythonic
way to do this sort of default handling without ending up with a single
shared object" As I said, I am pretty new to Python and I am pretty pleased
with Python so far and expect to be using it more and more, (currently
porting and re-writing a lot of clunky PHP). I figured I might as well take
the time to learn this finer point now.

This example is trivial and it may seem silly to waste time considering
the 5 lines in my constructor, but this is just a pared down skeleton of the
problem I ran into and I am about to write lots of classes with lots of
members and want to call constructors with different numbers of arguments
and have different sorts of defaults (not necessarily shared, but maybe) for
the arguments not passed explicitly. And so, if my workaround above is not a
very clean one, I want to learn to do it the right way before I amplify it
1000X.

So... Yes? Is there a better way to do this sort of thing, or is that
perfectly reasonable code?

Thanks for taking the time to read my post!

-ej

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Larry Bates]

I am rather new to Python and still learning the finer points. I wanted
to have a class which has a member dictionary of other class instances, a
constructor that would initiate that dict with an empty dict if you didn't
provide one, and a member function to stick more other classes into that
dictionary.

I wrote something akin to this:

#! /usr/bin/python

#=======================================================================
class Bar:

def __init__(self, foo_d={}):
self.foo_d = foo_d

def add_foo(self, foo):
self.foo_d[foo.key] = foo
#=======================================================================


#=======================================================================
class Foo:
pass
#=======================================================================


You might be surprised to find out that all the Bar objects end up
sharing the same foo_d dictionary.
I certainly was.



['__doc__', '__init__', '__module__', 'add_foo', 'foo_d']

{}


{}

{'key': <Test.Foo instance at 0x817821c>}


A single, shared dictionary is definitely not what I wanted or expected
and didn't see why it was happening. So... I struggled with that for a
while and eventually reasoned that the {} in the argument list to my
constructor is executed at the time of class definition, and is essentially
a reference to an instantiated dictionary object and therefor there is a
little empty dictionary sitting in memory somewhere prior to me ever
instantiating a Bar object. New Bar objects constructed without providing
their own foo_d end up sharing that one. I think this makes sense enough,
now (if I'm missing something, speak up).

So, one work-around is to do this:

#=======================================================================
class Bar:

def __init__(self, foo_d=None)
if foo_d:
self.foo_d = foo_d
else:
self.foo_d = {}
#=======================================================================


This works. I reasoned that, unlike above, the {} in this definition is
not executed until a Bar object is instantiated. But there is a little voice
in my head saying "This is clunky. There must be some smarter, more Pythonic
way to do this sort of default handling without ending up with a single
shared object" As I said, I am pretty new to Python and I am pretty pleased
with Python so far and expect to be using it more and more, (currently
porting and re-writing a lot of clunky PHP). I figured I might as well take
the time to learn this finer point now.

This example is trivial and it may seem silly to waste time considering
the 5 lines in my constructor, but this is just a pared down skeleton of the
problem I ran into and I am about to write lots of classes with lots of
members and want to call constructors with different numbers of arguments
and have different sorts of defaults (not necessarily shared, but maybe) for
the arguments not passed explicitly. And so, if my workaround above is not a
very clean one, I want to learn to do it the right way before I amplify it
1000X.

So... Yes? Is there a better way to do this sort of thing, or is that
perfectly reasonable code?

Thanks for taking the time to read my post!

-ej

I write this as:

#=======================================================================
class Bar:

def __init__(self, foo_d=None)
self.foo_d = foo_d or {}
#=======================================================================

but it is basically the same thing as others have suggested. Seems to
make it into a "one-liner" and I think the overhead is quite small to
evaluate the boolean 'or'.

Larry Bates

 

[Dan Perl]

Yes, that's right! I didn't try it in the context of a function definition
and thus I only checked that the dict({}) makes a different copy each time.
I realize now that what happens is that there is now still a 'd' object
(only in the context of the function) and it is shared by every instance.
The more I think about it, that's the way it's supposed to work, the default
value for the argument is calculated only once, when the function is
defined, no matter how that calculation is made.

Dan

I think that a more "pythonic" way to implement your class is with a
change in the definition of your __init__:
def __init__(self, foo_d=dict({})):
This will create a separate copy for each one of your class instances
instead of the common object that you are creating in the definition of
the function.

def x(d=dict({})): return d ...
a = x()
b = x()
a {}
b {}
a[3] = 4
b

{3: 4}

Taking out the {} doesn't help either.

 

[Erik Johnson]

I am rather new to Python and still learning the finer points....

Thanks to all of you who took the time to respond to my post! I do like the

foo_d or {}

paradigm - that's considerably cleaner.
Michael Hoffman's comments about explicitly testing against None duly noted!


And thanks to Mr. Holden for the words of encouragement - I have your
book by the way.
I am currently swamped with "stuff", but I browsed through enough of it to
know that Python was a viable option for us. I came into this job with PHP
already well underway. Though it has some smartly handy functions, I have
not been very impressed with it as a language overall. I am thankful I have
a cool boss who listens to what I think and so I think PHP, at this "shop"
anyway, is on its way out. I hope to spend some more time digging into
it (them - both the book and Python itself) soon!

Thanks again!
-ej

 

[Steve Holden]

Thanks to all of you who took the time to respond to my post! I do like the

foo_d or {}

paradigm - that's considerably cleaner.
Michael Hoffman's comments about explicitly testing against None duly noted!

Of course that means that

(foo_d is None) or {}

works, giving you the best of both worlds.

And thanks to Mr. Holden for the words of encouragement - I have your
book by the way.

Purchase is the sincerest form of flattery so I should thank you, too.

I am currently swamped with "stuff", but I browsed through enough of it to
know that Python was a viable option for us. I came into this job with PHP
already well underway. Though it has some smartly handy functions, I have
not been very impressed with it as a language overall. I am thankful I have
a cool boss who listens to what I think and so I think PHP, at this "shop"
anyway, is on its way out. I hope to spend some more time digging into
it (them - both the book and Python itself) soon!

Good for you! PHP was originally intended to make it easy to build
relatively simple programmed web sites, but now it has been pushed
beyond all reasonable limits to a perversion of its former self.

People have done many amazing things with it, but many of them would
have been easier in a "proper" programming language.

regards
Steve

 

[Steven Bethard]

Of course that means that

(foo_d is None) or {}

works, giving you the best of both worlds.

I don't think this does what the OP wants it to:
.... return (x is not None) or {}
....True

Note that (x is not None) evaluates to True, not x. Note that you can't solve
this by introducing an 'and' either because {} evaluates to False in a boolean
context:
.... return x is not None and x or {}
....False


(Another) Steve

 

[Lenard Lindstrom]

Yes, they will share. And you explanation is correct. The default
arguments are evaluated only once, when the def() is executed.

Now an explanation on what I mean when I say 'the def() is executed'.
It took me a while to really 'get' it, and I would not be surprised if
anyone ever made the same mistake.

-- When Python reads you code, it compiles it to bytecode. That's the
first step. It will always do it before execution.

-- It then executes your code -- *the entire code*. It includes class
definitions and function definitions.

-- In the case of class definitions, it really executes the class
statement body, but only once. After executing the class statement
body, it creates a class with all the symbols defined there. It's much
more flexible than a simple compilation step. When you instantiante a
object, it just reads this definition and calls the proper constructor
(__init__).

-- In the case of the def() -- or function definition -- it really
'executes the def() statement', which is not the same thing as to say
that it will execute its body. Understand? The def statement has an
effect -- it creates a function object, and associates it with the
code of the function. The default arguments are evaluated at this time
and also stored as part of the function object (you can do all types
of nice stuff with function objects, if that's something you like...
but I'll leave that for other opportunity).

-- Of course, the function body is executed only when you call it, by
referring to the function object.

That's it, in a nutshell.

This is essentially correct but leaves an opening for misunderstanding.
The assertion "-- It then executes your code -- *the entire code*." is
ambiguous. Only code reached during normal program flow is executed.
The python class and def statements are no exception. They are executed
each and every time they are encountered in the currently executing block
of code. If a definition is in a part of the code that is never reached
then that class or function will never be created.

Definitions as executable statements have their advantages. Closures
rely on an enclosed function's def being executed each time the enclosing
function is called. And a module can use an if statement to control which
version of a class or function is created when the module is loaded.

Lenard Lindstrom
<[email protected]>

 

[Erik Johnson]

I don't think this does what the OP wants it to:

Heh.... amazing how tricky things can get in only a few lines, eh?
Actually, I noticed that that wasn't really the same thing. Here's what one
of my constructors looked like from yesterday (essentially the form given by
exarkun):

#-------------------------------------------------------------------
def __init__(self, fields=None, tanks=None, pipes=None):
"Constructor."

WkSuper.__init__(self, fields)

if tanks is None: tanks = {}
self.tanks_d = tanks

if pipes is None: pipes = {}
self.pipes_d = pipes

#-------------------------------------------------------------------

This sort of a "two and a half liner" is not as beautiful as the
one-line 'or' statement (which I had put in there at first) but after
thinking about it for a while, I decided that it's not a totally
unreasonable thing to pass your own empty dictionary and then expect to have
your object use that and not it's own. I considered just ignoring that
issue and keeping in the back of my mind that if you passed an empty
dictionary it wasn't going to do what you probably expected, as exarkun
pointed out...

One could argue that there is something slightly wrong with both of these. Consider:

d = {}
b = Bar(d)
d['x'] = 'y'

One might reasonably expect b.foo_d['x'] to be 'y'. In the case of the

above, it will not be; a KeyError will be raised instead.


I eventually decided that ignoring the issue was not in the spirit of
that which I originally launched this post. These are all interesting issues
for a beginner, and I think I have become sharper at Python for having gone
through this. After thinking about this even more carefully today, a couple
of things come to mind:

1) Might it make sense for Python to have both a mutable and an
immutable version of a dictionary?

2) In terms of my own design and usage, I think the idea is that I'm
trying to make it convenient to pre-populate an object from a dictionary a
caller already has handy (which is probably me). I think I should probably
be doing a dictionary copy, not assignment. The passed dictionary is an
OBJECT INITIALZER. That is, once you've created the object, that's the place
to keep and manage your data - either through the object's exposed interface
(methods), or by directly setting member data fields. If you later muck
about with the dictionary from which the object was created, it probably
should NOT affect the object. (Granted, Python allows you to arrange to have
it work that way, which is cool, and other authors may have good reason to
do it that way, I think that is NOT the paradigm I want to encourage.)

So, these are rather subtle issues that were not crystal clear to me
yesterday morning. I thank you all for the helpful and enlightening
discussion. I feel much better than when I thought my interpreter was broken
because all my objects had the same dictionary!

-ej

 

[Michael Hoffman]

Note that (x is not None) evaluates to True, not x. Note that you can't solve
this by introducing an 'and' either because {} evaluates to False in a boolean
context:

Right. However you can do [x, {}][x is None].

But please don't! <double wink>

 

[bruno modulix]

I am rather new to Python and still learning the finer points. I wanted
to have a class which has a member dictionary of other class instances, a
constructor that would initiate that dict with an empty dict if you didn't
provide one, and a member function to stick more other classes into that
dictionary.

I wrote something akin to this:

#! /usr/bin/python

#=======================================================================
class Bar:

def __init__(self, foo_d={}):
self.foo_d = foo_d

Woops ! What this does is probably not what you'd think, and I guess
that you will complain that all instances of Bar share the same dict...

def add_foo(self, foo):
self.foo_d[foo.key] = foo
#=======================================================================


#=======================================================================
class Foo:
pass
#=======================================================================


You might be surprised to find out that all the Bar objects end up
sharing the same foo_d dictionary.

Bingo !-)

I certainly was.

I'm not. But I certainly was first time I stumbled upon this !-)
This is a FAQ. The correct idiom is :

class Bar:
def __init__(self, foo_d=None):
if foo_d is None:
self.foo_d = {}
else :
self.foo_d = foo_d

A single, shared dictionary is definitely not what I wanted or expected
and didn't see why it was happening. So... I struggled with that for a
while and eventually reasoned that the {} in the argument list to my
constructor is executed at the time of class definition,
Exactly.

and is essentially
a reference to an instantiated dictionary object and therefor there is a
little empty dictionary sitting in memory somewhere prior to me ever
instantiating a Bar object. New Bar objects constructed without providing
their own foo_d end up sharing that one. I think this makes sense enough,
now (if I'm missing something, speak up).

So, one work-around is to do this:

#=======================================================================
class Bar:

def __init__(self, foo_d=None)
if foo_d:
self.foo_d = foo_d
else:
self.foo_d = {}

You've almost got it. You just want to test against None, not again
False, since an empty dict evaluate to False in a boolean test.

Python 2.3.3 (#2, Feb 17 2004, 11:45:40)
[GCC 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information..... print "d"
.... else:
.... print "woops"
....
woops

False

This works.

No. There's still a bug, but more subtle. Guess what will happen if you
try something like :

>>> b1 = Bar()
>>> b2 = Bar(b1.foo_d)
>>> b1.foo_d['foo1'] = Foo()
>>> b2.foo_d['foo1']

I reasoned that, unlike above, the {} in this definition is
not executed until a Bar object is instantiated. But there is a little voice
in my head saying "This is clunky. There must be some smarter, more Pythonic
way to do this sort of default handling without ending up with a single
shared object"
(snip)

So... Yes? Is there a better way to do this sort of thing, or is that
perfectly reasonable code?

For mutable default args, testing against None is the default pythonic
way. Now there are a lot of tricks when it comes to dynamically doing
thing with Python, and you will probably discover'em sooner or later.

The python cookbook at o'reilly.com may be a good starting point, even
if some recipes are somewhat out of date.

Thanks for taking the time to read my post!

You're welcome.