Counting number of objects

[Kottiyath]

Hi,
I am creating a class called people - subclasses men, women, children
etc.
I want to count the number of people at any time.
So, I created code like the following:

class a(object):
counter = 0
def __new__(cls, *args, **kwargs):
a.counter += 1
return object.__new__(cls, *args, **kwargs)

def __del__(self):
a.counter -= 1

class aa(a):
pass

Now, the code works Ok. I have the following questions:
1. Is this code Ok? Is there any straightforward mechanism other than
this to get the number of objects?
2. I read in Python Documentation that inside __del__ we should the
minimum of interaction with external parameters. So, I am a little
worried in subclassing __del__ to check the counter. Is whatever I
have done Ok?

Another question - unrelated to the major topic:
How much time does it take to be proficient in Python? I have been
working exclusively in Python for close to 3 months now, and even now
I get inferiority complex when I read the answers sent by many of you.
I have been programming for close to 7 years now (earlier in a
language similar to COBOL).
Does it take quite a bit of time to be proficient - as many of you
guys - or am I just dumb?

 

[Steve Holden]

Hi,
I am creating a class called people - subclasses men, women, children
etc.
I want to count the number of people at any time.
So, I created code like the following:

class a(object):
counter = 0
def __new__(cls, *args, **kwargs):
a.counter += 1
return object.__new__(cls, *args, **kwargs)

def __del__(self):
a.counter -= 1

class aa(a):
pass

Now, the code works Ok. I have the following questions:
1. Is this code Ok? Is there any straightforward mechanism other than
this to get the number of objects?
2. I read in Python Documentation that inside __del__ we should the
minimum of interaction with external parameters. So, I am a little
worried in subclassing __del__ to check the counter. Is whatever I
have done Ok?

Yes. Just be aware that if instances become involved in cyclic data
structures (or in implementations other than CPython, where reference
counting isn't used) __del__ might not be called until garbage
collection kicks in, so you may want a more explicit way to stop an
instance from being in the count.

Another question - unrelated to the major topic:
How much time does it take to be proficient in Python? I have been
working exclusively in Python for close to 3 months now, and even now
I get inferiority complex when I read the answers sent by many of you.
I have been programming for close to 7 years now (earlier in a
language similar to COBOL).
Does it take quite a bit of time to be proficient - as many of you
guys - or am I just dumb?

By your code above you seem to be doing OK. Python is like an iceberg -
only an eighth of what goes on is above the surface. That eighth will
suffice for many people's total programming needs.

I've been using Python ten years, and I am still learning. Just go at
your own pace, and carry on asking for help when you need it.

regards
Steve

 

[Andreas Waldenburger]

Hi,
I am creating a class called people - subclasses men, women, children
etc.
I want to count the number of people at any time.
So, I created code like the following:

class a(object):
counter = 0
def __new__(cls, *args, **kwargs):
a.counter += 1
return object.__new__(cls, *args, **kwargs)

def __del__(self):
a.counter -= 1

class aa(a):
pass

This looks OK, although I'd suggest using "cls.counter += 1" instead of
"a.counter += 1" in the __new__() method. Just seems clearer to me,
esp. when you think about subclassing. This would create an asymmetry
with __del__() then. Oh well. So maybe use "self.__class__.counter -= 1"
there, even if it is a bit ugly-ish.

Another way to go would be to use the weakref module and create a
weakref-set (or list) as the counter. That way you would
only need to add the objects in the __new__() method and not worry
about removing them. I will admit that this is overengineering the
problem a bit, but might be a good exercise.

A third option could be to remove the counting functions from the class
altogether. From an OO-Design point of view this would seem
appropriate, because neither any individual nor mankind as a whole
would know the exact amount of people in the world of hand. An entity
that would actually count all the people in the world, however, would
know and it makes sense to implement it separately (as a subclass of
list with birth() and death() methods, for instance). I'm just saying
this to give you something to think about, I'm not saying that it is
necessarily better or worse for your example.

Now, the code works Ok. I have the following questions:
1. Is this code Ok? Is there any straightforward mechanism other than
this to get the number of objects?

I would say that you found the most obvious implementation there.
Depending on your definition of "straightforward" you could do it as I
outlined in my last example above, using explicit calls for births and
deaths. This would remove the "behind the scenes" magic a bit, which
may be a plus.

2. I read in Python Documentation that inside __del__ we should the
minimum of interaction with external parameters. So, I am a little
worried in subclassing __del__ to check the counter. Is whatever I
have done Ok?

Again, seems good to me, unless you do some other trickery that may
lead to __del__() not being called directly. In that case, do look at
the weakref module.

Another question - unrelated to the major topic:
How much time does it take to be proficient in Python?

Don't concern yourself with that question at all, is my advice.

Most people can learn to write programs in under a week, and many still
don't write *good* programs 20 years later. You'll get better over
time, as long as you keep doing it. Thinking about that you're not good
enough will just consume mental resources that would be better invested
in your programs.

I have been
working exclusively in Python for close to 3 months now, and even now
I get inferiority complex when I read the answers sent by many of you.

Hello?!

Three months is *nothing* compared to the time those "many of you"
folks have invested. Don't worry. Does it matter if you don't
understand some stuff people write? As long as you pick out the stuff
you do understand you're still gaining stuff from this group.

I have been programming for close to 7 years now (earlier in a
language similar to COBOL).

Well, you have quite a background then. Why all the worries?

Does it take quite a bit of time to be proficient - as many of you
guys

YES! Of course it does.

- or am I just dumb?

You're writing programs and you're communicating with like-minded
people about your problems (in a socially appropriate way). Not what
dumb people do, in my book.


cheers,
/W

 

[Gabriel Genellina]

En Sun, 25 Jan 2009 16:06:47 -0200, Andreas Waldenburger

This looks OK, although I'd suggest using "cls.counter += 1" instead of
"a.counter += 1" in the __new__() method. Just seems clearer to me,
esp. when you think about subclassing. This would create an asymmetry
with __del__() then. Oh well. So maybe use "self.__class__.counter -= 1"
there, even if it is a bit ugly-ish.

Using self.__class__ is safer, from a technical point of view. When
__del__ is executed at interpreter shutdown, "a" may not be available --
in general, __del__ methods should not rely on any globals (one can
"inject" names into the local namespace using default arguments).
See http://bugs.python.org/issue1717900

 

[Gabriel Genellina]

En Sun, 25 Jan 2009 16:06:47 -0200, Andreas Waldenburger

This looks OK, although I'd suggest using "cls.counter += 1" instead of
"a.counter += 1" in the __new__() method. Just seems clearer to me,
esp. when you think about subclassing. This would create an asymmetry
with __del__() then. Oh well. So maybe use "self.__class__.counter -= 1"
there, even if it is a bit ugly-ish.

Using self.__class__ is safer, from a technical point of view. When
__del__ is executed at interpreter shutdown, "a" may not be available --
in general, __del__ methods should not rely on any globals (one can
"inject" names into the local namespace using default arguments).
See http://bugs.python.org/issue1717900

 

[Mark Wooding]

Hmm. Exceptions raised during object creation make this rather
precarious. In your code, if object.__new__ raises an exception, the
counter will end up too high (the __del__ method won't get called in
this case).

One might try to rewrite it:

def __new__(cls, *args, **kw):
thing = object.__new__(cls, *args, **kw)
a.counter += 1
return thing

Now this won't work in subclasses if they also have a __new__ method:
again, if the subclass's __new__ raises an exception then __del__ won't
be called and the counter will be too high.

To make this technique work, I think you need to do the counter
increment in __init__ rather than __new__, and to set an attribute so
that __del__ knows whether to do the decrement. (If a subclass's
__init__ raises an exception before yours gets called, you don't want to
do the decrement because that'll leave the counter too low.)

This looks OK, although I'd suggest using "cls.counter += 1" instead
of "a.counter += 1" in the __new__() method. Just seems clearer to me,
esp. when you think about subclassing.

I'm not sure about clarity, but that would be semantically different.
The code as written counts all instances of a and its subclasses. Your
suggestion would count instances of subclasses independently. I don't
know which behaviour the OP would prefer, but I don't think choosing
between them is a matter of clarity.

Another way to go would be to use the weakref module and create a
weakref-set (or list) as the counter. That way you would only need to
add the objects in the __new__() method and not worry about removing
them. I will admit that this is overengineering the problem a bit, but
might be a good exercise.

This is a better approach, because it avoids the problems with
exceptions during object construction that I described above.

Don't concern yourself with that question at all, is my advice.

Indeed. Besides, it varies an awful lot.

I can't tell you from my personal experience, because I tend to learn
programming languages by osmosis. I sit and read language manuals and
newsgroups out of curiosity, because I find programming languages
intrinsically interesting. One day, maybe a few years later, I start
writing programs in some language. I'm usually up to half-decent
language-lawyer standards within a few days of this point -- but I've
had a /lot/ of practice at this game. My approach is almost certainly
extremely atypical.

But one of the best things about Python is the library. It's very big
and wide-ranging, but not overcomplex, and I'm still discovering cool
stuff, even though it's been there for years. And the advice to keep
the Library Reference under your pillow is excellent. Expect to be
flipping through it constantly.

You're writing programs and you're communicating with like-minded
people about your problems (in a socially appropriate way). Not what
dumb people do, in my book.

Absolutely! Besides, you're asking sensible questions about subtle
parts of the language -- I wouldn't say that __new__ was beginner
territory, for example. So, no, you certainly don't seem dumb to me.

-- [mdw]

 

[Andreas Waldenburger]

I'm not sure about clarity, but that would be semantically different.
The code as written counts all instances of a and its subclasses.
Your suggestion would count instances of subclasses independently. I
don't know which behaviour the OP would prefer, but I don't think
choosing between them is a matter of clarity.

Oh shoot. I should have known that. I hope I will from now on, at
least. :/

/W

 

[Steve Holden]

Hmm. Exceptions raised during object creation make this rather
precarious. In your code, if object.__new__ raises an exception, the
counter will end up too high (the __del__ method won't get called in
this case).

If object.__new__ raises an exception you have bigger problems to worry
about than an object count being wrong.

One might try to rewrite it:

def __new__(cls, *args, **kw):
thing = object.__new__(cls, *args, **kw)
a.counter += 1
return thing

Now this won't work in subclasses if they also have a __new__ method:
again, if the subclass's __new__ raises an exception then __del__ won't
be called and the counter will be too high.

To make this technique work, I think you need to do the counter
increment in __init__ rather than __new__, and to set an attribute so
that __del__ knows whether to do the decrement. (If a subclass's
__init__ raises an exception before yours gets called, you don't want to
do the decrement because that'll leave the counter too low.)

Yes, rather better to do it that way and decouple it from __new__(). Of
course super() might help here, though not everyone approves of it.

I'm not sure about clarity, but that would be semantically different.
The code as written counts all instances of a and its subclasses. Your
suggestion would count instances of subclasses independently. I don't
know which behaviour the OP would prefer, but I don't think choosing
between them is a matter of clarity.

Correct, but pointing out the differences has highlighted that is a real
design decision to be made in the event that subclassing is used.

Another way to go would be to use the weakref module and create a
weakref-set (or list) as the counter. That way you would only need to
add the objects in the __new__() method and not worry about removing
them. I will admit that this is overengineering the problem a bit, but
might be a good exercise.

This is a better approach, because it avoids the problems with
exceptions during object construction that I described above.

Don't concern yourself with that question at all, is my advice.

Indeed. Besides, it varies an awful lot.
[...]

You're writing programs and you're communicating with like-minded
people about your problems (in a socially appropriate way). Not what
dumb people do, in my book.

Absolutely! Besides, you're asking sensible questions about subtle
parts of the language -- I wouldn't say that __new__ was beginner
territory, for example. So, no, you certainly don't seem dumb to me.

Hear, hear!

regards
Steve

 

[Kottiyath]

Thank you everyone for your very helpful comments and suggestions. I
have interacted in other newsgroups, but this is the most helpful of
them all.

As per the comments, I have now decided to go with the weakref
mechanism - as per Andreas suggestion, functionally it looks correct
that the person should not know the total number of people.
So, in a higher level class, have a weakref list which contains a
reference to each person. Total count will be len(list) at any time.

Now, I couldnt find a weakref list - so I am using WeakKeyDictionary
with the value as None - since len(dict) also should give me the data
any time.

I have another question here. In the documentation, it is mentioned
that -
Note: Caution: Because a WeakKeyDictionary is built on top of a Python
dictionary, it must not change size when iterating over it. This can
be difficult to ensure for a WeakKeyDictionary because actions
performed by the program during iteration may cause items in the
dictionary to vanish "by magic" (as a side effect of garbage
collection).

Now, the only two operations that I am doing are ->
__init__:
d = weakref.WeakKeyDictionary()

method y:
x = aa()
d[x] = None

method z:
total = len(d)

I believe that all the commands which I perform on WeakKeyDictionary
here - (adding a new element) & (len(d)) - are atomic - or atleast
nothing that can cause any worry as per the Note given above. Can
anyone let me know whether my assumption is correct or not?

Reason: My code has many many number of threads which interact with
each other in myraid ways - so I do want to decrease the number of
locks as much as possible. Especially I do not want to block other
threads just for getting the count.