How to write Smart Python programs?

[Raj]

Hi,

We just executed a project with Python using TG. The feedback was to
use more python like programming rather than C style code executed in
Python. The feedback is from a Python purist and for some reasons we
cannot solicity his help.

So we'd like to do is to scrub through the codebase and identify places
where the codebase needs improvement, both from styling as well as
design. Is there any website that can provide me with advanced tips
rather than just tutorials coz thats not of much help.

The project involves:
1. A server module that receives messages from a datalogger to populate
a database
2. A DB access module that interfaces with the UI

The technologies we use are TurboGears, SQLAlchemy, CherryPy, Kid
templates.

Any help will be greatly appreciated. If you would like to get involved
in it for $$ (experienced in the tools mentioned above) as well do let
me know. I need help asap as the deadlines are very very short.

Thanks,
Raj.

 

[Bruno Desthuilliers]

Hi,

We just executed a project with Python using TG. The feedback was to
use more python like programming rather than C style code executed in
Python. The feedback is from a Python purist and for some reasons we
cannot solicity his help.

So we'd like to do is to scrub through the codebase and identify places
where the codebase needs improvement, both from styling as well as
design. Is there any website that can provide me with advanced tips
rather than just tutorials coz thats not of much help.

Googling for "python is not java" may be a good start. Also, here are 2
common C-style smells:

1/ using explicit indexing instead of iteration:

C-smell :
for i in range(len(seq)):
do_something_with(seq)

Pythonic:
for item in seq:

do_something_with(item)
# or if you need the index too:
for i, item in enumerate(seq):
do_something_with(i, item)

2/ functions that returns a status code and modify their arguments. The
pythonic way is to have the function return multiple args and raise an
exception if something went wrong

 

[Theerasak Photha]

2/ functions that returns a status code and modify their arguments.

Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

-- Theerasak

 

[Fredrik Lundh]

Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

sounds like they (or you) are confusing "arguments" with "free
variables", though. here's an example of the latter:

def outer():
var = [value]
def inner():
var[0] = new value
inner()

here's an example of the former:

def func(var):
var[0] = new value
return something else

myvar = [value]
res = func(myvar)

the second example feels rather contrived; if you need to return
multiple values, just return them.

</F>

 

[Theerasak Photha]

sounds like they (or you) are confusing "arguments" with "free
variables", though. here's an example of the latter:

This is (kind of) what we were talking about:

http://www.c2.com/cgi/wiki?ClosuresThatWorkAroundFinalLimitation

-- Theerasak

 

[Antoine De Groote]

Googling for "python is not java" may be a good start. Also, here are 2

common C-style smells:

Ok, the first Google result
(http://dirtsimple.org/2004/12/python-is-not-java.html) says this somewhere:

"Getters and setters are evil. Evil, evil, I say! Python objects are not
Java beans. Do not write getters and setters. This is what the
'property' built-in is for. And do not take that to mean that you should
write getters and setters, and then wrap them in 'property'. (1) That
means that until you prove that you need anything more than a simple
attribute access, don't write getters and setters. They are a waste of
CPU time, but more important, they are a waste of programmer time. Not
just for the people writing the code and tests, but for the people who
have to read and understand them as well.

In Java, you have to use getters and setters because using public fields
gives you no opportunity to go back and change your mind later to using
getters and setters. (2) So in Java, you might as well get the chore out
of the way up front. In Python, this is silly, because you can start
with a normal attribute and change your mind at any time, without
affecting any clients of the class. (3) So, don't write getters and
setters."

For the record, I'm coming from Java, but I have some Python experience
already.

Now here's what I don't understand.

What exactly is meant by (1)? The python library reference includes this
as an example:

class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")

To me this seems contradictory. Why would one not want to do something
that is included in the documentation? Or am I just confused? Does
anybody have an idea how to put me in the right direction?

And what does property mean anyway? The explanation in
http://docs.python.org/lib/built-in-funcs.html is not enough for me. Can
anybody point me to a more detailed documentation about this matter? (At
first sight, the Python tutorial doesn't seem to describe this. At least
it is not stated in the TOC, nor in chapter 9 which describes classes...)

And then, by (2) I believe that what is meant is that once some client
code uses public fields one can't make them private anymore because then
the clients are broke. That's clear to me. But I don't understand (3). I
don't know exactly what they mean. How would one want to change their
mind? In what way? I've been thinking about it some time now, but I'm
still kept in the dark. I would appreciate any explanation.

I'd really like to get this, because when I use a language I want to use
it "correctly", i.e. in the way it is intended, but right now I feel a
bit lost :-(

I'm not done with 'How To Ask Questions The Smart Way' yet, so please
forgive any violations. (http://catb.org/~esr/faqs/smart-questions.html)

Regards,
antoine

 

[Paul Rubin]

To me this seems contradictory. Why would one not want to do something
that is included in the documentation? Or am I just confused? Does
anybody have an idea how to put me in the right direction?

The library reference states the technical procedures for doing
various things. That is, it explains how the procedures work. It
doesn't say when to use one procedure or another. It's a technical
reference, not a style guide.

And then, by (2) I believe that what is meant is that once some client
code uses public fields one can't make them private anymore because
then the clients are broke. That's clear to me. But I don't understand
(3). I don't know exactly what they mean. How would one want to change
their mind? In what way? I've been thinking about it some time now,
but I'm still kept in the dark. I would appreciate any explanation.

I think it means the example you cited from the library doc is
inadvisable in practice, and should only be used as a template for
more complicated getters and setters. E.g. if you want the client
to be able to set foo.x to an arbitrary value, don't use a setter,
just say

foo.x = 237

and the attribute value gets set directly. If you want to limit the
value of foo.x to the range 0-100, and make setting to outside the
range clip to that range or raise an exception or something, THEN use
a setter. In Python, you can omit the setter at first (so there's no
restriction), then later decide you want to add the restriction, so
you add the setter. In Java, if you omit the setter at first and then
later want to add it, you have to change all the client calls.

 

[Ben Finney]

Ok, the first Google result
(http://dirtsimple.org/2004/12/python-is-not-java.html) says this somewhere:

"Getters and setters are evil. Evil, evil, I say! Python objects are
not Java beans. Do not write getters and setters. This is what the
'property' built-in is for. And do not take that to mean that you
should write getters and setters, and then wrap them in
'property'. (1) That means that until you prove that you need
anything more than a simple attribute access, don't write getters
and setters. They are a waste of CPU time, but more important, they
are a waste of programmer time. Not just for the people writing the
code and tests, but for the people who have to read and understand
them as well. [...]"

What exactly is meant by (1)? The python library reference includes
this as an example:

class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")

To me this seems contradictory. Why would one not want to do
something that is included in the documentation? Or am I just
confused? Does anybody have an idea how to put me in the right
direction?

Note that, to a user of that class, the fact that it's implemented as
a property is opaque. Thus, changing an attribute from a simple value
object, to a property, can be done as and when needed, without
changing any of the code that uses that class.

So, it's best to always code plain public attributes without getters
and setters in the first instance:
... def __init__(self):
... self.menu = "spam"
...
'sausage'

Because later, when one wants to refactor the implementation of that
attribute without changing the interface, only the implementation
needs to change:
... def __init__(self):
... self._menu = ["spam", "eggs", "beans"]
... self.current_item = 0
... def _get_menu(self):
... return self._menu[self.current_item]
... def _set_menu(self, value):
... self._menu[self.current_item] = value
... menu = property(_get_menu, _set_menu)
...

And the code that uses that attribute can remain the same:
'sausage'

Thus, coding getters and setters without a demonstrated need to do so
in a particular case is a waste of effort and an increase in
complexity, with no benefit.

I'm not done with 'How To Ask Questions The Smart Way' yet, so
please forgive any violations.

Your awareness of that document, and your attempt to follow it
sincerely, is much more important.

 

[Bruno Desthuilliers]

Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

Yes, but that's another point. What I was talking about (perhaps not
clearly) is mutable arguments used as returned values, ie:

C-smell:

def my_func(res):
bar = foo()
if bar is None:
return 1 # error
else:
res['bar'] = bar
res['baaz'] = "blah"
return 0 # no error

pythonic:

def my_func():
bar = foo()
if bar is None:
raise FooBarError("no bar from foo")
return dict(bar=bar, baaz='blah')

 

[Bruno Desthuilliers]

Ok, the first Google result
(http://dirtsimple.org/2004/12/python-is-not-java.html) says this
somewhere:

"Getters and setters are evil. Evil, evil, I say! Python objects are not
Java beans. Do not write getters and setters. This is what the
'property' built-in is for. And do not take that to mean that you should
write getters and setters, and then wrap them in 'property'. (1) That
means that until you prove that you need anything more than a simple
attribute access, don't write getters and setters. They are a waste of
CPU time, but more important, they are a waste of programmer time. Not
just for the people writing the code and tests, but for the people who
have to read and understand them as well.

In Java, you have to use getters and setters because using public fields
gives you no opportunity to go back and change your mind later to using
getters and setters. (2) So in Java, you might as well get the chore out
of the way up front. In Python, this is silly, because you can start
with a normal attribute and change your mind at any time, without
affecting any clients of the class. (3) So, don't write getters and
setters."

For the record, I'm coming from Java, but I have some Python experience
already.

Now here's what I don't understand.

What exactly is meant by (1)? The python library reference includes this
as an example:

class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")

To me this seems contradictory. Why would one not want to do something
that is included in the documentation? Or am I just confused? Does
anybody have an idea how to put me in the right direction?

Have you tried running the above snippet in a python shell ?

42


And what does property mean anyway?

See above. A property is a computed attribute : you access it like a
'data' attribute, but it really uses getters/setters. The point here is
that client code doesn't know nor need to know if it's a plain attribute
or a computed one. This let you change implementation (turn a plain
attribute into a computed one) without breaking the API. So the real
thing is not "don't use getters and setters", but "don't bother with
getters and setters, you'll be able to add them transparently if and
when needed".

As a side node, the getters/setters of a property are often either
marked as implementation (the _leading_underscore convention) or deleted
from the class's dict once the property is created or 'masked' one way
or another so they don't pollute the class's namespace.

Also and FWIW, properties are just one possible application of the
descriptor protocol. If you want to grasp Python's object model, I'd
recommand that you read about descriptors and metaclasses.

 

[Antoine De Groote]

See above. A property is a computed attribute : you access it like a
'data' attribute, but it really uses getters/setters. The point here is
that client code doesn't know nor need to know if it's a plain attribute
or a computed one. This let you change implementation (turn a plain
attribute into a computed one) without breaking the API. So the real
thing is not "don't use getters and setters", but "don't bother with
getters and setters, you'll be able to add them transparently if and
when needed".

Ok great, now I understand. It's nice actually

For anyone else trying this for the first time, the following might be
interesting, as I had trouble with this at first:
Don't forget to derive your class from object, otherwise properties
won't work.

As a side node, the getters/setters of a property are often either
marked as implementation (the _leading_underscore convention) or deleted
from the class's dict once the property is created or 'masked' one way
or another so they don't pollute the class's namespace.

You mean like manually removed from the dict? Because (at least on the
system I tried) it doesn't happen automatically. (Windwos XP, Python 2.5)

 

[Fredrik Lundh]

Don't forget to derive your class from object, otherwise properties
won't work.

getters work, setters don't (because the old-style objects checks the instance
before it looks for hooks in the class):
.... def getx(self):
.... return "x"
.... def setx(self, value):
.... print "set to", value
.... x = property(getx, setx)
....

'y'


You mean like manually removed from the dict?

explicitly deleted:
.... def getx(self):
.... return "x"
.... def setx(self, value):
.... print "set to", value
.... x = property(getx, setx)
.... del getx, setx
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Spam' object has no attribute 'getx'

(note that the class definition is executed in a class-specific namespace; whatever's
left in that namespace when you reach the end of the class statement is added to the
class).

</F>

 

[Roy Smith]

In Java, you have to use getters and setters because using public fields
gives you no opportunity to go back and change your mind later to using
getters and setters.

This is a good example of an anti-pattern. The general thought process
here is:

1) "I need the ability to control access to an object's internal state".
This is fine. Decoupling of interface from implementation is, in general,
a good thing, and one of the guiding principles of OOD.

2) "In the language I'm using, the only way to do that is to write getter
and setter functions". Fair enough. You gotta use the tools you have
available to you.

3) "Getters and setters are an essential part of OOD". Here's where things
go wrong. Just like one must learn to decouple interface from
implementation in your programs, so must one learn to decouple them in your
program design. Getters and setters are an language implementation detail.
There's nothing magic about adding the word "get" to the beginning of a
variable name and putting "()" after it. That's just how you spell
"property" in C++ or java.

It used to be said, "You can write Fortran in any language". The
implication there was that a literal line-by-line translation
(transliteration?) of a Fortran program into some other language usually
resulted in a mess. I think we can generalize that to "You can write any
language in any other language". Every language has a natural way of doing
things. When you try to bring the old idioms and design processes from one
language over to another, it usually doesn't work.

Once, years ago, I was working in a lisp environment for maybe a year. It
took me a little while to learn the syntax and basic function set, but I
never really "got" it. I was writing C programs in lisp. The same thing
happens when I try to speak Spanish. Even when I used Spanish a fair bit,
I never really thought in Spanish. I thought in English and translated.
That doesn't work very well.

 

[John Salerno]

Googling for "python is not java" may be a good start.

I have a question about this section of a blog with that title. I'll ask
the question first so it doesn't get lost at the bottom: does the
following opinion of XML apply to GUIs? (i.e., using an XML resource to
define a GUI layout rather than coding it yourself).

Thanks.

----------------------------------------------

XML is not the answer. It is not even the question. To paraphrase Jamie
Zawinski on regular expressions, "Some people, when confronted with a
problem, think "I know, I'll use XML." Now they have two problems."

This is a different situation than in Java, because compared to Java
code, XML is agile and flexible. Compared to Python code, XML is a boat
anchor, a ball and chain. In Python, XML is something you use for
interoperability, not your core functionality, because you simply don't
need it for that. In Java, XML can be your savior because it lets you
implement domain-specific languages and increase the flexibility of your
application "without coding". In Java, avoiding coding is an advantage
because coding means recompiling. But in Python, more often than not,
code is easier to write than XML. And Python can process code much, much
faster than your code can process XML. (Not only that, but you have to
write the XML processing code, whereas Python itself is already written
for you.)

If you are a Java programmer, do not trust your instincts regarding
whether you should use XML as part of your core application in Python.
If you're not implementing an existing XML standard for interoperability
reasons, creating some kind of import/export format, or creating some
kind of XML editor or processing tool, then Just Don't Do It. At all.
Ever. Not even just this once. Don't even think about it. Drop that
schema and put your hands in the air, now! If your application or
platform will be used by Python developers, they will only thank you for
not adding the burden of using XML to their workload.

(The only exception to this is if your target audience really really
needs XML for some strange reason. Like, they refuse to learn Python and
will only pay you if you use XML, or if you plan to give them a nice GUI
for editing the XML, and the GUI in question is something that somebody
else wrote for editing XML and you get to use it for free. There are
also other, very rare, architectural reasons to need XML. Trust me, they
don't apply to your app. If in doubt, explain your use case for XML to
an experienced Python developer. Or, if you have a thick skin and don't
mind being laughed at, try explaining to a Lisp programmer why your
application needs XML!)

 

[Bruno Desthuilliers]

Ok great, now I understand. It's nice actually

For anyone else trying this for the first time, the following might be
interesting, as I had trouble with this at first:
Don't forget to derive your class from object, otherwise properties
won't work.

FWIW, I don't see the point of still using old-style classes (apart for
compat with old Python versions...)

You mean like manually removed from the dict?
Yes.

Because (at least on the
system I tried) it doesn't happen automatically. (Windwos XP, Python 2.5)

My usual idiom for properties is:

class MyClass(object):
@apply
def propname():
def fget(self): return self._something * 2
def fset(self, value): self._something = int(value) / 2
return property(**locals())


(Don't remember where it comes from...)

 

[Bruno Desthuilliers]

I have a question about this section of a blog with that title. I'll ask
the question first so it doesn't get lost at the bottom: does the
following opinion of XML apply to GUIs? (i.e., using an XML resource to
define a GUI layout rather than coding it yourself).

There's a difference between not using XML and direct hand-coding -
Python is fairly usable as a configuration language (much more than XML
IMHO).

Now when it comes to GUI toolkits, they are usually coded in another
language, and usually come with language-agnostic GUI designers that use
XML. So the point is already solved somehow !-)

 

[John Salerno]

Now when it comes to GUI toolkits, they are usually coded in another
language, and usually come with language-agnostic GUI designers that use
XML. So the point is already solved somehow !-)

I'm thinking of hand-writing wxPython. I don't use a GUI designer for
it, I do it myself. So I can either write all the design code, or use a
program like XRCed to create the XML file that defines the layout.

 

[Antoine De Groote]

class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")

Altough I'm not the OP, thanks a lot to everyone, I've learned much in
this thread. But there's still something that puzzles me, I just realized.

In the snippet above (taken from the Python doc
http://docs.python.org/lib/built-in-funcs.html), self.__x is
initialized, but never used. I would appreciate any explanation for this.

Regards,
antoine

 

[Paul Rubin]

In the snippet above (taken from the Python doc
http://docs.python.org/lib/built-in-funcs.html), self.__x is
initialized, but never used. I would appreciate any explanation for
this.

Looks like a typo, should say self._x .

 

[Antoine De Groote]

Looks like a typo, should say self._x .

Ok, thanks, that's just what I was thinking..