What are modules really for?

[N.Davis]

I am very new to Python, but have done plenty of development in C++ and
Java.

One thing I find weird about python is the idea of a module. Why is this
needed when there are already the ideas of class, file, and package?

To my mind, although one CAN put many classes in a file, it is better to
put one class per file, for readability and maintainability.

One can then create packages and libraries, using groups of files, one
class per file.

Python seems to let you group classes together in one file and call it a
module, but what for?

I find that this, combined with mixins, makes it difficult to find out
where code is inherited from.

With single inheritance in C++ or Java, if you wanted to see what a
method did and it appeared to be inherited, you would simply look in the
base class's file, and if necessary recurse up the inheritance hierarchy
until you found the method.

With Python an inherited method could be in one of many base classes
and/or mixins and there seems no particular logic as to what the
filename would be.

Am I missing something?

Any comments would be appreciated.

Thanks
Nick Davis

 

[Richard Brodie]

To my mind, although one CAN put many classes in a file, it is better to
put one class per file, for readability and maintainability.

Zero classes in a file works well too

 

[skip]

Nick> One thing I find weird about python is the idea of a module. Why
Nick> is this needed when there are already the ideas of class, file,
Nick> and package?

A module is a namespace object that maps one-to-one to a file (ignoring the
new module's features). A file by itself is just a collection of bytes and
has no features to interpret the structure of its contents as a namespace.

Nick> Python seems to let you group classes together in one file and
Nick> call it a module, but what for?

You can define many other types of objects in modules besides classes.

Nick> With Python an inherited method could be in one of many base
Nick> classes and/or mixins and there seems no particular logic as to
Nick> what the filename would be.

I agree mixins can be troublesome. That's a problem peculiar to multiple
inheritance, not strictly to Python. I'm sure you can create file/class
relationships in Java or C++ that make it challenging to find the definition
of a mixin. I don't view multiple inheritance as a gotta-have feature of
OOP and avoid it when I can. Smalltalk has lived happily without multiple
inheritance for a few decades now.

Skip

 

[infidel]

I am very new to Python, but have done plenty of development in C++ and

Java.

And therein lies the root of your question, I believe.

One thing I find weird about python is the idea of a module. Why is this
needed when there are already the ideas of class, file, and package?

One reason is that functions need a place to exist too. In Java, every
function, even "static" ones has to be a class method. In Python,
"static" functions are best kept in a module. Another thing is that
packages were a later addition to the language.

To my mind, although one CAN put many classes in a file, it is better to
put one class per file, for readability and maintainability.

Personally I find it easier to maintain a set of related classes when
they're all in the same file. I've got a few modules that I'm
currently hacking on, each of which contains a handful of classes.
Maybe it's just a matter of scale, since these are both fairly small
libraries, but I just don't see any advantage to splitting them up into
multiple files.

One can then create packages and libraries, using groups of files, one
class per file.

Since Java's compiler enforces this, perhaps you've just come to accept
it as "normal".

Python seems to let you group classes together in one file and call it a
module, but what for?

What if one of your classes creates/manipulates other classes. If
they're in the same module then they all exist in the same namespace
and you don't have to have modules importing each other or such things.

I find that this, combined with mixins, makes it difficult to find out
where code is inherited from.

Perhaps you are relying too much on inheritance, then?

With single inheritance in C++ or Java, if you wanted to see what a
method did and it appeared to be inherited, you would simply look in the
base class's file, and if necessary recurse up the inheritance hierarchy
until you found the method.

With Python an inherited method could be in one of many base classes
and/or mixins and there seems no particular logic as to what the
filename would be.

It shouldn't be too hard to figure out, unless someone was being
intentially vague. You could always fire up the interpreter, import
your class, and check it's .mro property (method resolution order),
which lists the classes in the order they will be examined to find a
method named at runtime.

Am I missing something?

I just think you're thinking in terms of Java. You'll pick things up
quickly, though

 

[Guest]

With single inheritance in C++ or Java, if you wanted to see what a
method did and it appeared to be inherited, you would simply look in the
base class's file, and if necessary recurse up the inheritance hierarchy
until you found the method.

With Python an inherited method could be in one of many base classes

And Python doesn't differ from C++ in this matter, as C++ have multiple
inheritance during (more than) last 15 years. If you don't use it in C++
then feel free not to use it in Python as well.

 

[N.Davis]

Thanks very much for your helpful replies.
I totally accept my C++/Java background is a lot of this, and I need to
make the shift to Python's way of thinking.
As for multiple inheritance, yes I've always been aware of it being
available in C++, but I learned C++ at a company which banned multiple
inheritance in their coding standards, with comments about "The GOTO of
the 1990s". Like people here, I've no complaints about not using
multiple inheritance. ;-) But Zope/Plone uses it therefore I have to
understand it...
Functions existing in a module? Surely if "everything is an object" (OK
thats Java-talk but supposedly Python will eventually follow this too)
then there should be nothing in a module thats not part of a class. Even
a static method is simply a class function that operates on the
"collection of all instances" rather than a single instance.
Related classes in the same file? Be careful. Doesn't anything "knowing"
about anything else compromise encapsulation? Why would
properly-designed classes have such a close relationship?
Having back in the day worked on big real-time systems where being very
strict about encapsulation was a god-send for fighting complexity, I
feel unnerved by Perl and Python's laid-back OO culture of "you can do
it if you feel like it but don't have to". While you could do all manner
of nasty hacks in C++ I worked with people who carefully avoided this.
Maybe that was just luck....

Thanks again. Python is great anyway.



Nick

 

[N.Davis]

Thanks very much for your helpful replies.
I totally accept my C++/Java background is a lot of this, and I need to
make the shift to Python's way of thinking.
As for multiple inheritance, yes I've always been aware of it being
available in C++, but I learned C++ at a company which banned multiple
inheritance in their coding standards, with comments about "The GOTO of
the 1990s". Like people here, I've no complaints about not using
multiple inheritance. ;-) But Zope/Plone uses it therefore I have to
understand it...
Functions existing in a module? Surely if "everything is an object" (OK
thats Java-talk but supposedly Python will eventually follow this too)
then there should be nothing in a module thats not part of a class. Even
a static method is simply a class function that operates on the
"collection of all instances" rather than a single instance.
Related classes in the same file? Be careful. Doesn't anything "knowing"
about anything else compromise encapsulation? Why would
properly-designed classes have such a close relationship?
Having back in the day worked on big real-time systems where being very
strict about encapsulation was a god-send for fighting complexity, I
feel unnerved by Perl and Python's laid-back OO culture of "you can do
it if you feel like it but don't have to". While you could do all manner
of nasty hacks in C++ I worked with people who carefully avoided this.
Maybe that was just luck....

Thanks again.



Nick

 

[Dan]

Functions existing in a module? Surely if "everything is an object" (OK

thats Java-talk but supposedly Python will eventually follow this too)
then there should be nothing in a module thats not part of a class.

No, it's a bit the other way around. In Python, functions are objects:
('abc', 'xyz')

Related classes in the same file? Be careful. Doesn't anything "knowing"
about anything else compromise encapsulation?

You might think of modules in Python as like packages in Java. However,
putting classes in the same module doesn't give them any additional
powers to interact with each other. (At least, not that I know of.)

 

[Tito]

Functions existing in a module? Surely if "everything is an object" (OK
thats Java-talk but supposedly Python will eventually follow this too)
then there should be nothing in a module thats not part of a class.

Well, all data in a Python program are objects, in the sense that they
have an identity on their own, and variables are only references to them.

Why is it necessary to have all code into classes for meeting the
"everything is an object" panacea?

Even
a static method is simply a class function that operates on the
"collection of all instances" rather than a single instance.

That is not true. A static method knows nothing about the instances of a
class, unless you do it your own. Besides, it will work whether you
have created instances of the class or not.

So, a static method is just a global method declared withing a class,
which just serves as a namespace for it.

Related classes in the same file? Be careful. Doesn't anything "knowing"
about anything else compromise encapsulation? Why would
properly-designed classes have such a close relationship?

Question back: why do you think having classes defined in the same file
compromises encapsulation? Classes don't know more about each other for
the fact of being written into the same file. Anyway, in Python, classes
know always too much from each other, don't they?, as there are no
access modifiers for class attributes.

Having back in the day worked on big real-time systems where being very
strict about encapsulation was a god-send for fighting complexity, I
feel unnerved by Perl and Python's laid-back OO culture of "you can do
it if you feel like it but don't have to".

Well, that is the case with whatever general-purpose programming
language. You have virtually an infinite number of ways to do things,
and most of them are not appropriate.

The thing with Python is, in my opinion, that it wants to put all the
power in your hands to do whatever you want in the fastest way possible.
If I want to do something, I don't declare a class that knows how to do
it, then create it and invoke the right method, I just put the code to
do what I want to do. If, in between, I find that one class would make
my life easier, I declare it just in place and go on. If I find
repetitive tasks I can declare functions, but I won't go for a class
immediately. Why do you think putting code into functions is not
encapsulating?

While you could do all manner
of nasty hacks in C++ I worked with people who carefully avoided this.

Well done, but messes you can do in whatever language.

Regards,
Tito

 

[Neil Benn]

N.Davis wrote:



Well, all data in a Python program are objects, in the sense that they
have an identity on their own, and variables are only references to them.

Why is it necessary to have all code into classes for meeting the
"everything is an object" panacea?

If you don't have a class how can you combine functionality with data
and hold state - that is one of the points of a class. Python has the
added concept that if you don;t need this you can create functions with
no associations - lots of languages do that but Java, Eiffel, C#, etc
don't have that concept. You need to make a difference between
everything is an object and everything is a class here. A programmer
coming from Java/C# wouldn't think about that.

That is not true. A static method knows nothing about the instances of a
class, unless you do it your own. Besides, it will work whether you
have created instances of the class or not.

So, a static method is just a global method declared withing a class,
which just serves as a namespace for it.

In java, a static method is the same as a class method in Python, you
can then use the cls param to access class attributes (that is what teh
OP implied). However a static method can help with namespacing. What
looks better :

initialise_local_message_router()
objPublisher = get_publisher_from_local_router('bob')

or

LocalRouter.initialise()
objPublisher = LocalRouter.getPublisher('bob')

IMHO, the second case makes much more sense that a floating function
which makes things less expressive.

Question back: why do you think having classes defined in the same file
compromises encapsulation? Classes don't know more about each other for
the fact of being written into the same file. Anyway, in Python, classes
know always too much from each other, don't they?, as there are no
access modifiers for class attributes.

If I want to change one class and replace the file on the install then I
need to put a whole bunch of classes on - increasing the change of
making a mistake. Module scoping exists with globals, also you have the
convetnion of creating classes with the name of _XXX to mean module
level only.

Well, that is the case with whatever general-purpose programming
language. You have virtually an infinite number of ways to do things,
and most of them are not appropriate.

Some languages are more strict than others - yes you need covnentions
but you will need more conventions in a less strict language. The
upside is that, if you are careful, you can leverage the added oosness
to implement features which would be a pain in other languages.
Enterprise systems have different objectives than a cgi script or single
client install stuff. It's a judgement call.

The thing with Python is, in my opinion, that it wants to put all the
power in your hands to do whatever you want in the fastest way possible.
If I want to do something, I don't declare a class that knows how to do
it, then create it and invoke the right method, I just put the code to
do what I want to do. If, in between, I find that one class would make
my life easier, I declare it just in place and go on. If I find
repetitive tasks I can declare functions, but I won't go for a class
immediately. Why do you think putting code into functions is not
encapsulating?




Well done, but messes you can do in whatever language.

Agreed but in some languages you can cause more havoc than in others,
look at the distinction .NET makes between managed and unmanaged code to
get a handle on this.

Regards,
Tito

Cheers,

Neil

--

Neil Benn
Senior Automation Engineer
Cenix BioScience
BioInnovations Zentrum
Tatzberg 47
D-01307
Dresden
Germany

Tel : +49 (0)351 4173 154
e-mail : (e-mail address removed)
Cenix Website : http://www.cenix-bioscience.com

 

[Peter Hansen]

You might think of modules in Python as like packages in Java. However,
putting classes in the same module doesn't give them any additional
powers to interact with each other. (At least, not that I know of.)

Use of "global" to rebind shared module-scope names...

-Peter

 

[Sion Arrowsmith]

[ somebody else wrote: ]
To my mind, although one CAN put many classes in a file, it is better to
put one class per file, for readability and maintainability.

Personally I find it easier to maintain a set of related classes when
they're all in the same file.

Real world example: I have here an API I'm one of the maintainers of
which is available in C++, Java and Python versions. Part of the
nature of this beast is that it has about 20 exception classes derived
from the same base class. So that's one .py for the Python interface,
a .h and a .cxx for the C++ (there is some logic in the base class
which counter-indicates stuffing it all in one .h), and 21 .java files
(plus their corresponding .classes). Take a guess which ones[*] are
easiest to maintain. ([*] hint)

 

[bruno modulix]

N.Davis wrote:
(snip)

Functions existing in a module? Surely if "everything is an object" (OK
thats Java-talk but supposedly Python will eventually follow this too)
then there should be nothing in a module thats not part of a class.

Why ? The name of the paradigm is *object* oriented, not *class*
oriented !-)

Believe it or else, there are OO languages that don't even have a
concept of 'class' (self, Javascript, Io, ...). They are called
prototype-based languages (by opposition to class-based), and are
usually much more object-oriented than Java.

In Python, "everything is an object" means that even classes and
functions are objects. Try this:

def fun(): pass
print fun.__class__.__name__

in fact, 'def' is just synctatic sugar to create a new instance of class
'function'


Now try this:

class Function(object):
def __call__(self, arg):
print "%s called with arg %s" % (self, arg)

f = Function()
f(42)

So in fact, a function is just a special case of a callable object.

When we say that Python is 100% OO, we mean it !-)

Even
a static method is simply a class function that operates on the
"collection of all instances" rather than a single instance.

Nope. In most Java code, static methods are functions in disguise. Write
a Java class with only static methods and static variables, and you have
a very classic procedural module...

Related classes in the same file? Be careful.

Of what ? This is a common practice in most OO languages, you know...

Doesn't anything "knowing"
about anything else compromise encapsulation?

Nope. Any of your objects needs to know at least something about at
least another object, or how could they send messages to each other ?

BTW encapsulation is *not* data-hiding.

Why would
properly-designed classes have such a close relationship?

I don't think you understand what kind of 'close relationship' we're
talking about. It's not about "a class knowing each and every
implementation detail of another class", it's about a bunch of classes
designed to work together to achieve a given goal.

BTW, note that the stronger coupling in OO is probably implementation
inheritence, and it's overabused in most static-typed languages.

Having back in the day worked on big real-time systems where being very
strict about encapsulation was a god-send for fighting complexity, I
feel unnerved by Perl and Python's laid-back OO culture

Please avoid comparing oranges and apples. Perl culture is about hacks
and unreadable code, when Python's culture is about simplicity and
readability. Just not the same culture.

of "you can do
it if you feel like it but don't have to".

Some languages have what we call a "discipline-and-bondage" culture.
Some others have a "we're all consenting adults here" culture. The
difference is that the first category of languages impose arbitrary
restrictions on the programmer, that usually lead to more verbose and
complex code, which usually leads to much opportunities for bugs.

While you could do all manner
of nasty hacks in C++ I worked with people who carefully avoided this.
Maybe that was just luck....

Nope, just common sens I'd say. But also, C++ - just like C - is a
low-*level language that allows you to do what you want with resources
(and specially with memory), so the consequences of a programming error
can be much more desastrous.

Now "nasty hacks" are not part of the Python culture. Dynamicity and
simplicity are not nasty hacks, just a way to get rid of useless complexity.

If you have a problem with having many classes in one file, then you'll
have a hard time with some other Python's features, like eg relying on
conventions for access control[1] or duck-typing[2]


[1] 'aName' => public, '_aName' => protected, '__aName' => private

[2] "if it walks like a duck and quake like a duck, then it's close
enough to a duck for what we want to do with it...."


So my best advice here is: don't fight against the language, don't waste
your time trying to enforce Java idioms in Python (I did, so believe me
when I tell you it's a pure waste of time), just forget Java idioms and
learn Python as it is. You'll discover that Python is not a "scripting"
language, but one of the most powerful and enjoyable languages around.


(disclaimer : my sig is a deliberate counter-exemple of good Python
coding style)

 

[bruno modulix]

Neil Benn wrote:
(snip)

If you don't have a class how can you combine functionality with data
and hold state

In Python, functions can hold state in various ways: closures,
generators, and - Python functions being instances of the function class
- attributes.

- that is one of the points of a class.

Yeps, but not the only way to solve the problem (even if it's actually
my favorite one !-)

(snip)

Enterprise systems have different objectives than a cgi script or single
client install stuff.

Would you call Zope a cgi script or a single client install stuff ?

 

[Tito]

If you don't have a class how can you combine functionality with data
and hold state - that is one of the points of a class.

Yes, of course. You don't have OO if you cannot define your own classes.
I only said that I don't find it necessary to have all code confined in
classes.

Python has the
added concept that if you don;t need this you can create functions with
no associations - lots of languages do that but Java, Eiffel, C#, etc
don't have that concept. You need to make a difference between
everything is an object and everything is a class here. A programmer
coming from Java/C# wouldn't think about that.
In java, a static method is the same as a class method in Python, you
can then use the cls param to access class attributes (that is what teh
OP implied). However a static method can help with namespacing. What
looks better :

initialise_local_message_router()
objPublisher = get_publisher_from_local_router('bob')

or

LocalRouter.initialise()
objPublisher = LocalRouter.getPublisher('bob')

IMHO, the second case makes much more sense that a floating function
which makes things less expressive.

Yes. So we agree on the fact that the class' name serves as a namespace.

If I want to change one class and replace the file on the install then I
need to put a whole bunch of classes on - increasing the change of
making a mistake.

Sorry, I don't understand the previous sentence. What is meant by
"replace on the install"? And by "to put a whole bunch of classes on"?

Module scoping exists with globals, also you have the
convetnion of creating classes with the name of _XXX to mean module
level only.

Do you mean that _XXX server as an access modifier among modules? Yes,
that can be a reason to split definitions of classes into different modules.

Some languages are more strict than others - yes you need covnentions
but you will need more conventions in a less strict language. The
upside is that, if you are careful, you can leverage the added oosness
to implement features which would be a pain in other languages.
Enterprise systems have different objectives than a cgi script or single
client install stuff. It's a judgement call.

Agreed but in some languages you can cause more havoc than in others,
look at the distinction .NET makes between managed and unmanaged code to
get a handle on this.

Well, I wasn't talking exactly about making use of dangerous features of
languages, but more about the way of using imperative object-oriented
languages in a non-object-oriented way.

Regards,
Tito

 

[Tito]

[1] 'aName' => public, '_aName' => protected, '__aName' => private

I didn't know this one, as I am quite new to Python. Is it really
general use?

Regards,
Tito

 

[bruno modulix]

[1] 'aName' => public, '_aName' => protected, '__aName' => private

I didn't know this one, as I am quite new to Python. Is it really
general use?

Yes, it's the convention.

Well, to be more exact, this is:

name => interface (intended for public use)
_name => implementation (not intented for public use)
__name => mangled (should not be hidden or overriden) [1]
__name__ => magic (has special meaning for the Python interpreter) [2]

BTW, from module import * tends to only import 'public' symbols, so you
can use _name to prevent _name being imported by accident.

[1] The purpose here is not to enforce privacy, but to protect the
symbolf from being accidentaly overriden or hidden.
given:
class Foo(object):
def __init__(self, name):
self.__name = name
f = Foo('foo')

You'll have to use :
f._Foo__name

to access foo.__name from outside the class (this hold for derived
classes too).


[2] __name__ attributes (variables and functions) usually have special
behaviours attached to them. For exemple,
- obj.__init__(self, ...) is called on object initialisation
- seq.__len__() is called by len(seq),
- obj.__str__() is called by str(obj)
- obj.__add__ is the '+' operator called by obj1 + obj2 ->
obj1.__add__(self, obj2),
- obj.__getattr__(self, name) is called when you lookup an inexistant
attribute of obj, etc...

You'll find all this (and much more) in the doc.

HTH

 

[Guest]

As for multiple inheritance, yes I've always been aware of it being
available in C++, but I learned C++ at a company which banned multiple
inheritance in their coding standards, with comments about "The GOTO of
the 1990s".

Looks like something religious. It seems they was completely ignorant about
what the multiple inheritance is and what is it for.

 

[Mike Meyer]

Use of "global" to rebind shared module-scope names...

You can rebind module-scope names from outside the module. It's an
ugly practice, and I wouldn't recommend it, but it *is* possible.

But you're on the right track. If two consenting classes want to share
_named module variables, it's best that they be in the same module.

<mike

 

[Neil Benn]

Sorry, I don't understand the previous sentence. What is meant by
"replace on the install"? And by "to put a whole bunch of classes on"?




<snip>

Suppose you have a logistics tracking system available on every install
in your company - there are 55 installs throughout the company. You
wish to push through a patch because of a problem. If you have one
class per file you can push that class through onto the client install.
However, if you have 15 different classes in one file - you will need to
drop all 15 classes through, thereby increasing the likelihood of
accidently pushing a bug onto the install. If you want to do live
updating (don;t think that this is a feature of Python so it's acadmenic
here) then what do you do, reload all 15 classes - just the one you've
changed?

Well, I wasn't talking exactly about making use of dangerous features of
languages, but more about the way of using imperative object-oriented
languages in a non-object-oriented way.

well, in languages like C#, Java and eiffel - you shouldn't really be
doing that anyways. that is why they are poor choices for hobbies,
hacks and quickie.

Regards,
Tito

P.S. why is Glenn McGrath walking about??

Cheers,

Neil

--

Neil Benn
Senior Automation Engineer
Cenix BioScience
BioInnovations Zentrum
Tatzberg 47
D-01307
Dresden
Germany

Tel : +49 (0)351 4173 154
e-mail : (e-mail address removed)
Cenix Website : http://www.cenix-bioscience.com