Q on naming nested packages/modules

[kj]

I'm having a hard time getting the hang of Python's package/module
scheme. I'd like to find out what's considered best practice when
dealing with the scenario illustrated below.

The quick description of the problem is: how can I have two nested
modules, spam.ham and spam.ham.eggs?

Suppose I have a module (I'm not even sure this is the right word)
called spam.ham, so I start out with the following file structure:

spam/
|-- ham.py
`-- __init__.py

With this arrangement, the line

import spam.ham

....in client code works as expected.

But now suppose that I want to factor out some code in spam/ham.py
to a helper module. (The reason behind factoring out this new
module is to "declutter" spam/ham.py, and improve its readibility.)
My instinct (from my Perl past) is to put this factored-out code
in a file spam/ham/eggs.py, i.e. to create the "nested" module
spam.ham.eggs, which requires expanding the tree as follows

spam/
|-- ham/
| |-- eggs.py
| `-- __init__.py
|-- ham.py
`-- __init__.py

....and adding the following spam/ham.py to

# spam/ham.py
from . import eggs

This doesn't work so well, because now spam/ham.py is not read.
It seems that adding the spam/ham directory, or maybe adding the
file spam/ham/__init__.py, causes spam/ham.py to be overlooked.


Clearly, I'm not playing this game right...

What is considered "best practice" for the use case sketched above?
Should I, e.g. rename the directory spam/ham something like spam/ham_
and refer to the helper module as spam.ham_.eggs? Or is some other
convention preferred?

I consulted PEP 8, but besides recommending "short, all-lowercase
names" for modules, it gives little guidance on the situation
described above.

TIA!

kynn

 

[Benjamin Kaplan]

I'm having a hard time getting the hang of Python's package/module
scheme.  I'd like to find out what's considered best practice when
dealing with the scenario illustrated below.

The quick description of the problem is: how can I have two nested
modules, spam.ham and spam.ham.eggs?

Suppose I have a module (I'm not even sure this is the right word)
called spam.ham, so I start out with the following file structure:

 spam/
 |-- ham.py
 `-- __init__.py

With this arrangement, the line

import spam.ham

...in client code works as expected.

But now suppose that I want to factor out some code in spam/ham.py
to a helper module.  (The reason behind factoring out this new
module is to "declutter" spam/ham.py, and improve its readibility.)
My instinct (from my Perl past) is to put this factored-out code
in a file spam/ham/eggs.py, i.e. to create the "nested" module
spam.ham.eggs, which requires expanding the tree as follows

 spam/
 |-- ham/
 |   |-- eggs.py
 |   `-- __init__.py
 |-- ham.py
 `-- __init__.py

...and adding the following spam/ham.py to

# spam/ham.py
from . import eggs

This doesn't work so well, because now spam/ham.py is not read.
It seems that adding the spam/ham directory, or maybe adding the
file spam/ham/__init__.py, causes spam/ham.py to be overlooked.


Clearly, I'm not playing this game right...

What is considered "best practice" for the use case sketched above?
Should I, e.g. rename the directory spam/ham something like spam/ham_
and refer to the helper module as spam.ham_.eggs?  Or is some other
convention preferred?

I consulted PEP 8, but besides recommending "short, all-lowercase
names" for modules, it gives little guidance on the situation
described above.

Take everything in ham.py and stick it in ham/__init__.py instead.
This will give you the behavior you're looking for (don't import
spam.ham.__init__. Everything in __init__.py is loaded into spam.ham)

 

[kj]

I'm having a hard time getting the hang of Python's package/module
scheme. I'd like to find out what's considered best practice when
dealing with the scenario illustrated below.

The quick description of the problem is: how can I have two nested
modules, spam.ham and spam.ham.eggs?

Following up my own post...

From inspecting the directory structure of some of the standard
Python modules I infer the following rules:

1. the source for "leaf" modules lives in files named after them
(e.g. if x.y.z is a "leaf" module, its source code is in x/y/z.py)

2. the source for "non-leaf" modules lives in files named __init__.py
(e.g. if x.y is a "non-leaf" module, its source code lives in
the file x/y/__init__.py)

In the examples above, the module x.y is a "non-leaf" module because
there is a module x.y.z.

I.e. the "leaf"-ness of a module depends solely on whether other
modules deeper in the hierarchy are present.

An implication of all this is that if now I wanted to create a new
module x.y.z.w, this means that the previously "leaf"-module x.y.z
would become "non-leaf". In other words, I'd have to:

1. create the new directory x/y/z
2. *rename* the file x/y/z.py to x/y/z/__init__.py
3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
module

Is the above correct? (BTW, to my Perl-pickled brain, step 2 above
is the one that causes most distress... But I think I can cope.)

kynn

 

[Carl Banks]

I'm having a hard time getting the hang of Python's package/module
scheme.  I'd like to find out what's considered best practice when
dealing with the scenario illustrated below.

The quick description of the problem is: how can I have two nested
modules, spam.ham and spam.ham.eggs?

Suppose I have a module (I'm not even sure this is the right word)
called spam.ham, so I start out with the following file structure:

  spam/
  |-- ham.py
  `-- __init__.py

With this arrangement, the line

import spam.ham

...in client code works as expected.

But now suppose that I want to factor out some code in spam/ham.py
to a helper module.  (The reason behind factoring out this new
module is to "declutter" spam/ham.py, and improve its readibility.)
My instinct (from my Perl past) is to put this factored-out code
in a file spam/ham/eggs.py, i.e. to create the "nested" module
spam.ham.eggs, which requires expanding the tree as follows

  spam/
  |-- ham/
  |   |-- eggs.py
  |   `-- __init__.py
  |-- ham.py
  `-- __init__.py

...and adding the following spam/ham.py to

# spam/ham.py
from . import eggs

This doesn't work so well, because now spam/ham.py is not read.
It seems that adding the spam/ham directory, or maybe adding the
file spam/ham/__init__.py, causes spam/ham.py to be overlooked.

There's a hint here.

You can move the contents of ham.py into ham/__init__.py, and it'll
work the way you want, maybe with only a minor change or two.

Clearly, I'm not playing this game right...

What is considered "best practice" for the use case sketched above?
Should I, e.g. rename the directory spam/ham something like spam/ham_
and refer to the helper module as spam.ham_.eggs?  Or is some other
convention preferred?

The way you were intending is often the approach people use. I doubt
there are any detailed consensus recommendations about this in the
Python community.

I consulted PEP 8, but besides recommending "short, all-lowercase
names" for modules, it gives little guidance on the situation
described above.

Of course it doesn't, PEP 8 is a style guide, not a project
organization guide.


Carl Banks

 

[Ethan Furman]

Following up my own post...

Python modules I infer the following rules:

1. the source for "leaf" modules lives in files named after them
(e.g. if x.y.z is a "leaf" module, its source code is in x/y/z.py)

2. the source for "non-leaf" modules lives in files named __init__.py
(e.g. if x.y is a "non-leaf" module, its source code lives in
the file x/y/__init__.py)

In the examples above, the module x.y is a "non-leaf" module because
there is a module x.y.z.

I.e. the "leaf"-ness of a module depends solely on whether other
modules deeper in the hierarchy are present.

An implication of all this is that if now I wanted to create a new
module x.y.z.w, this means that the previously "leaf"-module x.y.z
would become "non-leaf". In other words, I'd have to:

1. create the new directory x/y/z
2. *rename* the file x/y/z.py to x/y/z/__init__.py
3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
module

Is the above correct? (BTW, to my Perl-pickled brain, step 2 above
is the one that causes most distress... But I think I can cope.)

kynn

Looking at the layout of the (most excellent!-) xlrd package, the bulk
of the code is in the __init__.py file, and other supporting code is the
same directory, accessed in __init__ with normal imports.

I also am unclear on when it's best to have supporting files in the same
directory versus a subdirectory... perhaps it is that flat is better
than nested?

~Ethan~

 

[Terry Reedy]

But now suppose that I want to factor out some code in spam/ham.py
to a helper module. (The reason behind factoring out this new
module is to "declutter" spam/ham.py, and improve its readibility.)
My instinct (from my Perl past) is to put this factored-out code
in a file spam/ham/eggs.py, i.e. to create the "nested" module
spam.ham.eggs, which requires expanding the tree as follows

spam/
|-- ham/
| |-- eggs.py
| `-- __init__.py
|-- ham.py
`-- __init__.py

...and adding the following spam/ham.py to

# spam/ham.py
from . import eggs

This doesn't work so well, because now spam/ham.py is not read.
It seems that adding the spam/ham directory, or maybe adding the
file spam/ham/__init__.py, causes spam/ham.py to be overlooked.


Clearly, I'm not playing this game right...

One way is the __init__.py answer you have already. Another is to put
eggs.py (or _eggs.py to indicate that it is private) in spam and skip
the ham directory.

spam/
__init__.py
ham.py
_eggs.py

tjr

 

[Rami Chowdhury]

An implication of all this is that if now I wanted to create a new

module x.y.z.w, this means that the previously "leaf"-module x.y.z
would become "non-leaf". In other words, I'd have to:

1. create the new directory x/y/z
2. *rename* the file x/y/z.py to x/y/z/__init__.py
3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
module

With regard to point 2 -- would it be possible to just move z.py into
x/y/z, and put 'from z import *' into x/y/z/__init__.py, for the same
effect? Or is that not a good idea?

 

[kj]

With regard to point 2 -- would it be possible to just move z.py into
x/y/z, and put 'from z import *' into x/y/z/__init__.py, for the same
effect? Or is that not a good idea?

I think in this case what you would need is something like 'from
...z import *', but this does not work either; one gets the error
"SyntaxError: 'import *' not allowed with 'from .'".

A third solution would be to change step 2 above to this:

2. create the *symlink* x/y/z/__init__.py --> ../z.py

This certainly would make things clearer-looking to me. Having
all these files called the same thing, __init__.py, but having
completely different meaning, makes my head hurt... In contrast,
the __init__.py *symlinks* would be there just to keep python happy;
the real content is where I'd expect it.

I ran a simple test of this idea and it works, but there may be
some subtle reasons for not doing it... Or maybe it's not a good
idea simply because it is not the Python way, and I'd agree that
this would be reason enough to avoid it.

kynn