import statement - package visibility problem

[Laszlo Zsolt Nagy]

Hi All!

I have the following structure:


/Lib/Server/Db/
__init_.py
DatabaseConnection.py
Adapters/
__init__.py
FireBirdConnection.py

Important file contents are:

/Lib/Server/Db/__init__.py:

import DatabaseConnection
import Adapters

/Lib/Server/Db/DatabaseConnection.py:

class DatabaseConnection(object):
pass


/Lib/Server/Db/Adapters/__init__.py

import FireBirdConnection

/Lib/Server/Db/Adapters/FireBirdConnection.py:

from DatabaseConnection import DatabaseConnection
class FireBirdConnection(DatabaseConnection):
pass

Here is the problem. I go to the directory where 'Lib' resides. Then I do:

C:\Temp\ccc>python
Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "Lib\__init__.py", line 1, in ?
import Server
File "Lib\Server\__init__.py", line 1, in ?
import Db
File "C:\Python\Lib\Db\__init__.py", line 29, in ?
import Adapters
File "C:\Python\Lib\Db\Adapters\__init__.py", line 21, in ?
import FireBirdConnection
File "Db\Adapters\FireBirdConnection.py", line 27, in ?
ImportError: No module named DatabaseConnection
My problem is that Lib/Server/Db/Adapters/FireBirdConnection.py does not
see Lib/Server/Db/DatabaseConnection.py.
Of course I read the documentation about the import statement but I do
not see a good solution.

Possible solutions and their drawbacks:

Solution 1:

Put the 'Db' directory into my PYTHONPATH or append it to "sys.path". It
is too bad. This could make my 'Db' module
visible but this is not a good solution. For example, I can have this
module structure:

Lib/
Client/
Db/
Adapters/
Server/
Db/
Adapters/

E.g. there could be two "Db" modules, one for Client and one for Server.
Clearly, I cannot add EVERY module path to sys.path. It would be
ambiguous to import 'Adapters', for example.

Solution 2:

Add the outermost 'Lib' directory to sys.path and use fully qualified
module names. In the example above I could use this:

/Lib/Server/Db/Adapters/FireBirdConnection.py:

from Lib.Server.Db.Adapters.DatabaseConnection import DatabaseConnection

In this case FireBirdConnection.py would be dependent on the full module
path. Otherwise it is independent on the whole module path, it only
depends on the 'upper level' module, regardless of its name. So here are
the problems with this solution:


- What if I want to rename 'Server' to 'Middletire'? Should I change all
of my files inside the 'Midletire' dir?
-What if I would like to unify the server and client Db sides? Should I
rename "from Lib.Server.Db.Adapters.DatabaseConnection import
DatabaseConnection" to "from Lib.Db.Adapters.DatabaseConnection import
DatabaseConnection" in all files?
- Lastly, the import statements are too long. They are hard to manage.

I would like to use something like

from __uppermodule__.DatabaseConnection import DatabaseConnection

but probably there is a standard Pythoninc way to do it. Just I can't
find it.
Maybe we can start a new PEP on this.

Please advise.

p.s.: sorry for the long e-mail

--
_________________________________________________________________
Laszlo Nagy web: http://designasign.biz
IT Consultant mail: (e-mail address removed)

Python forever!

 

[Paul Clinch]

Laszlo,
For :-

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "Lib\__init__.py", line 1, in ?
import Server
File "Lib\Server\__init__.py", line 1, in ?
import Db
File "C:\Python\Lib\Db\__init__.py", line 29, in ?
import Adapters
File "C:\Python\Lib\Db\Adapters\__init__.py", line 21, in ?
import FireBirdConnection
File "Db\Adapters\FireBirdConnection.py", line 27, in ?
ImportError: No module named DatabaseConnection I get:

Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named Lib

I guess there's a Lib/__init__.py.

But onto the point you're making. I think its possibly a mis-viewing
of the package idea in Python. A package creates a name space. If you
create Lib/Server/Db with all the __init__.py files, its because you
want to import Lib.Server.Db, rather than a way of organising your
source files.

If you want to have a single name space, and keep the nested
arrangement of directories, you can add python code in Lib/__init__.py
to traverse directories and import packages that you find. Or you can
define any rules for mapping files and directories to the name space
you desire.

Regards, Paul

 

[Laszlo Zsolt Nagy]

I get:


Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named Lib

I guess there's a Lib/__init__.py.

You are totally right. I was trying to create an example but Python
found something on my PYTHONPATH.
Sorry for the confusion, I sent a bad example. Well, could not be so bad
because you understood the point.
I'll try to create a new good example today and make it available.

But onto the point you're making. I think its possibly a mis-viewing
of the package idea in Python. A package creates a name space. If you
create Lib/Server/Db with all the __init__.py files, its because you
want to import Lib.Server.Db, rather than a way of organising your
source files.

I agree. Sometimes I need to import Lib.Server.Db but sometimes I need
to import Lib.Server.Db.Adapters.PostgreSQLConnection.
Importing from a not package related source code is not a problem,
really. My problem is about importing inside a package.

Lib/Server/Db/__init__.py __can__ import Lib/Server/Db/Adapters usign
relative paths, but

Lib/Server/Db/Adapters/PostgreSQLConnection.py __cannot__ import from
Lib/Server/Db using relative paths

Of course I can do the latter import using absolute paths but that is
something I would like to avoid. In other words:
it is possible to import using relative lib paths only in one direction.
Why is that? Why can't I import something
"from the containing package"?

Why I would like to do that? Please see my reasons in my original
e-mail. If you think that I want a programatically
bad thing, please tell me what is the right way to do it. I'm sorry if I
look demanding. I'm open to any suggestions.
I can hardly find a design problem in the language itself, I believe
these problems are just mine. I have a bug in my
mind. Please help me to catch it.

If you want to have a single name space, and keep the nested
arrangement of directories, you can add python code in Lib/__init__.py
to traverse directories and import packages that you find.

Ok, in the example, you will see that this is not possible. I'm going to
upload it soon.

Or you can define any rules for mapping files and directories to the name space
you desire.

How to do that? By hacking?


--
_________________________________________________________________
Laszlo Nagy web: http://designasign.biz
IT Consultant mail: (e-mail address removed)

Python forever!

 

[Laszlo Zsolt Nagy]

Here is the example:

http://designasign.biz/Lib2.zip

Please extract the file and try to import Lib2

Although \Lib2\Client\Db\DatabaseConnection.py and
\Lib2\Server\Db\DatabaseConnection.py have the same class name but they
are in a different sub-package. From a programmer's view, I hope this is
okay.
<module 'Lib2.Server.Db.Adapters.OracleConnection' from
'Lib2\Server\Db\Adapters\OracleConnection.pyc'>

As an example of my problem, please look inside OracleConnection. You
will see that it imports DatabaseConnection with an absolute package name:

from Lib2.Client.Db.DatabaseConnection import DatabaseConnection

so the OracleConnection.py file is dependent of its location in the
package tree. I would like to make it independent of the full package
tree since it only depends on 'the upper level'. I could have 10
different types of adapters and 3 drivers for each. This is not a
special exampe, I would like to do the same with many other classes:

- SQLProcessor (with implementations connected directly to databases,
logging to files, printing to stdout etc)
- ReportExporter (with implementations exporting reports to excel files,
text files, CSV files, XML files, PDF files etc.)

You can think about others but with the same structure:

- Create the abstraction level as the upper level package and
- not just separate the implementation by a new directory but also with
a new namespace (new sub-package).

The advantages are clear:

- Keep the implementation level and the abstraction level in well
separated namespaces
- Do not have redundacy in the code - every module can contain
dependencies but only to other modules/packages that are really needed
(for example, OracleConnection should not rely on the fact that the 'Db'
package is inside the 'Server' package. It should not even rely on that
the name of the containing package is 'Db'. Also it should not rely on
that the 'Lib2' package on sys.path etc.)
- As a result, you can move a sub-package to another location without
the need to modify possibly hundreds of files
- As a result you can move the whole package tree to anywhere and import
it without changes in your enviroment - this can lead to interesting
uses as well

Are these advantages good enough to start developing a magic
'__upperpackage__' variable?

This is my dream: assuming that the containing package is already
imported to Python, this statement:

from __upperpackage__.DatabaseConnection import DatabaseConnection

would bind '__upperpackage__' to the containing package. Of course, if
the containing package has not been imported yet then it would raise an
exception (ImportError: no module named __upperpackage__).

What do you think?

--
_________________________________________________________________
Laszlo Nagy web: http://designasign.biz
IT Consultant mail: (e-mail address removed)

Python forever!

 

[Laszlo Zsolt Nagy]

But onto the point you're making. I think its possibly a mis-viewing
of the package idea in Python. A package creates a name space. If you
create Lib/Server/Db with all the __init__.py files, its because you
want to import Lib.Server.Db, rather than a way of organising your
source files.

I went forward. I read all the documentation in the language reference.

About the import statement:

http://docs.python.org/ref/import.html

For importing packages and using hierarchical module names, there was no
detailed description but a link to an essay:

http://www.python.org/doc/essays/packages.html

There is a section in that essay named "Intra-package references". It says:

"""The submodules often need to refer to each other. For example, the
surround module might use the echo module. In fact, such references are
so common that the import statement first looks in the containing
package before looking in the standard module search path. Thus, the
surround module can simply use |import echo| or |from echo import
echofilter|. If the imported module is not found in the current package
(the package of which the current module is a submodule), the import
statement looks for a top-level module with the given name.

When packages are structured into subpackage (as with the Sound package
in the example), there's no shortcut to refer to submodules of sibling
packages - the full name of the subpackage must be used. For example, if
the module Sound.Filters.vocoder needs to use the echo module in the
Sound.Effects package, it can use |from Sound.Effects import echo|.

(One could design a notation to refer to parent packages, similar to the
use of ".." to refer to the parent directory in Unix and Windows
filesystems. In fact, ni supported this using __ for the package
containing the current module, __.__ for the parent package, and so on.
This feature was dropped because of its awkwardness; since most packages
will have a relative shallow substructure, this is no big loss.)
"""

So finally I got the answer: there was a "parent package" feature in the
ni module but it was dropped of its awkwardness. This is not a big loss
but this is exatly the feature that I need. Is there a person on this
list who was against the "parent package" idea? He must know the answer
or a workaround

--

_________________________________________________________________
Laszlo Nagy web: http://designasign.biz
IT Consultant mail: (e-mail address removed)

Python forever!

 

[Laszlo Zsolt Nagy]

So finally I got the answer: there was a "parent package" feature in
the ni module but it was dropped of its awkwardness. This is not a big
loss but this is exatly the feature that I need. Is there a person on
this list who was against the "parent package" idea? He must know the
answer or a workaround

I just created a new module that can handle the 'from __.__.SomeModule
import SomeClass' notation.
Keywords: Python import parent package
What is the right place to make it available for others?

Laci 2.0

--
_________________________________________________________________
Laszlo Nagy web: http://designasign.biz
IT Consultant mail: (e-mail address removed)

Python forever!



#!/usr/bin/env python

"""Import hack module.

Importing this module replaces the __builtin__.__import__ function
(called by the import statement). After that you will be able to
import parent packages this way:

from __.__.SomeModule import SomeClass

where __.__ means a package that is one level above in the filesystem.
Please note that you can use the special '__.__' name only in the import
statement. The name '__.__' itself cannot be used to reference the
parent package elsewhere.

NOTE: This syntax follows the convention introduced by the 'ni' package,
first appeared in Python 1.5. The 'parent package' idea was dropped
because of its awkwardness. For details, please visit

http://www.python.org/doc/essays/packages.html

If you still feel that you need to use this feature, you can use this module.


LICENSE: Python license (http://www.python.org/psf/)

@author: <[email protected]>
@modified: 2005-04-08 15:42

"""

import __builtin__
import imp
import os

# Keep a reference to the original __import__ function
_orig_import = __builtin__.__import__

def __import__(name,globals,locals,fromlist):
"""Specialized import function that can import names from parent packages
using this special form:

from __.__.SomeModule import SomeClass

Please note that you can use the special '__.__' name only in the import
statement. The name '__.__' itself cannot be used to reference
the parent package elsewhere.
"""
global _orig_import

def _tail_module_path(path):
"""Returns the tail of a module path.split.

If there is no tail (no dot in the module path) then raises an ImportError.

Example:
_tail_module_path('Lib.Db.PostgreSQL') will return 'Lib.Db'

"""
index = path.rfind('.')
if index <= 0:
raise ImportError('Cannot import __parentpackage__ from a top level package.')
return path[:index]
#print "__import__ was called with name %s" % name
if name.startswith(PARENTPACKAGE):
#print "Trying to import %s" % name
# Determine the file that needs to be imported
package_fpath = globals['__file__']
(dir,fname) = os.path.split(package_fpath)
double_parent = not fname.startswith('__init__.')
# Determine the new package name
package_ppath = globals['__name__']
upper_package_ppath = _tail_module_path(package_ppath)
if double_parent:
upper_package_ppath = _tail_module_path(upper_package_ppath)
#print " Package name in the module where import was executed: %s" % package_ppath
#print " Package name for the parent package: %s" % upper_package_ppath
name = name.replace(PARENTPACKAGE,upper_package_ppath)
return _orig_import(name,globals,locals,fromlist)

# Install the new __import__ function
__builtin__.__import__ = __import__

 

[Paul Clinch]

Laszlo,

....

Importing from a not package related source code is not a problem,
really. My problem is about importing inside a package.

Lib/Server/Db/__init__.py __can__ import Lib/Server/Db/Adapters usign
relative paths, but

Lib/Server/Db/Adapters/PostgreSQLConnection.py __cannot__ import from
Lib/Server/Db using relative paths

Of course I can do the latter import using absolute paths but that is
something I would like to avoid. In other words:
it is possible to import using relative lib paths only in one direction.
Why is that? Why can't I import something
"from the containing package"?

Yes it might be convenient for what you want to do, but importing from
containing package means traversing up the directory structure from
importing module.

How to do that? By hacking?

Well, yes, Python hacking. See the imp module for interesting
functions such as load_module:-

import imp
filename = 'Lib2/Server/Db/DatabaseConnection.py'
connection = load_module( 'connection', open(filename),
filename,('.py', 'U', 1) )

There is a load_package function not documented, I haven't attempted
to use it, but you could read the source (import.c).

You could also look at the builtin function __import__.

Lastly you could look at the new import hooks (PEP 302) in the sys
module, implemented in 2.3, where you define a find_module that
returns a loader object.

Regards, Paul Clinch

 

[Terry Hancock]

I just created a new module that can handle the 'from __.__.SomeModule
import SomeClass' notation.
Keywords: Python import parent package
What is the right place to make it available for others?

Well, generally speaking, it seems that most people would stick it on their
website and post a link here. Or you could just post the text here as an
example (I imagine it's pretty short?).

I'd find it useful myself.

Cheers,
Terry