Relative Imports, why the hell is it so hard?

[Maxim Khitrov]

Does it not bother you that a module that uses relative imports cannot
be run on its own anymore?

$ python --help
<snip>
-m mod : run library module as a script (terminates option list)
<snip>
$ python -m some.module.name

Works perfectly fine with relative imports.

- Max

 

[Steve Holden]

Top responses guys! This has all helped increadibly.

Bearophile,

My applogies if I have offended you, but:

1. "I can't know much about you from the start" - Is kind of my point.
Perhaps it would be better to avoid jumping to conclusions and pre-
judging someones abilities simply because they are lacking knowledge
in a particular area.

I agree in your case it meant you were misjudged, but experience has
proved that it's important to establish the correct level of discourse
before proceeding to discuss solutions.

Would it have been better if I had opened my thread with a copy of my
CV? I've got a Degree in Digital Systems Engineering (yes I am also an
Engineer)... I went on to do a Phd in AI and Robotics where I also
developed embedded systems. I bummed out on that after 3 years and
went on to work for a games company where I worked on 6 game titles
across 5 different platforms. I was a Lead Software Engineer for the
last 5 years. Before now moving on again.

No, none of that would have helped ;-)

2. I am also very much a follower of the K.I.S.S. approach, 9 times
out of 10 the simplest solution is often the best.

As an Engineer though I would also suggest that the best way to learn
is to try solving a problem being sure to constantly assess your
approach and then your final solution. By dismissing a possible avenue
you are dismissing a whole new path of knowledge. Is it not better to
try and fail than never try at all? Surely this is how we gain the
valuable experience we need.

By simply suggesting a "simple default" solution, I may never have
consider the alternatives nor understand why they are or are not
suitable.

That's true, but we seem to be converging to rational discussion.

3. I get your point about Students, sometimes there is such a thing as
too much knowledge or information overload. Whilst doing a PhD I had
to take labs and teach students... The approach I tried to take was
one off, "You may not need packages, why do you think you need
packages?" or "This is how packages would be used and why they would
be used... do you still think you need packages" or better still, for
a capable student, "This is how packages work, try solving your
problem and then tell me if you think it was a good solution."


Going with R. David Murray, perhaps I also jumped too my own
conclusion too quickly and for that I appologise.

People round here generally have broad shoulders and are slow to take
offense. No need to worry - and welcome to c.l.py!

regards
Steve

 

[Istvan Albert]

Works perfectly fine with relative imports.

This only demonstrates that you are not aware of what the problem
actually is.

Try using relative imports so that it works when you import the module
itself. Now run the module as a program. The same module that worked
fine when you imported it will raise the exception:

ValueError: Attempted relative import in non-package

when running it on its own.

Istvan

 

[Carl Banks]

This only demonstrates that you are not aware of what the problem
actually is.

Try using relative imports so that it works when you import the module
itself. Now run the module as a program. The same module that worked
fine when you imported it will raise the exception:

PEP 366 addresses this issue.

Not the best solution, one that still involves boilerplate, but it is
much less of a hack than your version, and at least it has the
blessing of the language designers so it won't unceremoniously break
at some point.


Carl Banks

 

[Kay Schluehr]

PEP 366 addresses this issue.

Not the best solution, one that still involves boilerplate, but it is
much less of a hack than your version, and at least it has the
blessing of the language designers so it won't unceremoniously break
at some point.

Carl Banks

A workaround that is hardly acceptable when we are working with /
debugging 3rd party packages. Python was simpler without relative
imports and occasional ambiguity resolutions by means of absolute
imports. Unfortunately Brett Cannons reconstruction of import
semantics comes a little late for Python 3 and I suppose we have to
live with the current mess.

 

[Gabriel Genellina]

En Tue, 24 Mar 2009 21:57:12 -0300, Istvan Albert

yeah yeah, could we not get sidetracked with details that are not
relevant? what it obviously means is to import it in all of your
modules that need to access to relative paths

Uh? You posted a module containing a function and some code that modifies
sys.path. I imagine you are interested in those side effects in sys.path
-- which will only happen the first time it is imported. Isn't it relevant?

That's only because you have not had to deal with the problem that it
solves.

But I had to deal with the problem that it *creates*, even before relative
imports existed, and it's a lot worse.

confuse the import system? what the heck does that mean? You either
have a path in the sys.path or not. FWIW it is far cleaner than doing
a relative import that does not work correctly.

Others have posted the way to execute modules inside a package. For the
problem of adding random directories to sys.path, see:
http://groups.google.com/group/comp...e31ca3d411/e22039a2cf166f42?#e22039a2cf166f42

 

[Carl Banks]

A workaround that is hardly acceptable when we are working with /
debugging 3rd party packages. Python was simpler without relative
imports and occasional ambiguity resolutions by means of absolute
imports. Unfortunately Brett Cannons reconstruction of import
semantics comes a little late for Python 3 and I suppose we have to
live with the current mess

Out of curiosity, is there anything--aside from explicit relative
imports--worked before but doesn't now?

The issue Istvan is talking about exists even with the implicit
imports.


Carl Banks

 

[Aahz]

I'd recommend the oposite - use relative (intra-package) imports when
possible. Explicit is better than implicit - and starting with 2.7 -when
"absolute" import semantics will be enabled by default- you'll *have* to
use relative imports inside a package, or fail.

Really? I thought you would still be able to use absolute imports; you
just won't be able to use implied relative imports instead of explicit
relative imports.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan

 

[Gabriel Genellina]

Really? I thought you would still be able to use absolute imports; you
just won't be able to use implied relative imports instead of explicit
relative imports.

You're right, I put it wrongly. To make things clear, inside a package
"foo" accessible thru sys.path, containing a.py and b.py:

site-packages/
foo/
a.py
b.py
__init__.py

Currently, the "a" module can import "b" this way:

from foo import b
import foo.b
from . import b
import b

When implicit relative imports are disabled ("from __future__ import
absolute_import", or after 2.7 supposedly) the last one won't find b.py
anymore.
(I hope I put it right this time).

 

[Kay Schluehr]

You're right, I put it wrongly. To make things clear, inside a package
"foo" accessible thru sys.path, containing a.py and b.py:

site-packages/
foo/
a.py
b.py
__init__.py

Currently, the "a" module can import "b" this way:

from foo import b
import foo.b
from . import b
import b

When implicit relative imports are disabled ("from __future__ import
absolute_import", or after 2.7 supposedly) the last one won't find b.py
anymore.
(I hope I put it right this time).

So it even breaks more code which is great

Do you know of any near or far past attempts to re-design the import
system from the ground up? I do not mean a rather faithful and
accessible reconstruction such as Brett Cannons work but a radical re-
design which starts with a domain model and does not end with Loaders,
Importers and Finders which are actually services that pretend to be
objects.

Kay

 

[s4g]

Hi,

I was looking for a nice idiom for interpackage imports as I found
this thread.
Here come a couple of solutions I came up with. Any discussion is
welcome.

I assume the same file structure

\ App
| main.py
+--\subpack1
| | __init__.py
| | module1.py
|
+--\subpack2
| | __init__.py
| | module2.py


When you run main.py all imports relative to \App work fine, so the
only problem is running a module from within a subpackage as a script.
I therefore assume we want to run module1.py as a script, which wants
to import module2.

I hope the following solutions are self-evident

================= solution 1
--> in module1.py
import sys, os
if __name__ == '__main__':
sys.path.append(os.path.normpath(__file__+'/../..'))

import subpack2.module2

================= solution 2
--> in subpack1/__init__.py
import sys, os

_top_package_level = 1 # with current package being level 0

_top_package = os.path.normpath(__file__ + '/..'*(_top_package_level
+1))
if _top_package not in sys.path:
sys.path.append(_top_package)

--> in module1 or any module in the package, which requires import
relative to the package top
import __init__
import subpack2.module2


================= solution 3
--> in package_import.py, somewhere on the PYTHONPATH ( perhaps in
standard lib

def set_top_package(module, level):
_top_package = os.path.normpath(module + '/..'*(level+1))
if _top_package not in sys.path:
sys.path.append(_top_package)

class absolute_import(object):
def __init__(self, module, level):
self.level = level
self.module = module

def __enter__(self):
sys.path.insert( 0,
os.path.normpath(self.module + '/..'*(self.level+1))
)

def __exit__(self, exc_type, exc_val, exc_tb):
del sys.path[0]

--> in module1
import package_import
package_import.set_top_package(__file__, 1)
import subpack2.module2

--> or in module1
import package_import
with package_import.absolute_import(__file__, 1):
import subpack2.module2
...

 

[Kay Schluehr]

Hi,

I was looking for a nice idiom for interpackage imports as I found
this thread.
Here come a couple of solutions I came up with. Any discussion is
welcome.

I assume the same file structure

\ App
| main.py
+--\subpack1
| | __init__.py
| | module1.py
|
+--\subpack2
| | __init__.py
| | module2.py

When you run main.py all imports relative to \App work fine, so the
only problem is running a module from within a subpackage as a script.
I therefore assume we want to run module1.py as a script, which wants
to import module2.

I hope the following solutions are self-evident

================= solution 1
--> in module1.py
import sys, os
if __name__ == '__main__':
    sys.path.append(os.path.normpath(__file__+'/../..'))

import subpack2.module2

================= solution 2
--> in subpack1/__init__.py
import sys, os

_top_package_level = 1   # with current package being level 0

_top_package = os.path.normpath(__file__ + '/..'*(_top_package_level
+1))
if _top_package not in sys.path:
    sys.path.append(_top_package)

--> in module1 or any module in the package, which requires import
relative to the package top
import __init__
import subpack2.module2

================= solution 3
--> in package_import.py, somewhere on the PYTHONPATH ( perhaps in
standard lib

def set_top_package(module, level):
    _top_package = os.path.normpath(module + '/..'*(level+1))
    if _top_package not in sys.path:
        sys.path.append(_top_package)

class absolute_import(object):
    def __init__(self, module, level):
        self.level = level
        self.module = module

    def __enter__(self):
        sys.path.insert( 0,
            os.path.normpath(self.module + '/..'*(self.level+1))
            )

    def __exit__(self, exc_type, exc_val, exc_tb):
        del sys.path[0]

--> in module1
import package_import
package_import.set_top_package(__file__, 1)
import subpack2.module2

--> or in module1
import package_import
with package_import.absolute_import(__file__, 1):
    import subpack2.module2
    ...

This and similar solutions ( see Istvan Alberts ) point me to a
fundamental problem of the current import architecture. Suppose you
really want to run a module as a script without a prior import from a
module path:

....A\B\C> python my_module.py

then the current working directory C is added to sys.path which means
that the module finder searches in C but C isn't a known package.
There is no C package in sys.modules even if the C directory is
"declared" as a package by placing an __init__.py file in it. Same
goes of course with B and A. Although the ceremony has been performed
basically correct the interpreter god is not pacified and doesn't
respond. But why not? Because it looks up for *living* imported
packages in the module cache ( in sys.modules ).

I don't think there is any particular design idea behind it. The
module cache is just a simple flat dictionary; a no-brainer to
implement and efficient for look ups. But it counteracts a domain
model. All you are left with is those Finders, Loaders and Importers
in Brett Cannons importlib. Everything remains deeply mysterious and I
don't wonder that it took long for him to work this out.

 

[Terry Reedy]

On 31 Mrz., 18:48, s4g <[email protected]> wrote:

This and similar solutions ( see Istvan Alberts ) point me to a
fundamental problem of the current import architecture. Suppose you
really want to run a module as a script without a prior import from a
module path:

...A\B\C> python my_module.py

then the current working directory C is added to sys.path which means
that the module finder searches in C but C isn't a known package.
There is no C package in sys.modules even if the C directory is
"declared" as a package by placing an __init__.py file in it. Same
goes of course with B and A.

Nothing is added to sys.modules, except the __main__ module, unless
imported (which so are on startup).

Although the ceremony has been performed
basically correct the interpreter god is not pacified and doesn't
respond.

But the import 'ceremony' has not been performed.

But why not? Because it looks up for *living* imported
packages in the module cache ( in sys.modules ).

I don't think there is any particular design idea behind it. The
module cache is just a simple flat dictionary; a no-brainer to
implement and efficient for look ups.

This all dates to the time before packages and imports from zip files
and such.

> But it counteracts a domain model.

What is that?

All you are left with is those Finders, Loaders and Importers
in Brett Cannons importlib. Everything remains deeply mysterious and I
don't wonder that it took long for him to work this out.

And your proposal is?

tjr

 

[Kay Schluehr]

Nothing is added to sys.modules, except the __main__ module, unless
imported (which so are on startup).

Yes. The startup process is opaque but at least user defined modules
are not accidentally imported.

But the import 'ceremony' has not been performed.

There is no import ceremony. Imports are just stated in the source.
There is a package ceremony for whatever reasons.

This all dates to the time before packages and imports from zip files
and such.


What is that?

Object oriented programming.

And your proposal is?

I have still more questions than answers.

 

[Carl Banks]

I have still more questions than answers.

That's obvious.

Perhaps you should also refrain from making sweeping negative
judgments about a system you have more questions than answers about.

(Feel free to make sweeping negative judgments once you have the
answers, though.)


Carl Banks

 

[Terry Reedy]

There is no import ceremony. Imports are just stated in the source.
There is a package ceremony for whatever reasons.

Sorry, I have no idea what 'package ceremony' means.

Object oriented programming.

Domain model == oop? New one for me.

I have still more questions than answers.

OK. I will just note that import statements are syntactic sugar for
__import__ calls and name bindings. One could try out other import
schemes, just without the sugar.

tjr

 

[Kay Schluehr]

That's obvious.

Perhaps you should also refrain from making sweeping negative
judgments about a system you have more questions than answers about.

(Feel free to make sweeping negative judgments once you have the
answers, though.)

Carl Banks

Diagnosing a problem means having a detailed cure? Wow, that's
critical thinking I guess.

O.K. I have got some ideas for a new import system and I'm going to
blog about them within the next days. If I have some results I'll
leave a notice in this thread.

 

[Carl Banks]

Diagnosing a problem means having a detailed cure? Wow, that's
critical thinking I guess.

A diagnosis commesurate with the understanding you've displayed in
this thread so far would be something like "Python's import system is
too complicated", not "Python's import system is fundamentally
broken".

The latter is not something someone who has more questions than
answers has any grounds to claim.

But, as I said, once you have learned the answers to your questions
and you claim that, it's all good.


Carl Banks