Relative Imports, why the hell is it so hard?

C

CinnamonDonkey

Hi All,

I'm fairly new to Python so I still have a lot to learn. But I'd like
to know how to correectly use relative imports.

Please, please... please! don't go off on rants about why you think
relative imports should not be used. I've got 15+ years in C++ and
relative inclusion of other sections of code has never been a problem.
As far as I am concerned what I am trying to do is perfectly
reasonable and valid.

Thank you in advance to everyone who helps solve this, because I just
don't get it.

Example:

\ App
| main.py
+--\subpack1
| | __init__.py
| | module1.py
|
+--\subpack2
| | __init__.py
| | module2.py


Module1 needs to access functionality in Module2.

#module1.py
from ..subpack2 import module2

Seems reasonable to me... but it just does not work and I was so
liking Python. :(
 
M

Maxim Khitrov

Hi All,

I'm fairly new to Python so I still have a lot to learn. But I'd like
to know how to correectly use relative imports.

Please, please... please! don't go off on rants about why you think
relative imports should not be used. I've got 15+ years in C++ and
relative inclusion of other sections of code has never been a problem.
As far as I am concerned what I am trying to do is perfectly
reasonable and valid.

Thank you in advance to everyone who helps solve this, because I just
don't get it.

Example:

\ App
|   main.py
+--\subpack1
|   |   __init__.py
|   |   module1.py
|
+--\subpack2
|   |   __init__.py
|   |   module2.py


Module1 needs to access functionality in Module2.

#module1.py
from ..subpack2 import module2

Seems reasonable to me... but it just does not work and I was so
liking Python. :(

Relative imports are perfectly fine, in my opinion. Do you have "from
__future__ import absolute_import" at the top of module1.py? Should
work fine once you add that line.

- Max
 
S

skip

However, C++ != Python. Regardless whether or not you can "make it work",
translating idioms from one language to another is often suboptimal. That
may not be the case here, but it bears keeping in mind.

You might try removing a dot from your import statement. Python is also not
Unix. Popping up one level in the package hierarchy is done with ".", not
"..".
 
C

CinnamonDonkey

Hi Guys,

Thanx for the quick responses, it is very much appreciated!

Skip, that's a good point about "C++ != Python" and I assure you I am
very much aware of that ;-).

Looking at http://www.python.org/dev/peps/pep-0328/#guido-s-decision
would suggest, unless I am completely miss-understanding the example,
that '.' refers to the current level and '..' pops up a level. First
three uses:

# Access moduleY in same level as ModuleX
from .moduleY import spam
from .moduleY import spam as ham
from . import moduleY


Just to be sure though I tried both ;):

from ..subpack2 import module1 #ValueError: Attempted relative import
beyond toplevel package
from .subpack2 import module1 #ImportError: No module named subpack2
from . import subpack2 #ImportError: cannot import name subpack2
from .. import subpack2 #ValueError: Attempted relative import beyond
toplevel package


Max, thank you for the response... I tried adding "from __future__
import absolute_import" which made no difference. I still get exactly
the same error messages. Perhaps I should have mentioned that I am
using Python 2.5, which I understand alread supports relative imports
out of the box. I'll keep this line in for now anyway though :)
Cheers!

#subpack1.module1
from __future__ import absolute_import

from .. import subpack2

def subpack1_module1_foo():
print "subpack1_module1_foo()"
call_subpack2_module1()

def call_subpack2_module1():
subpack2.module2.subpack2_module2_foo()


#subpack2.module2
def subpack2_module2_foo():
print "subpack2_module2_foo()"

#main.py
import subpack1.module1

if __name__ == "__main__":
subpack1.module1.subpack1_module1_foo()



I am starting the sandbox app with the command line: python main.py
with the current working directory in \app where main.py is located.

Shaun >8)
 
M

Maxim Khitrov

Looking at http://www.python.org/dev/peps/pep-0328/#guido-s-decision
would suggest, unless I am completely miss-understanding the example,
that '.' refers to the current level and '..' pops up a level.

That is correct, but you cannot jump beyond the parent package, which
is why your code isn't working.
Max, thank you for the response... I tried adding "from __future__
import absolute_import" which made no difference. I still get exactly
the same error messages. Perhaps I should have mentioned that I am
using Python 2.5, which I understand alread supports relative imports
out of the box. I'll keep this line in for now anyway though :)
Cheers!

Sorry, I use that line to avoid conflicts with standard modules, and
forgot that relative imports are already enabled.

Basically, the reason your code doesn't work is because you're trying
to use relative imports between two separate packages. As far as I
know, this isn't possible. What I did to get your code working was
move main.py one directory up and create an empty __init__.py under
\App. The following code should then work:

# main.py
import App.subpack1.module1

if __name__ == "__main__":
App.subpack1.module1.subpack1_module1_foo()

# App.subpack1.module1
from ..subpack2 import module2

def subpack1_module1_foo():
print "subpack1_module1_foo()"
call_subpack2_module1()

def call_subpack2_module1():
module2.subpack2_module2_foo()

# App.subpack2.module2
def subpack2_module2_foo():
print "subpack2_module2_foo()"

- Max
 
G

Gabriel Genellina

En Mon, 23 Mar 2009 12:22:21 -0300, CinnamonDonkey

Another name for relative imports is "intra-package imports". They work
*inside* a package, and you cannot go out of the package.
If App is not a package, then subpack1 and subpack2 are separate packages
and you cannot use relative imports between them. So module1 must refer to
module2 absolutely:

from subpack2 import module2
from ..subpack2 import module1 #ValueError: Attempted relative import
beyond toplevel package

See the exception message.
Max, thank you for the response... I tried adding "from __future__
import absolute_import" which made no difference. I still get exactly
the same error messages. Perhaps I should have mentioned that I am
using Python 2.5, which I understand alread supports relative imports
out of the box. I'll keep this line in for now anyway though :)

That __future__ line is not to enable relative imports (since they have
incompatible syntax, don't require anything special) but to ensure Python
interprets "normal" imports (that is, without leading dots) always as
absolute. The default behavior in 2.5 is to try *both* ways before failing.
 
C

CinnamonDonkey

My applogies if this is a silly question... but what makes something a
package? and does that mean that what I am trying to do is not
possible ?

:(
 
M

Maxim Khitrov

My applogies if this is a silly question... but what makes something a
package? and does that mean that what I am trying to do is not
possible ?

A package is a directory that has an __init__.py file. That file can
be empty, or contain some initialization code. In your original
example, subpack1 and subpack2 are packages. By adding an empty
__init__.py file under \App, I made App into a package, which allowed
me to execute "import App.subpack1.module1" in main.py.

See the following url for additional info:
http://docs.python.org/tutorial/modules.html

- Max
 
G

Gabriel Genellina

En Mon, 23 Mar 2009 13:19:51 -0300, CinnamonDonkey
My applogies if this is a silly question... but what makes something a
package?

A package is a directory with an __init__.py file [that Python is aware
of].
and does that mean that what I am trying to do is not
possible ?

You can do an "absolute" import of subpack1 and subpack2. But you cannot
import them "relatively" - not using your current configuration, but see
Maxim Khitrov post for an alternate disposition that works.
 
R

rocky

Hi Guys,

Thanx for the quick responses, it is very much appreciated!

Skip, that's a good point about "C++ != Python" and I assure you I am
very much aware of that ;-).

Looking athttp://www.python.org/dev/peps/pep-0328/#guido-s-decision
would suggest, unless I am completely miss-understanding the example,
that '.' refers to the current level and '..' pops up a level. First
three uses:

  # Access moduleY in same level as ModuleX
  from .moduleY import spam
  from .moduleY import spam as ham
  from . import moduleY

Just to be sure though I tried both ;):

 from ..subpack2 import module1 #ValueError: Attempted relative import
beyond toplevel package
 from .subpack2 import module1 #ImportError: No module named subpack2
 from . import subpack2 #ImportError: cannot import name subpack2
 from .. import subpack2 #ValueError: Attempted relative import beyond
toplevel package

Max, thank you for the response... I tried adding "from __future__
import absolute_import" which made no difference. I still get exactly
the same error messages. Perhaps I should have mentioned that I am
using Python 2.5, which I understand alread supports relative imports
out of the box. I'll keep this line in for now anyway though :)
Cheers!

I wrote http://code.google.com/p/pyimport-relative so I could have a
simple and more reliable way to specify a *file* relative import in
Python 2.5 (and it works with 2.6 as well).

I'm not sure it's for everyone, or that it is flawless. Actually, it's
not. But I use it for in pydbgr (http://code.google.com/p/pydbgr) so
see that for examples of how to use. It allows me to run the
development code out of the source tree while having possibly a
different version installed as an egg.
 
B

bearophileHUGS

CinnamonDonkey:
what makes something a package?

If you don't know what a package is, then maybe you don't need
packages.

In your project is it possible to avoid using packages and just use
modules in the same directory?

Bye,
bearophile
 
C

CinnamonDonkey

Thanx Max - your explanation sorted it :), and a big thank you to
everyone else also!

From the various posts, Python considers any directory containing the
__init__.py file to be a package. The top level package is the highest
directory (closest to root) with a __init__.py file.

Inter-package communication is not allowed unless the packages
themselves are contained by a parent package.

How does this relate to the site-packages folder? Is it a top level
package for all installed packages?

Let's say I have installed the "Trac" system which uses "Genshi", they
are both packages. They are also both installed at the same level and
I know "Trac" uses "Genshi" to work. \Python25\Lib\site-packages does
not contain a __init__.py file so it is not a package (i.e. not a
parent package to "Trac" and "Genshi") :0.


-=- bearophile -=-

Hi Bearophile,

Thanx for taking the time to post a response but I am afraid I feel
the need to point out that it is exactly this kind of response that I
find un-helpful. It is neither constructive nor educational.

It's a bit like saying "If you don't know what a function is, then
maybe you don't need it. ... have you tried having a single block of
code?"

The point of people coming to these forums is to LEARN and share
knowledge. Perhaps it's not the best solution for me right now but
without trying it I won't know when or how to apply it as a solution.

By the way, my project has about 50 files (modules) in it with a lot
of shared code that could be used across other projects... seems as
good a reason as any to try packages out ;-)

Thanx anyway :)
 
M

Maxim Khitrov

Thanx Max - your explanation sorted it :), and a big thank you to
everyone else also!

__init__.py file to be a package. The top level package is the highest
directory (closest to root) with a __init__.py file.

Inter-package communication is not allowed unless the packages
themselves are contained by a parent package.

How does this relate to the site-packages folder? Is it a top level
package for all installed packages?

Let's say I have installed the "Trac" system which uses "Genshi", they
are both packages. They are also both installed at the same level and
I know "Trac" uses "Genshi" to work. \Python25\Lib\site-packages does
not contain a __init__.py file so it is not a package (i.e. not a
parent package to "Trac" and "Genshi") :0.

Trac does not use relative imports to access genshi. When relative
imports are not used, python goes through sys.path list to find
modules (with a small exception made when absolute_imports are not
enabled, but that should be default in 2.7). The site-packages
directory is added to sys.path, so when trac executes something like
"from genshi import some_module", python will look in site-packages,
among other directories, for a directory called "genshi" that contains
an __init__.py file.

When you execute a script, the directory of that script is
automatically added to sys.path, so with your example you could have
used absolute imports between subpack1 and subpack2, with the \App
directory performing the same function as site-packages (Gabriel's
suggestion). This is for your original version of the code when
main.py was under App.

Once you moved main.py outside of \App, running "import sybpack2"
would no longer work. You can, however, append directories to
sys.path, so by doing the following in main.py you could again allow
non-relative imports between subpack1 and subpack2:

import os
import sys

sys.path.append(os.path.realpath('App'))

- Max
 
R

R. David Murray

Top posting corrected for clarity.

CinnamonDonkey said:
Hi Bearophile,

Thanx for taking the time to post a response but I am afraid I feel
the need to point out that it is exactly this kind of response that I
find un-helpful. It is neither constructive nor educational.

It's a bit like saying "If you don't know what a function is, then
maybe you don't need it. ... have you tried having a single block of
code?"

The point of people coming to these forums is to LEARN and share
knowledge. Perhaps it's not the best solution for me right now but
without trying it I won't know when or how to apply it as a solution.

By the way, my project has about 50 files (modules) in it with a lot
of shared code that could be used across other projects... seems as
good a reason as any to try packages out ;-)

Thanx anyway :)

I think bearophile could have left out the first sentence, but otherwise
his question is perfectly sensible. If you have a bunch of independent
modules, then you don't need to put them in packages. Your example
only showed one module file in each package...I understand now that was
just for simplicity of the example, but we had no way of knowing that.
We've had newbies come in and think they _need_ to put a module file into
a subpackage even when they'd only have one module file per subdirectory
and they don't really know what a package is...thus bearophile's (perhaps
poorly phrased) question.

Now that you know what packages are and what the restrictions on relative
imports are, and you've told us that you have '50 modules' with a lot
of shared code that 'could be used across other projects', perhaps you
see why relative imports are generally discouraged. If you use only
relative imports, the code _can't_ be shared across multiple projects,
because all that project code would have to be in one monster package,
and not be separate projects at all.

So now you'll know better where it makes Pythonic (as opposed to C++)
sense to use it and where not.
 
B

bearophileHUGS

CinnamonDonkey:
It is neither constructive nor educational.

It's a bit like saying "If you don't know what a function is, then
maybe you don't need it. ... have you tried having a single block of
code?"

The point of people coming to these forums is to LEARN and share
knowledge. Perhaps it's not the best solution for me right now but
without trying it I won't know when or how to apply it as a solution.

By the way, my project has about 50 files (modules) in it with a lot
of shared code that could be used across other projects... seems as
good a reason as any to try packages out ;-)

I don't agree. My answer can be wrong for your situation, but I can't
know much about you from the start, and for most people my answer was
the right one.

When a student asks me how to improve his/her usage of metaclasses I
usually answer that metaclasses aren't required to solve that small
problem.

Generally in engineering the simplest solution is the one you have to
try first (and often second and third), and only later, if practical
experience shows the simple solution doesn't work, you try a more
complex solution.

So I have suggested you a simple solution by "default". If later you
see that you have many modules and you really need packages, then it's
easy to ignore my first answer.

Bye,
bearophile
 
C

CinnamonDonkey

Top responses guys! This has all helped increadibly.

Bearophile,

My applogies if I have offended you, but:

1. "I can't know much about you from the start" - Is kind of my point.
Perhaps it would be better to avoid jumping to conclusions and pre-
judging someones abilities simply because they are lacking knowledge
in a particular area.

Would it have been better if I had opened my thread with a copy of my
CV? I've got a Degree in Digital Systems Engineering (yes I am also an
Engineer)... I went on to do a Phd in AI and Robotics where I also
developed embedded systems. I bummed out on that after 3 years and
went on to work for a games company where I worked on 6 game titles
across 5 different platforms. I was a Lead Software Engineer for the
last 5 years. Before now moving on again.

2. I am also very much a follower of the K.I.S.S. approach, 9 times
out of 10 the simplest solution is often the best.

As an Engineer though I would also suggest that the best way to learn
is to try solving a problem being sure to constantly assess your
approach and then your final solution. By dismissing a possible avenue
you are dismissing a whole new path of knowledge. Is it not better to
try and fail than never try at all? Surely this is how we gain the
valuable experience we need.

By simply suggesting a "simple default" solution, I may never have
consider the alternatives nor understand why they are or are not
suitable.

3. I get your point about Students, sometimes there is such a thing as
too much knowledge or information overload. Whilst doing a PhD I had
to take labs and teach students... The approach I tried to take was
one off, "You may not need packages, why do you think you need
packages?" or "This is how packages would be used and why they would
be used... do you still think you need packages" or better still, for
a capable student, "This is how packages work, try solving your
problem and then tell me if you think it was a good solution."


Going with R. David Murray, perhaps I also jumped too my own
conclusion too quickly and for that I appologise.

Cheers,
Shaun
 
I

Istvan Albert

I'm fairly new to Python so I still have a lot to learn. But I'd like
to know how to correectly use relative imports.

Relative imports are *fundamentally* broken in python. You will soon
see that code using relative import will break if you attempt to run
the module on its own. Yes, it is mindboggling!

Why is it so you ask? It is one of those issue that would be trivial
to implement correctly (it is relative to the current file location,
duh!!!), would be tremendously useful yet for some reason it is
controversial with those who would need to implement it.

It looks like they think that the expected mode of operation would
greatly confuse the 'rest' of us. So that is the reason you end up
with a broken implementation that is worse than not having it at all.
All I can do is recommend that you avoid relative imports.

The easiest way around the issue is to create a module named
pathfix.py like the one below and import it in all of your modules.
This is the only way to fix this issue in a way that does not come
back and bite you, it is ugly, you are repeating yourself in multiple
places, but it is still better than the alternative.

-------------------

import os, sys

def path_join(*args):
return os.path.abspath(os.path.join(*args))

# adds base directory to import path to allow relative imports
__currdir = os.path.dirname( __file__ )
__basedir = path_join(__currdir, '..' )
if __basedir not in sys.path:
sys.path.append( __basedir )
 
G

Gabriel Genellina

En Tue, 24 Mar 2009 12:49:08 -0300, Istvan Albert
Relative imports are *fundamentally* broken in python. You will soon
see that code using relative import will break if you attempt to run
the module on its own. Yes, it is mindboggling!

How import works in general is hard to grasp -- the semantics are rather
complicated, partly because the import system has grown patch over patch
over the years, and because it's mostly underdocumented.
But I would not say that relative imports are "fundamentally broken" --
they're one of the best improvements to the import system!
Why is it so you ask? It is one of those issue that would be trivial
to implement correctly (it is relative to the current file location,
duh!!!), would be tremendously useful yet for some reason it is
controversial with those who would need to implement it.

A module doesn't *have* to reside in the filesystem - so in the general
case, there is no such thing as "current file location". Packages provide
such hierarchical disposition - and relative imports work with packages
only.
It looks like they think that the expected mode of operation would
greatly confuse the 'rest' of us. So that is the reason you end up
with a broken implementation that is worse than not having it at all.
All I can do is recommend that you avoid relative imports.

I'd recommend the oposite - use relative (intra-package) imports when
possible. Explicit is better than implicit - and starting with 2.7 -when
"absolute" import semantics will be enabled by default- you'll *have* to
use relative imports inside a package, or fail.
The easiest way around the issue is to create a module named
pathfix.py like the one below and import it in all of your modules.
This is the only way to fix this issue in a way that does not come
back and bite you, it is ugly, you are repeating yourself in multiple
places, but it is still better than the alternative.

"import it in all of your modules"?
Did you know, once a module is imported by the first time, subsequent
imports simply return the same module instance? The initialization code is
not executed again.
import os, sys

def path_join(*args):
return os.path.abspath(os.path.join(*args))

# adds base directory to import path to allow relative imports
__currdir = os.path.dirname( __file__ )
__basedir = path_join(__currdir, '..' )
if __basedir not in sys.path:
sys.path.append( __basedir )

I don't understand, how is this supposed to help relative imports?
Bindly inserting directories into sys.path can easily confuse the import
system (and you!)
 
G

Gabriel Genellina

En Tue, 24 Mar 2009 09:01:01 -0300, R. David Murray
I think bearophile could have left out the first sentence, but otherwise
his question is perfectly sensible. If you have a bunch of independent
modules, then you don't need to put them in packages. Your example
only showed one module file in each package...I understand now that was
just for simplicity of the example, but we had no way of knowing that.

Especially, since people coming from Java:

- think that they *have* to use a package
- think that they *have* to put a single class per file

Neither is true in Python. And from the example alone, one could infer the
OP was making such mistakes. Glad to see it was not the case.
 
I

Istvan Albert

Did you know, once a module is imported by the first time

yeah yeah, could we not get sidetracked with details that are not
relevant? what it obviously means is to import it in all of your
modules that need to access to relative paths
I don't understand, how is this supposed to help relative imports?

That's only because you have not had to deal with the problem that it
solves.
If you need to have a module that can do both:

1. access relative paths (even other packages)
2. be executable on its own (for example a modules may execute its own
doctests when running them directly)

this is the only way to achieve it.
I'd recommend the oposite - use relative (intra-package) imports when possible.

Does it not bother you that a module that uses relative imports cannot
be run on its own anymore? To me that is irritating because it forces
me to change a habit (running the doctests when the module is
executed) that I consider a very good practice. It is extremely handy
to be writing a module, press a key and the module is executed and I
can see the tests results. The relative import takes away this from
me. Like I said, it is irritating.
Bindly inserting directories into sys.path can easily confuse the import systemn

confuse the import system? what the heck does that mean? You either
have a path in the sys.path or not. FWIW it is far cleaner than doing
a relative import that does not work correctly.

Istvan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top