Basic 'import' problem

[Frantisek Fuka]

This thing keeps bugging me. It's probably some basic misunderstanding
on my part but I am stumped. Let's say I have two Python files: file.py
and file2.py. Their contents is as follows:

file.py:
---------------------
import file2
def hello():
print "Hello"
file2.hello2()

file2.py:
---------------------
import file
def hello2():
print "Hello2"
file.hello()

Now, when I run "file.py", I get the following error:

Traceback (most recent call last):
File "file.py", line 1, in ?
import file2
File "/cygdrive/t/python/file2.py", line 1, in ?
import file
File "T:\python\file.py", line 4, in ?
file2.hello2()
AttributeError: 'module' object has no attribute 'hello2'

I think I grasp why I got this error (file2.hello2() is referenced
before its definition was "compiled" by Python) but I am puzzled as to
how to do this properly. Isn't it fairly common for two source files to
reference each other in this way? I know I can solve this by putting the
import statement after the def statement, but I have the similar problem
in much larger project, with many files and many includes and I'd like
to keep all my includes at the beginning of the file.

 

[Mark McEahern]

This thing keeps bugging me. It's probably some basic misunderstanding
on my part but I am stumped. Let's say I have two Python files:
file.py and file2.py. Their contents is as follows:

You have circular references. The short answer is, "Don't do that." ;-)

Also, "file" is not a good name for a module since you clobber the
builtin file.

Cheers,

// m

 

[Diez B. Roggisch]

I think I grasp why I got this error (file2.hello2() is referenced

before its definition was "compiled" by Python) but I am puzzled as to
how to do this properly. Isn't it fairly common for two source files to
reference each other in this way? I know I can solve this by putting the
import statement after the def statement, but I have the similar problem
in much larger project, with many files and many includes and I'd like
to keep all my includes at the beginning of the file.

First of all, don't use the name "file" - your are shadowing the type file
with it.

Now to your problem: Its perfectly legal to have circular dependencies at
declaration level - thus this works in python:

def foo():
bar()

def bar():
if moon_is_in_the_third_house:
foo()

In C, you had to define a prototype of bar, so that foo can use it.

But what you try is to already execute code from file in file2 _while_
importing file2 inside file, because you have statements on the
module-level - and that can't work. And I doubt that a language exists
where thats possible.

 

[Frantisek Fuka]

First of all, don't use the name "file" - your are shadowing the type file
with it.

OK. I used that nome only in this simplified exmaple.

Now to your problem: Its perfectly legal to have circular dependencies at
declaration level - thus this works in python:

def foo():
bar()

def bar():
if moon_is_in_the_third_house:
foo()

In C, you had to define a prototype of bar, so that foo can use it.

But what you try is to already execute code from file in file2 _while_
importing file2 inside file, because you have statements on the
module-level - and that can't work. And I doubt that a language exists
where thats possible.

Yes, this seems logical. It looks like I have some fundamental problems
with grasping the philosophy of Python...

My application is started from base.py. This file imports many other
files (modules) that do different stuff. Base.py references dozens of
classes and functions in other modules. After a while, there comes a
time when some of these modules need to reference some basic function
inluded in base.py. Which means (I thought until now) that I have to
include base.py in these files. Or not? Are you saying there is no
chance of doing this? Let's forget "import" for now and please explain
to me: Is there way to create a Python application consisting of several
modules that can freely call and reference stuff in each other? From
your answer it seems that the anwer is "no".

 

[Mel Wilson]

My application is started from base.py. This file imports many other
files (modules) that do different stuff. Base.py references dozens of
classes and functions in other modules. After a while, there comes a
time when some of these modules need to reference some basic function
inluded in base.py. Which means (I thought until now) that I have to
include base.py in these files. Or not? Are you saying there is no
chance of doing this? Let's forget "import" for now and please explain
to me: Is there way to create a Python application consisting of several
modules that can freely call and reference stuff in each other? From
your answer it seems that the anwer is "no".

To evade your problem, design your modules (as much as
you can) as though they were subroutine libraries usable
from any program.

If you can't do that, maybe pass the main-level functions
as parameters to the lower-level module functions, as
"callbacks".

A really filthy, but workable, last resort would be for
main-level code to poke the necessary function into the
sub-modules namespace, e.g.

==================================
"sub1.py"
sub1_func1 ():
main_func1()


==================================
"base.py"
import sub1
def func1 ():
pass
sub1.main_func = func1


Another thought, in sub1.py, define a class with most of
the functionality. Then, in base.py, define a descendant of
that class with its own special method to perform the
main-level operation.



Good Luck. Mel.

 

[Josiah Carlson]

Yes, this seems logical. It looks like I have some fundamental problems

with grasping the philosophy of Python...

Where did you think that Python's philosophy would allow you to do
circular imports? It doesn't make sense in /any/ language.

My application is started from base.py. This file imports many other
files (modules) that do different stuff. Base.py references dozens of
classes and functions in other modules. After a while, there comes a
time when some of these modules need to reference some basic function
inluded in base.py. Which means (I thought until now) that I have to
include base.py in these files. Or not? Are you saying there is no
chance of doing this? Let's forget "import" for now and please explain
to me: Is there way to create a Python application consisting of several
modules that can freely call and reference stuff in each other? From
your answer it seems that the anwer is "no".

Of course you can, but is that really good design?

If the thread were up on google, I'd link it, a thread started with the
subject of "software design question".

What you want can be done in two ways. Quoting myself from that thread,
there's the kludge:

main = someclass()
import sys
sys.modules['external'].main = main


And there's the standard method:

import module1
....

class main:
def __init__(self, args...):
self.c1 = module1.class1(args...)
#where args... is the standard initialization for your class,
# and any additional objects/methods that c1 needs
# access to.

Pass what is needed. If you can't pass what is needed when external
module classes are initialized, then set the attribute later.

c1instance.attribute = value


- Josiah

 

[Frantisek Fuka]

If the thread were up on google, I'd link it, a thread started with the
subject of "software design question".

What you want can be done in two ways. Quoting myself from that thread,
there's the kludge:

main = someclass()
import sys
sys.modules['external'].main = main


And there's the standard method:

import module1
...

class main:
def __init__(self, args...):
self.c1 = module1.class1(args...)
#where args... is the standard initialization for your class,
# and any additional objects/methods that c1 needs
# access to.

Pass what is needed. If you can't pass what is needed when external
module classes are initialized, then set the attribute later.

c1instance.attribute = value

Thanks for that. It makes sense.

Now, if X.py imports Y.py and Y.py imports X.py, does this present any
fundamental problems? (e.g. something get initizlized twice...)

 

[John Roth]

Diez B. Roggisch wrote:

PL/1 at least.

Yes, this seems logical. It looks like I have some fundamental problems
with grasping the philosophy of Python...

My application is started from base.py. This file imports many other
files (modules) that do different stuff. Base.py references dozens of
classes and functions in other modules. After a while, there comes a
time when some of these modules need to reference some basic function
inluded in base.py. Which means (I thought until now) that I have to
include base.py in these files. Or not? Are you saying there is no
chance of doing this? Let's forget "import" for now and please explain
to me: Is there way to create a Python application consisting of several
modules that can freely call and reference stuff in each other? From
your answer it seems that the anwer is "no".

Actually, you can. There is a general principle that will
solve most of these problems, and a couple of emergency
(meaning completely non-obvious so the magic needs to
be thoroughly documented) procedures that will solve the rest.

The basic answer to this is to structure your modules in
layers as much as possible so that you avoid import loops.

The key to doing this is your class structure. The basic fact
is that your class structure cannot contain cycles. Your
module structure has to reflect your class inheritance
structure for this reason.

If you still have modules that require out of order
imports, you can solve some of these cases by putting
the out of order imports at the bottom of the modules
that need them.

If that doesn't work, then you need to do the out of order
imports in a second pass. There are two ways of doing
this.

One is to encapsulate the out of order imports in a module
level function and call that function from the driver after
all the modules have been imported; the other is simply
to slam the bindings for the out of order modules into the
modules that need them using setattr. I'd recommend
the first since it's easier to document.

John Roth

 

[Josiah Carlson]

Thanks for that. It makes sense.

Now, if X.py imports Y.py and Y.py imports X.py, does this present any
fundamental problems? (e.g. something get initizlized twice...)

No, Python is smart about that. However, realize that if you try to do
anything with X in Y before X is done being imported, then Y will have
issues.

Don't:
#in X.py
from Y import *

#in Y.py
from X import *

The above will result in Y getting a partial namespace update from X,
really only getting everything initialized before the 'from Y import *'
statement in X.py.

Won't work:
#in main.py
import X

#in X.py
import Y
def blah():
pass

#in Y.Py
import X
X.blah()


Will work:
#main.py
import X
import Y
X.runme()
Y.runme()

#in X.py
import Y
def runme():
print "calling Y.runyou"
Y.runyou()

def runyou():
print "called X.runyou"

#in Y.py
import X
def runme():
print "calling X.runyou"
X.runyou()

def runyou():
print "called Y.runyou"


Notice in the last version, we allow the entirety of the function
definitions and namespaces to be completely initialized by the time we
call anything? Yeah, that is another method.

- Josiah

 

[Diez B. Roggisch]

My application is started from base.py. This file imports many other

files (modules) that do different stuff. Base.py references dozens of
classes and functions in other modules. After a while, there comes a
time when some of these modules need to reference some basic function
inluded in base.py. Which means (I thought until now) that I have to
include base.py in these files. Or not? Are you saying there is no
chance of doing this? Let's forget "import" for now and please explain
to me: Is there way to create a Python application consisting of several
modules that can freely call and reference stuff in each other? From
your answer it seems that the anwer is "no".

Not really. I try to explain the difference without the import-stuff as you
suggested:

What works is this:

def a():
# calls a function that is defined later
b()

def b():
# calls a()
a()

Now this example would be endlessly recursive, but thats not the point right
now.

now what you try to do is this:

c()

def c():
...

Please note the difference here: The _call_ is made on the indentation-level
0, thus c() gets called _before_ it is defined! Somebody in this thread
said such things were possible with PL/1 - well, that might be, but only if
you had a two-pass-compilation scheme that ignores statements in the first
pass. But your _module_ is _executing_code_ that whilst it is beeing
imported. Thats not working. And I don't see what good thats for - of
course you can do initalization-stuff then, but this must rely _only_ on
the module itself, not something thats defined later. If your app is
created in such a way that things are executed in such a difficult way,
that clearly shows bad design - it would e.g. depend on the order of things
beeing imported, and that shouldn't make a difference. You will create a
nightmare for debugging then.

If you have code that is used by several modules, factorize it out and
import it on all occasions its used.