Importing modules

[e4me4m]

Coming from a scripting background where we used to write everything
into one script, I'm now going modular with Python. I place related
functions in one module, and other functions in other modules.

This all works OK, but I'm a bit confused about importing modules from
the standard library. For example, say 3 of my 12 modules use os and
os.path. Right now, I have import statements in all 3 modules that
import both os and os.path. This seems wrong to me.

I read the docs and understand that before loading a module that
Python is smart enough to look and see if it's already loaded, so
there is no performance issue (it would seem), but I was thinking
there is probably a way to load all the modules I need in some orderly
fashion without having these multiple import statements in my separate
modules.

Could someone point me to some docs that explain the Python way of
loading modules when breaking old, big (everything in one script) into
more manageable modular scripts?

Thanks,

e4m

 

[Diez B. Roggisch]

Coming from a scripting background where we used to write everything
into one script, I'm now going modular with Python. I place related
functions in one module, and other functions in other modules.

This all works OK, but I'm a bit confused about importing modules from
the standard library. For example, say 3 of my 12 modules use os and
os.path. Right now, I have import statements in all 3 modules that
import both os and os.path. This seems wrong to me.

I read the docs and understand that before loading a module that
Python is smart enough to look and see if it's already loaded, so
there is no performance issue (it would seem), but I was thinking
there is probably a way to load all the modules I need in some orderly
fashion without having these multiple import statements in my separate
modules.

Could someone point me to some docs that explain the Python way of
loading modules when breaking old, big (everything in one script) into
more manageable modular scripts?

There is no way around the multiple imports. Or, at least not a clean way -
you *could* stuff things into __builtin__ in one module, effectively making
it available globally.

But this is a hack at best, and certainly not the way to go.

The thing about modularization is that you want single pieces of code that
can be read and understood and extended by themselves (even if embedded and
useful in - or only in - a larger context).

So the seemingly unnecessary imports serve documentation purposes - what is
actually needed by the current module.

Diez

 

[Steve Holden]

Coming from a scripting background where we used to write everything
into one script, I'm now going modular with Python. I place related
functions in one module, and other functions in other modules.

This all works OK, but I'm a bit confused about importing modules from
the standard library. For example, say 3 of my 12 modules use os and
os.path. Right now, I have import statements in all 3 modules that
import both os and os.path. This seems wrong to me.

It isn't.

I read the docs and understand that before loading a module that
Python is smart enough to look and see if it's already loaded, so
there is no performance issue (it would seem), but I was thinking
there is probably a way to load all the modules I need in some orderly
fashion without having these multiple import statements in my separate
modules.

Think of the import statement as giving access to a particular
namespace, with the side-effect of loading the code if it's the first
import.

Could someone point me to some docs that explain the Python way of
loading modules when breaking old, big (everything in one script) into
more manageable modular scripts?

Import each module into every other module that requires its functionality.

regards
Steve

 

[Paul McGuire]

....and don't worry about a possible performance issue of importing os
(or any other module) multiple times - the Python import manager is
smart enough to recognize previously imported modules, and wont import
them again.

If a module uses the os module, then it should import it - that's just
it.

Another consideration might be that you are breaking up your own
program modules too much. For instance, if I had a program in which I
were importing urllib in lots of modules, it might indicate that I
still have some regrouping to do, and that I could probably gather all
of my urllib dependent code into a single place.

-- Paul

 

[Mel]

(e-mail address removed) wrote: [ ... ]

Could someone point me to some docs that explain the Python way of
loading modules when breaking old, big (everything in one script) into
more manageable modular scripts?

Import each module into every other module that requires its
functionality.

True. The O.P. can note one of the miracles of Python: (for an example,
I'll pick a library module at random .. image) One can write a module --
myimage.py, say, that imports image, and has some functions that return
objects based on calls to image functions.

A program that imports myimage can use these objects in all possible ways
and not have to import image. It doesn't need to know anything about just
where myimage gets its stuff. Importing only strictly needs to be one
level deep. (Of course, the programmer needs to know what all those
possible ways are. The programmer may have to have read the docs for the
image module ... )

Mel.

 

[Mudcat]

This is something I've wondered about for a while. I know that
theoretically Python is supposed to auto-recognize duplicate imports;
however I've run into problems in the past if I didn't arrange the
imports in a certain way across multiple files. As a result, I worry
about conflicts that arise because something has been imported twice.
So...I'm not sure if Python *always* gets this correct.

Also, I understand what you're saying about the organization of files
based on modules and maybe regrouping based on use. However I like the
organization of my files to be a grouping of logical components in my
application. This makes it easy to keep things separated and keeps
files from getting to long (which takes longer to program because
you're always bouncing up and down large files). As a result, if I
have to worry about grouping by shared modules then it makes that more
difficult.

I think it would great to have access to a file (like the __init__.py
file for packages) which all the files in the same directory would
have access to for common imports. That way you could push out the
repeated imports and also clean up the look a little bit as well.

 

[Marco Mariani]

This is something I've wondered about for a while. I know that
theoretically Python is supposed to auto-recognize duplicate imports;
however I've run into problems in the past if I didn't arrange the
imports in a certain way across multiple files.

I think you've probably had issues with circular imports (i.e. mutual
dependencies), unless you can precisely remember what you were doing and
what went wrong.

As a result, I worry about conflicts that arise because something has been imported twice.
So...I'm not sure if Python *always* gets this correct.

It doesn't import twice, and never did.

Also, I understand what you're saying about the organization of files
based on modules and maybe regrouping based on use. However I like the
organization of my files to be a grouping of logical components in my
application.

I can make up three or four different logical groupings in my
applications... so what is 'logical' could not be the same for everyone,
or from every point of view.

 

[alex23]

This is something I've wondered about for a while. I know that
theoretically Python is supposed to auto-recognize duplicate imports;
however I've run into problems in the past if I didn't arrange the
imports in a certain way across multiple files. As a result, I worry
about conflicts that arise because something has been imported twice.
So...I'm not sure if Python *always* gets this correct.

Python -will- only import something once. Every time you import a
module, Python looks up its module dictionary to see if it contains a
module of that name. If not, it imports it, adds a new module object
and binds the name to that object in the calling namespace. If it does
exist, it retrieves the module object from the dict, and then binds
the name to that object in the calling namespace.

Its best to view each Python module as a completely independent
namespace. If you need to use something from another module within
that namespace, you need to import it. If I have a function 'f' in
module 'm' that uses 'math.pow', I can't just import 'f' into a module
that has imported 'math', I need to import 'math' into 'm'. Regardless
of where the function (or class) ends up being used, where it is
defined -must- have all of the imports that function needs to run.

If you're having to set up your imports in a specific order, odds are
you have either a circular dependency or are overusing 'from <module>
import *'. You should -never- (IMO) do something like 'from <some
library> import x, y, z' in module 'a' and then 'from a import *' in
module 'b'. If 'b' uses x, y & z, it should import them itself. If you
-must- use 'import *', specify exactly what should be exported via
'__all__'...in most cases, it should be restricted to only objects
defined within that module.

I think it would great to have access to a file (like the __init__.py
file for packages) which all the files in the same directory would
have access to for common imports. That way you could push out the
repeated imports and also clean up the look a little bit as well.

Well, you can always stick those imports into a 'common.py' and do
'from common import *' in each file that uses them. But doing so can
be a pain to maintain and debug for anything more than the most simple
of applications. Being able to see a module's dependencies clearly
listed at the top of the file is exceptionally handy.

 

[Mudcat]

I think you've probably had issues with circular imports (i.e. mutual

dependencies), unless you can precisely remember what you were doing and
what went wrong.

That's possible, but circular imports become more of a hazard if you
have to import in several locations. Unify that to one file, and those
problems are much easier to avoid.

And I don't remember exactly what the problem was, but I think it had
to do with calling Tkinter in two different files. Somehow I was
getting an error in one of the files and removing Tkinter from
importing in one of the files solved it.

I can make up three or four different logical groupings in my
applications... so what is 'logical' could not be the same for everyone,
or from every point of view.

That's not the point. The point is that multiple imports can be a
limiting factor if the module imports don't happen to align with the
model they'd like to use for their file layout.

However playing around with the files, I guess it is possible to
create a file that just does imports and then reference them all just
like you would any other extension of the namespace. I created a file
called imports and was able to access the sys module within it by
importing all from imports and calling sys. That way at least all you
have to do is import the one file each time.

 

[Mudcat]

Well, you can always stick those imports into a 'common.py' and do
'from common import *' in each file that uses them. But doing so can
be a pain to maintain and debug for anything more than the most simple
of applications. Being able to see a module's dependencies clearly
listed at the top of the file is exceptionally handy.

I x-posted the same thing but am curious why you think it's such a
problem to maintain. Having dependencies at the top is handy only
because it's usually quicker to get to the top of a file than it is to
open up another one. However if you have your common file open in your
IDE you can have it open in a split window where it's viewable all the
time. As long as you don't go crazy with "import *" in your common
file then you shouldn't get into trouble with duplicated namespaces.
As I mentioned before it should also cut down on the circular
imports.

So just preference aside, is there any problems with the actual
execution of the code if it's done this way? To me it simply compares
to putting an __init__ file in a package, except it applies to only
one directory.

 

[Aahz]

If you're having to set up your imports in a specific order, odds are
you have either a circular dependency or are overusing 'from <module>
import *'. You should -never- (IMO) do something like 'from <some
library> import x, y, z' in module 'a' and then 'from a import *' in
module 'b'. If 'b' uses x, y & z, it should import them itself. If you
-must- use 'import *', specify exactly what should be exported via
'__all__'...in most cases, it should be restricted to only objects
defined within that module.

There's one other way to run into this problem: I forget the exact
mechanics, but if you mix up absolute and relative imports for modules
inside packages, you can get namespace problems. (It's also possible
newer versions of Python don't have this issue.)

 

[rdmurray]

[attribution omitted by Mudcat]

I think you've probably had issues with circular imports (i.e. mutual
dependencies), unless you can precisely remember what you were doing and
what went wrong.

That's possible, but circular imports become more of a hazard if you
have to import in several locations. Unify that to one file, and those
problems are much easier to avoid.

Personally I've never run into circular import problems when importing
library functions. Then, again, I always use 'from x import y' (or
'y as z' for those modules that use non-unique names for things, like
os.path.split).

Where I've run into circular import problems when when _my_ modules
were circularly importing things. Which just meant that my code needed
some refactoring.

That's not the point. The point is that multiple imports can be a
limiting factor if the module imports don't happen to align with the
model they'd like to use for their file layout.

I'm inclined to agree with you that organizing code around which other
modules it imports is (usually) not the optimal organization. But it is
worth _considering_, because sometimes it does reveal a useful pattern
to refactor by.

However playing around with the files, I guess it is possible to
create a file that just does imports and then reference them all just
like you would any other extension of the namespace. I created a file
called imports and was able to access the sys module within it by
importing all from imports and calling sys. That way at least all you
have to do is import the one file each time.

If you are talking about making a file and then doing from X import *,
I'd shudder a little bit looking at such code. It just doesn't feel
like the python way . Which doesn't mean you shouldn't do it, if it
works for you, it is more of a personal preference thing.

What I'd respond to better (and I almost started doing this in my
code...but it's more typing so in the end I didn't) is to have a module,
say 'lib', and just do 'import lib'. Then everything is called through
'lib': lib.os.path.split('/some/path'), etc, etc.

The difference is that that is explicit. If I'm reading your code
and I see that, I look at the top and I see the import for 'lib', and I
know to look for a lib module. If you use the from X import * pattern,
I lose that ability. Sure, I can look at the top, and see the 'from X
import *', and if that's the only 'from *' import I can guess that the
functions not defined in the file come from that one...but all of that
takes more brain cycles. It just doesn't feel as clean

But, in the end, it is mostly personal preference. I'm sure there are
Pythonistas out there whose lips will curl when seeing
'from os.path import split as pathsplit, join as pathjoin' in my code

--RDM