redundant imports

[max(01)*]

hi everybody.

suppose that code-1.py imports code-2.py and code-3.py (because it uses
names from both), and that code-2.py imports code-3.py.

if python were c, code-1.c should only *include* code-2.c, because the
latter in turns includes code-3.c.

inclusion of modules in c is a purely preprocessing textual matter
(compilation is deferred to after the fact), i guess, so that such
things are possible. import of modules in python is a different beast,
so the "redundancy" is (i think) necessary.

any comment/suggestion/idea?

bye

macs

 

[Tim Jarman]

hi everybody.

suppose that code-1.py imports code-2.py and code-3.py (because it uses
names from both), and that code-2.py imports code-3.py.

if python were c, code-1.c should only *include* code-2.c, because the
latter in turns includes code-3.c.

inclusion of modules in c is a purely preprocessing textual matter
(compilation is deferred to after the fact), i guess, so that such
things are possible. import of modules in python is a different beast,
so the "redundancy" is (i think) necessary.

any comment/suggestion/idea?

bye

macs

It's not as redundant as it looks. Once a module has been imported it goes
into sys.modules and any subsequent imports refer to that original import,
so the overhead of reading and parsing the file is only incurred once. As
you're probably aware, Python also caches compilation results in *.pyc
files; it will only compile the imported module if it changed since the
last compilation.

Check out the docs for the full skinny, in particular
http://www.python.org/doc/2.4/ref/import.html

HTH,
Tim J

 

[Peter Hansen]

hi everybody.

suppose that code-1.py imports code-2.py and code-3.py (because it uses
names from both), and that code-2.py imports code-3.py.

if python were c, code-1.c should only *include* code-2.c, because the
latter in turns includes code-3.c.

inclusion of modules in c is a purely preprocessing textual matter
(compilation is deferred to after the fact), i guess, so that such
things are possible. import of modules in python is a different beast,
so the "redundancy" is (i think) necessary.

any comment/suggestion/idea?

You're mixed up about this whole idea.

You say first that "[code-1] uses names from both",
which by definition means that it needs to import
both. (I say by definition because import is *how*
a module gets names from another module.)

Then you say that code-1 can choose not to include
code-3 because some other module is including it...
but so what? That doesn't change the fact that
code-1 needs names from code-3, and just because
code-3 is imported elsewhere is no reason to think
that code-1 magically gets its names too. Should
all modules magically see all names in all modules
which are imported, even by other modules? That
would pretty much defeat most of the value of
namespaces.

Anyway, why this concern over so-called redundant
imports? The source code itself is not parsed
and compiled all over again, and even the .pyc file
is not re-read... once any module has imported a
module any other import just retrieves a reference
to that module from the sys.modules dictionary,
which is practically a free operation.

-Peter

 

[max(01)*]

max(01)* wrote:




It's not as redundant as it looks.

that's why i used quotes ;-)

Once a module has been imported it goes
into sys.modules and any subsequent imports refer to that original import,
so the overhead of reading and parsing the file is only incurred once. As
you're probably aware, Python also caches compilation results in *.pyc
files; it will only compile the imported module if it changed since the
last compilation.

this leads me to another question. since *.pyc files are automatically
created the first time an import statement in executed on a given
module, i guess that if i ship a program with modules for use in a
directory where the user has no write privileges then i must ship the
*.pyc files along too. right?

Check out the docs for the full skinny, in particular
http://www.python.org/doc/2.4/ref/import.html

HTH,
Tim J

thanks a lot

bye

macs

 

[max(01)*]

You're mixed up about this whole idea.

that's why i am asking for advice here.

You say first that "[code-1] uses names from both",
which by definition means that it needs to import
both. (I say by definition because import is *how*
a module gets names from another module.)

Then you say that code-1 can choose not to include
code-3 because some other module is including it...
but so what? That doesn't change the fact that
code-1 needs names from code-3, and just because
code-3 is imported elsewhere is no reason to think
that code-1 magically gets its names too. Should
all modules magically see all names in all modules
which are imported, even by other modules?

that's how inclusion works in c, which is why i asked for clarification
about the differences between c-style includes and python-style imports.

now i can see that python-style import is more like the inclusion of
declarations in c (the "names"), usually shipped as *.h header files.

the real *definition* of names is python (contents of variables,
definition of functions, and so on) i guess is inserted later in the
process of execution.

That
would pretty much defeat most of the value of
namespaces.

Anyway, why this concern over so-called redundant
imports? The source code itself is not parsed
and compiled all over again, and even the .pyc file
is not re-read... once any module has imported a
module any other import just retrieves a reference
to that module from the sys.modules dictionary,
which is practically a free operation.

-Peter

my concern was motivated by a (clumsy) attempt to understand the
difference of mechanism between the approach to modular programming in a
more "traditional" language (c) and python.

thanks again for your couseling.

bye

macs

 

[Peter Hansen]

this leads me to another question. since *.pyc files are automatically
created the first time an import statement in executed on a given
module, i guess that if i ship a program with modules for use in a
directory where the user has no write privileges then i must ship the
*.pyc files along too. right?

Not required except for performance reasons. If the .pyc
files don't exist, the .py files are recompiled and the
resulting bytecode is simply held in memory and not cached
and the next startup will recompile all over again.

Note also that the main file (the one invoke from the
command line) is never cached in a .pyc...

-Peter

 

[Steve Holden]

Not required except for performance reasons. If the .pyc
files don't exist, the .py files are recompiled and the
resulting bytecode is simply held in memory and not cached
and the next startup will recompile all over again.

Note also that the main file (the one invoke from the
command line) is never cached in a .pyc...

-Peter

Also, beware that you ship the right .pyc files - they are
version-dependent, so shipping 2.3 binaries to a 2.4 user will actually
cause a small slow-down, since the interpreter will have to check the
..pyc's for the correct magic number before ignoring them.

regards
Steve

 

[max(01)*]

Not required except for performance reasons. If the .pyc
files don't exist, the .py files are recompiled and the
resulting bytecode is simply held in memory and not cached
and the next startup will recompile all over again.

Note also that the main file (the one invoke from the
command line) is never cached in a .pyc...

but the other files *are* compiled, right? so the initial question
remains unanswered: *if* they are compiled, where are they put, if the
corresponding *.py files are on a non-writeable directory?

 

[Peter Hansen]

but the other files *are* compiled, right?

Yes, definitely. I did say that.

so the initial question remains unanswered:

No it doesn't. I thought I was clear, but I can reword
it for you: the files are compiled *in-memory* and the
results are never written to disk.

> *if* they are compiled, where are they put, if the
corresponding *.py files are on a non-writeable directory?

They are not put anywhere. Compilation is a process
by which the source is converted to executable bytecodes.
This says nothing about where either the source or the
compiled result resides. The actual compilation works
on a long series of bytes in memory, and the result is
just another long series of bytes. Nothing requires that
either of these things are even stored in a file.

Not sure what else I could say to make it clearer...

(I'm not misunderstanding you am I? I get the feeling
we're not quite on the same page here.)
-Peter

 

[max(01)*]

Yes, definitely. I did say that.



No it doesn't. I thought I was clear, but I can reword
it for you: the files are compiled *in-memory* and the
results are never written to disk.




They are not put anywhere. Compilation is a process
by which the source is converted to executable bytecodes.
This says nothing about where either the source or the
compiled result resides. The actual compilation works
on a long series of bytes in memory, and the result is
just another long series of bytes. Nothing requires that
either of these things are even stored in a file.

ok, maybe it is an implementation-dependent issue after all.

i am working on a debian woody platform with the standard python2.3 package.

consider this:

[email protected]:~/tmp/import-enigma$ ll
total 8
-rw-r--r-- 1 max2 max2 36 2005-04-02 17:44 imported.py
-rw-r--r-- 1 max2 max2 33 2005-04-02 17:44 importer.py
[email protected]:~/tmp/import-enigma$ cat importer.py
import imported
imported.fun_1()
[email protected]:~/tmp/import-enigma$ cat imported.py
def fun_1():
print "I am fun_1()"
[email protected]:~/tmp/import-enigma$ python importer.py
I am fun_1()
[email protected]:~/tmp/import-enigma$ ll
total 12
-rw-r--r-- 1 max2 max2 36 2005-04-02 17:44 imported.py
-rw-r--r-- 1 max2 max2 307 2005-04-02 17:45 imported.pyc
-rw-r--r-- 1 max2 max2 33 2005-04-02 17:44 importer.py
[email protected]:~/tmp/import-enigma$

see?

bye

macs

 

[Mike Meyer]

that's why i am asking for advice here.
my concern was motivated by a (clumsy) attempt to understand the
difference of mechanism between the approach to modular programming in
a more "traditional" language (c) and python.

[Names changed to be valid python module names.]

I feel I ought to point out that you don't really *have* to import the
code_3.py in code_1.py. You can get to things code_3.py in code_1.py
as code_2.code_3.<thing>.

The semantic behavior of "include" in C is the same as "from module
import *" in python. Both cases add all the names in the included
namespace directly to the including namespace. This usage is
depreciated in Python, because it leads to problems figuring out where
a specific variable came from. In C, it creates a problem called "name
space pollution". This is the case when a file1.c gets all the symbols
for some_header.h, even though it doesn't include/need those symbols,
because some header file1.c included needed a symbol from
some_header.h. This is especially galling if the pollution collides
with some_header2.h that file1.c actually needs.

<mike

 

[Bengt Richter]

that's why i am asking for advice here.
my concern was motivated by a (clumsy) attempt to understand the
difference of mechanism between the approach to modular programming in
a more "traditional" language (c) and python.

[Names changed to be valid python module names.]

I feel I ought to point out that you don't really *have* to import the
code_3.py in code_1.py. You can get to things code_3.py in code_1.py
as code_2.code_3.<thing>.

The semantic behavior of "include" in C is the same as "from module

Bzzt ;-) It's not the same semantics!

import *" in python. Both cases add all the names in the included
namespace directly to the including namespace. This usage is

But a C #include results in processing as if the _source_ were substituted
into the including source in place of the #include line. That's not what
from module import * does, because when import executes the module's source
(happening only the first time BTW, unlike in-place #include)
it creates a module global that is distinct from the includer's global.
The import * creates local bindings to the objects visible as bindings
in the global space of the module, but the objects retain their module
global references (if any, since there doesn't have to be any).

IMO execfile('module.py') is closer to C's #include effect. Note:
----
g = 'g in global space of impex.py'
def showg(): print g

---- g in global space of impex.py

Note that showg insisted on looking for g in it's idea of global.
Now we'll bring in g and showg via execfile: g in global space of importer

Note that because execfile executed the definition of showg in the
interactive global space, it sees the interactive change of g's binding
(in what it now also showg's global space).

(Execfile inside a function or class definition needs careful control, but can be done).

depreciated in Python, because it leads to problems figuring out where
a specific variable came from. In C, it creates a problem called "name
space pollution". This is the case when a file1.c gets all the symbols
for some_header.h, even though it doesn't include/need those symbols,
because some header file1.c included needed a symbol from
some_header.h. This is especially galling if the pollution collides
with some_header2.h that file1.c actually needs.

Of course in C you could write some #ifxxx kludges to control inclusion of named things
from a given header file somewhat. But name space pollution is a pox, to be sure ;-)

Regards,
Bengt Richter

 

[Serge Orlov]

The semantic behavior of "include" in C is the same as "from module
import *" in python. Both cases add all the names in the included
namespace directly to the including namespace. This usage is
depreciated in Python ...

Did you mean discouraged? Or it's really slated for deprecation?

Serge.

 

[Peter Hansen]

ok, maybe it is an implementation-dependent issue after all.

Not really.

consider this: [snip]
-rw-r--r-- 1 max2 max2 307 2005-04-02 17:45 imported.pyc

see?

Yes, but you don't, yet.

Obviously the .pyc file is being written, so my comments
above, out of context, is wrong.

Now please go put them back in context. You asked what
would happen if the directory was not writable. That's
the context in which to interpret my claims that the
bytecode (the *result* of the compilation) is not
written to disk.

I'll try one last time, before giving up in abject
failure and letting someone else take a stab at this:
the compilation will occur every time if a .pyc file
does not exist. The interpreter will attempt to write
the results of the compilation process to disk in a
..pyc file to cache it for the next time, to avoid
having to recompile. *If* this is not possible, then
no caching takes place, no .pyc file is written, and
the next time you run the code, the compilation step
will occur all over again (note: with the results being
held in memory only while the program runs, then
discarded).

Please tell me it's clear now.

-Peter

 

[Peter Hansen]

Mike Meyer wrote:




Did you mean discouraged? Or it's really slated for deprecation?

Deprecated basically means "the use of this is discouraged",
though because it is often followed by a comment like
"and it will be removed in a future release", people
sometimes are misled into thinking "deprecate" refers
to the pending act of removal rather than the discouragement
itself.

So, yes, its use is deprecated (though I'm not sure if that's
by any official statement, or simply by widespread convention),
but no, that doesn't mean it is going to go away any time soon.

-Peter

 

[max(01)*]

ok, maybe it is an implementation-dependent issue after all.

Not really.

consider this:
[snip]

-rw-r--r-- 1 max2 max2 307 2005-04-02 17:45 imported.pyc

see?

Yes, but you don't, yet.

Obviously the .pyc file is being written, so my comments
above, out of context, is wrong.

oops!

Now please go put them back in context. You asked what
would happen if the directory was not writable. That's
the context in which to interpret my claims that the
bytecode (the *result* of the compilation) is not
written to disk.

I'll try one last time, before giving up in abject
failure and letting someone else take a stab at this:
the compilation will occur every time if a .pyc file
does not exist. The interpreter will attempt to write
the results of the compilation process to disk in a
..pyc file to cache it for the next time, to avoid
having to recompile. *If* this is not possible, then
no caching takes place, no .pyc file is written, and
the next time you run the code, the compilation step
will occur all over again (note: with the results being
held in memory only while the program runs, then
discarded).

Please tell me it's clear now.

it's not clear. it's crystalline.

thanks for your patience and help.

best regards

macs

 

[max(01)*]

that's why i am asking for advice here.
my concern was motivated by a (clumsy) attempt to understand the
difference of mechanism between the approach to modular programming in
a more "traditional" language (c) and python.

[Names changed to be valid python module names.]

I feel I ought to point out that you don't really *have* to import the
code_3.py in code_1.py. You can get to things code_3.py in code_1.py
as code_2.code_3.<thing>.

oh. it never occured to me. interesting i must say...

The semantic behavior of "include" in C is the same as "from module
import *" in python. Both cases add all the names in the included
namespace directly to the including namespace. This usage is
depreciated in Python, because it leads to problems figuring out where
a specific variable came from.

so 'import module' is to be preferred, right?

In C, it creates a problem called "name
space pollution". This is the case when a file1.c gets all the symbols
for some_header.h, even though it doesn't include/need those symbols,
because some header file1.c included needed a symbol from
some_header.h. This is especially galling if the pollution collides
with some_header2.h that file1.c actually needs.

<mike

thanks a lot, mike!