PyWart: Import resolution order

[Rick Johnson]

Python's import resolution order is terrible.[1]

The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:

1. Current package or directory
2. stdlib
3. under the bed
4. who cares at this point

Look, if you want to keep you foolish imports starting from the stdlib, fine, just create a switch so we can change it to "package/directory" if we like.

If /only/ the Python gods had placed all stdlib modules in a package named,i don't know, "lib" then none of this would be an issue. You could happilyhave a stdlib::math and a mylib::math, and when you import either module your code will reflect that path explicitly. Hmm, this last sentence reminded me of something??? Oh yes -> "Explicit is better than implicit".


[1]: Yes, yes, i know about sys.path.insert. *puke*!

 

[Chris Angelico]

The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:

1. Current package or directory
2. stdlib
3. under the bed
4. who cares at this point

Why is it better to import from the current directory first? Windows
has that policy for executable commands; Unix, on the other hand,
requires that you put an explicit path for anything that isn't in the
standard search path. Which of these options is the more likely to
produce security holes and/or unexpected behaviour?

Welcome back to the list, Rick. Got any demonstrable code for Python 4000 yet?

ChrisA

 

[Terry Reedy]

Python's import resolution order is terrible.[1]

The fact that Python looks in the stdlib _first_ is not a good idea.

And the fact is that it does not do so. The order depends on sys.path,
and '' is the first entry.

It would seem more intuitive for a custom "math" module (living in
the current directory) to /override/ the stlib "math" module.

Try it.

This is a nuisance though, when not intended. Someone writes a
random.py, and a year later in the same directory, an unrelated
hopscotch.py, which tries to import random.exponential. The import fails
and they post here, having forgotten about their own random.py, which
does not have such a function. Posts like this happen a few times a year.

 

[Michael Torrie]

Python's import resolution order is terrible.[1]

The fact that Python looks in the stdlib _first_ is not a good idea.

Whether or not the default behavior is desirable or not, sys.path is set
by default to look in the current directory first on any Python
distribution I have looked at. As Terry says, this fact causes a lot of
problems for newbies who accidentally override standard library modules
with their own python files and chaos ensues.

If your python installation does not search the current directory first,
then you must have changed sys.path.

 

[Rick Johnson]

And the fact is that it does not do so. The order depends on sys.path,
and '' is the first entry.


This is a nuisance though, when not intended. Someone writes a
random.py, and a year later in the same directory, an unrelated
hopscotch.py, which tries to import random.exponential. The import fails
and they post here, having forgotten about their own random.py, which
does not have such a function. Posts like this happen a few times a year.

That's why i also mentioned the failure of Python to wrap stdlib modules in a package. If we would protect all built-in modules by placing them in a package (lib or py) then this problem would never happen.

Of course many people will piss and moan about the extra typing. I say, you have a choice: a few extra chars or multitudes of extra headaches -- I choose the first option.

Since more time is spent /maintaining/ code bases than /writing/ them, the explicit path is always the correct path to choose. Anyone who says otherwise is either careless or selfish (aka: seeking job security).

 

[Rick Johnson]

Why is it better to import from the current directory first?

Opps. I was not explicit enough with my explanation . I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

Windows
has that policy for executable commands; Unix, on the other hand,
requires that you put an explicit path for anything that isn't in the
standard search path. Which of these options is the more likely to
produce security holes and/or unexpected behaviour?

I prefer the latter of course .

I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

So the conclusion is:

* We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

* if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

Welcome back to the list, Rick. Got any demonstrable code
for Python 4000 yet?

I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

 

[Rick Johnson]

Why is it better to import from the current directory first?

Opps. I was not explicit enough with my explanation . I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

Windows
has that policy for executable commands; Unix, on the other hand,
requires that you put an explicit path for anything that isn't in the
standard search path. Which of these options is the more likely to
produce security holes and/or unexpected behaviour?

I prefer the latter of course .

I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

So the conclusion is:

* We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

* if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

Welcome back to the list, Rick. Got any demonstrable code
for Python 4000 yet?

I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

 

[Chris Angelico]

I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

So all I have to do is paper my programming world and I win?

Sorry, can't hang around arguing about module search paths and your
recommendations to add piles of syntactic salt. Gotta finish building
and playing with today's Linux kernel (3.2.7, it's looking good so
far).

ChrisA

 

[Ian Kelly]

Opps. I was not explicit enough with my explanation . I meant, "look inthe current directory FIRST when in a package". Since many times (most alltimes) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.

And again, in Python 2.x this is already the case. When importing in
a package, it tries to do a relative import before it even looks at
sys.path.

I think if python where *strict* about full paths for non-builtins, then we would be in a better place.

And again, in Python 3, where implicit relative imports have been
removed from the language, it already is strict about using full
paths. You can still do relative imports, but you have to be explicit
about them.

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

What if I create a package named "math"? Does that also automatically
get renamed to "lib.math"? How is it decided what package names are
proper; is it just because it happens to clash with a stdlib name that
the package gets magically renamed?

What if I create a package, and then later a module with the same name
happens to be added to the stdlib? My program that uses the package
just breaks because it no longer imports the correct thing?

Damn i am full of good ideas!

Your ideas might be better if you first spent some time gaining a
better understanding of how the language works as is.

 

[alex23]

Of course many people will piss and moan about the extra typing.

You just ignored the fact that your original claim was incorrect and
kept going on with your rant anyway.

Since more time is spent /maintaining/ code bases than /writing/ them

In your case, more time is actually spent on insisting _other_ people
maintain code bases.

Everyone: PLEASE STOP FEEDING THE TROLL

 

[alex23]

I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

I am so confidant that this will never happen that if you _do_ ever
produce _anything_ that even remotely resembles your claims, I pledge
to provide you with enough funding to continue full-time development
on it for 5 years, let's say 5 years @ US$50k per year.

However, one condition for acceptance will be that you never post here
again.

 

[88888 Dihedral]

Ianæ–¼ 2013å¹´1月12日星期六UTC+8下åˆ3時36分43秒寫é“：

And again, in Python 2.x this is already the case. When importing in

a package, it tries to do a relative import before it even looks at

sys.path.






And again, in Python 3, where implicit relative imports have been

removed from the language, it already is strict about using full

paths. You can still do relative imports, but you have to be explicit

about them.






What if I create a package named "math"? Does that also automatically

get renamed to "lib.math"? How is it decided what package names are

proper; is it just because it happens to clash with a stdlib name that

the package gets magically renamed?



What if I create a package, and then later a module with the same name

happens to be added to the stdlib? My program that uses the package

just breaks because it no longer imports the correct thing?






Your ideas might be better if you first spent some time gaining a

better understanding of how the language works as is.

OK, I think to develop a GUI with auto-code
translations in an IDE with python as the CAD/CAM scripting language can be helpful.

But usually this kind of sotware projects is in the
commercial part.

 

[88888 Dihedral]

Ianæ–¼ 2013å¹´1月12日星期六UTC+8下åˆ3時36分43秒寫é“：

And again, in Python 2.x this is already the case. When importing in

a package, it tries to do a relative import before it even looks at

sys.path.






And again, in Python 3, where implicit relative imports have been

removed from the language, it already is strict about using full

paths. You can still do relative imports, but you have to be explicit

about them.






What if I create a package named "math"? Does that also automatically

get renamed to "lib.math"? How is it decided what package names are

proper; is it just because it happens to clash with a stdlib name that

the package gets magically renamed?



What if I create a package, and then later a module with the same name

happens to be added to the stdlib? My program that uses the package

just breaks because it no longer imports the correct thing?






Your ideas might be better if you first spent some time gaining a

better understanding of how the language works as is.

OK, I think to develop a GUI with auto-code
translations in an IDE with python as the CAD/CAM scripting language can be helpful.

But usually this kind of sotware projects is in the
commercial part.