PyWart: Import resolution order

Discussion in 'Python' started by Rick Johnson, Jan 11, 2013.

  1. Rick Johnson

    Rick Johnson Guest

    Python's import resolution order is terrible.[1]

    The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:

    1. Current package or directory
    2. stdlib
    3. under the bed
    4. who cares at this point

    Look, if you want to keep you foolish imports starting from the stdlib, fine, just create a switch so we can change it to "package/directory" if we like.

    If /only/ the Python gods had placed all stdlib modules in a package named,i don't know, "lib" then none of this would be an issue. You could happilyhave a stdlib::math and a mylib::math, and when you import either module your code will reflect that path explicitly. Hmm, this last sentence reminded me of something??? Oh yes -> "Explicit is better than implicit".


    [1]: Yes, yes, i know about sys.path.insert. *puke*!
    Rick Johnson, Jan 11, 2013
    #1
    1. Advertising

  2. On Fri, Jan 11, 2013 at 5:13 PM, Rick Johnson
    <> wrote:
    > The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:
    >
    > 1. Current package or directory
    > 2. stdlib
    > 3. under the bed
    > 4. who cares at this point


    Why is it better to import from the current directory first? Windows
    has that policy for executable commands; Unix, on the other hand,
    requires that you put an explicit path for anything that isn't in the
    standard search path. Which of these options is the more likely to
    produce security holes and/or unexpected behaviour?

    Welcome back to the list, Rick. Got any demonstrable code for Python 4000 yet?

    ChrisA
    Chris Angelico, Jan 11, 2013
    #2
    1. Advertising

  3. Rick Johnson

    Terry Reedy Guest

    On 1/11/2013 1:13 AM, Rick Johnson wrote:
    >
    > Python's import resolution order is terrible.[1]
    >
    > The fact that Python looks in the stdlib _first_ is not a good idea.


    And the fact is that it does not do so. The order depends on sys.path,
    and '' is the first entry.

    > It would seem more intuitive for a custom "math" module (living in
    > the current directory) to /override/ the stlib "math" module.


    Try it.

    This is a nuisance though, when not intended. Someone writes a
    random.py, and a year later in the same directory, an unrelated
    hopscotch.py, which tries to import random.exponential. The import fails
    and they post here, having forgotten about their own random.py, which
    does not have such a function. Posts like this happen a few times a year.

    --
    Terry Jan Reedy
    Terry Reedy, Jan 11, 2013
    #3
  4. On 01/10/2013 11:13 PM, Rick Johnson wrote:
    >
    > Python's import resolution order is terrible.[1]
    >
    > The fact that Python looks in the stdlib _first_ is not a good idea.


    Whether or not the default behavior is desirable or not, sys.path is set
    by default to look in the current directory first on any Python
    distribution I have looked at. As Terry says, this fact causes a lot of
    problems for newbies who accidentally override standard library modules
    with their own python files and chaos ensues.

    If your python installation does not search the current directory first,
    then you must have changed sys.path.
    Michael Torrie, Jan 11, 2013
    #4
  5. Rick Johnson

    Rick Johnson Guest

    On Friday, January 11, 2013 7:35:37 AM UTC-6, Terry Reedy wrote:
    > On 1/11/2013 1:13 AM, Rick Johnson wrote:
    > > The fact that Python looks in the stdlib _first_ is not a good idea.

    >
    > And the fact is that it does not do so. The order depends on sys.path,
    > and '' is the first entry.
    >
    > > It would seem more intuitive for a custom "math" module (living in
    > > the current directory) to /override/ the stlib "math" module.

    >
    > This is a nuisance though, when not intended. Someone writes a
    > random.py, and a year later in the same directory, an unrelated
    > hopscotch.py, which tries to import random.exponential. The import fails
    > and they post here, having forgotten about their own random.py, which
    > does not have such a function. Posts like this happen a few times a year.


    That's why i also mentioned the failure of Python to wrap stdlib modules in a package. If we would protect all built-in modules by placing them in a package (lib or py) then this problem would never happen.

    Of course many people will piss and moan about the extra typing. I say, you have a choice: a few extra chars or multitudes of extra headaches -- I choose the first option.

    Since more time is spent /maintaining/ code bases than /writing/ them, the explicit path is always the correct path to choose. Anyone who says otherwise is either careless or selfish (aka: seeking job security).
    Rick Johnson, Jan 12, 2013
    #5
  6. Rick Johnson

    Rick Johnson Guest

    On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
    > Why is it better to import from the current directory first?


    Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


    But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

    > Windows
    > has that policy for executable commands; Unix, on the other hand,
    > requires that you put an explicit path for anything that isn't in the
    > standard search path. Which of these options is the more likely to
    > produce security holes and/or unexpected behaviour?


    I prefer the latter of course :).

    I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

    For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

    So the conclusion is:

    * We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

    * if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

    By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

    > Welcome back to the list, Rick. Got any demonstrable code
    > for Python 4000 yet?


    I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.
    Rick Johnson, Jan 12, 2013
    #6
  7. Rick Johnson

    Rick Johnson Guest

    On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
    > Why is it better to import from the current directory first?


    Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


    But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

    > Windows
    > has that policy for executable commands; Unix, on the other hand,
    > requires that you put an explicit path for anything that isn't in the
    > standard search path. Which of these options is the more likely to
    > produce security holes and/or unexpected behaviour?


    I prefer the latter of course :).

    I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

    For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

    So the conclusion is:

    * We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

    * if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

    By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

    > Welcome back to the list, Rick. Got any demonstrable code
    > for Python 4000 yet?


    I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.
    Rick Johnson, Jan 12, 2013
    #7
  8. On Sat, Jan 12, 2013 at 4:28 PM, Rick Johnson
    <> wrote:
    > On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
    >> Welcome back to the list, Rick. Got any demonstrable code
    >> for Python 4000 yet?

    >
    > I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.


    So all I have to do is paper my programming world and I win?

    Sorry, can't hang around arguing about module search paths and your
    recommendations to add piles of syntactic salt. Gotta finish building
    and playing with today's Linux kernel (3.2.7, it's looking good so
    far).

    ChrisA
    Chris Angelico, Jan 12, 2013
    #8
  9. Rick Johnson

    Ian Kelly Guest

    On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
    <> wrote:
    > On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
    >> Why is it better to import from the current directory first?

    >
    > Opps. I was not explicit enough with my explanation :). I meant, "look inthe current directory FIRST when in a package". Since many times (most alltimes) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


    And again, in Python 2.x this is already the case. When importing in
    a package, it tries to do a relative import before it even looks at
    sys.path.

    > I think if python where *strict* about full paths for non-builtins, then we would be in a better place.


    And again, in Python 3, where implicit relative imports have been
    removed from the language, it already is strict about using full
    paths. You can still do relative imports, but you have to be explicit
    about them.

    > For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".


    What if I create a package named "math"? Does that also automatically
    get renamed to "lib.math"? How is it decided what package names are
    proper; is it just because it happens to clash with a stdlib name that
    the package gets magically renamed?

    What if I create a package, and then later a module with the same name
    happens to be added to the stdlib? My program that uses the package
    just breaks because it no longer imports the correct thing?

    > Damn i am full of good ideas!


    Your ideas might be better if you first spent some time gaining a
    better understanding of how the language works as is.
    Ian Kelly, Jan 12, 2013
    #9
  10. Rick Johnson

    alex23 Guest

    On 12 Jan, 14:50, Rick Johnson <> wrote:
    > Of course many people will piss and moan about the extra typing.


    You just ignored the fact that your original claim was incorrect and
    kept going on with your rant anyway.

    > Since more time is spent /maintaining/ code bases than /writing/ them


    In your case, more time is actually spent on insisting _other_ people
    maintain code bases.

    Everyone: PLEASE STOP FEEDING THE TROLL
    alex23, Jan 12, 2013
    #10
  11. Rick Johnson

    alex23 Guest

    On Jan 12, 3:28 pm, Rick Johnson <> wrote:
    > I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.


    I am so confidant that this will never happen that if you _do_ ever
    produce _anything_ that even remotely resembles your claims, I pledge
    to provide you with enough funding to continue full-time development
    on it for 5 years, let's say 5 years @ US$50k per year.

    However, one condition for acceptance will be that you never post here
    again.
    alex23, Jan 13, 2013
    #11
  12. Ianæ–¼ 2013å¹´1月12日星期六UTC+8下åˆ3時36分43秒寫é“:
    > On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
    >
    > <> wrote:
    >
    > > On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:

    >
    > >> Why is it better to import from the current directory first?

    >
    > >

    >
    > > Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have thesame name as a stdlib module, then looking in the package FIRST makes sense.

    >
    >
    >
    > And again, in Python 2.x this is already the case. When importing in
    >
    > a package, it tries to do a relative import before it even looks at
    >
    > sys.path.
    >
    >
    >
    > > I think if python where *strict* about full paths for non-builtins, then we would be in a better place.

    >
    >
    >
    > And again, in Python 3, where implicit relative imports have been
    >
    > removed from the language, it already is strict about using full
    >
    > paths. You can still do relative imports, but you have to be explicit
    >
    > about them.
    >
    >
    >
    > > For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

    >
    >
    >
    > What if I create a package named "math"? Does that also automatically
    >
    > get renamed to "lib.math"? How is it decided what package names are
    >
    > proper; is it just because it happens to clash with a stdlib name that
    >
    > the package gets magically renamed?
    >
    >
    >
    > What if I create a package, and then later a module with the same name
    >
    > happens to be added to the stdlib? My program that uses the package
    >
    > just breaks because it no longer imports the correct thing?
    >
    >
    >
    > > Damn i am full of good ideas!

    >
    >
    >
    > Your ideas might be better if you first spent some time gaining a
    >
    > better understanding of how the language works as is.


    OK, I think to develop a GUI with auto-code
    translations in an IDE with python as the CAD/CAM scripting language can be helpful.

    But usually this kind of sotware projects is in the
    commercial part.
    88888 Dihedral, Jan 13, 2013
    #12
  13. Ianæ–¼ 2013å¹´1月12日星期六UTC+8下åˆ3時36分43秒寫é“:
    > On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
    >
    > <> wrote:
    >
    > > On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:

    >
    > >> Why is it better to import from the current directory first?

    >
    > >

    >
    > > Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have thesame name as a stdlib module, then looking in the package FIRST makes sense.

    >
    >
    >
    > And again, in Python 2.x this is already the case. When importing in
    >
    > a package, it tries to do a relative import before it even looks at
    >
    > sys.path.
    >
    >
    >
    > > I think if python where *strict* about full paths for non-builtins, then we would be in a better place.

    >
    >
    >
    > And again, in Python 3, where implicit relative imports have been
    >
    > removed from the language, it already is strict about using full
    >
    > paths. You can still do relative imports, but you have to be explicit
    >
    > about them.
    >
    >
    >
    > > For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

    >
    >
    >
    > What if I create a package named "math"? Does that also automatically
    >
    > get renamed to "lib.math"? How is it decided what package names are
    >
    > proper; is it just because it happens to clash with a stdlib name that
    >
    > the package gets magically renamed?
    >
    >
    >
    > What if I create a package, and then later a module with the same name
    >
    > happens to be added to the stdlib? My program that uses the package
    >
    > just breaks because it no longer imports the correct thing?
    >
    >
    >
    > > Damn i am full of good ideas!

    >
    >
    >
    > Your ideas might be better if you first spent some time gaining a
    >
    > better understanding of how the language works as is.


    OK, I think to develop a GUI with auto-code
    translations in an IDE with python as the CAD/CAM scripting language can be helpful.

    But usually this kind of sotware projects is in the
    commercial part.
    88888 Dihedral, Jan 13, 2013
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andrew FPGA
    Replies:
    0
    Views:
    956
    Andrew FPGA
    Sep 26, 2005
  2. Roland Hedberg

    import order or cross import

    Roland Hedberg, Jan 3, 2007, in forum: Python
    Replies:
    5
    Views:
    345
    Gabriel Genellina
    Jan 3, 2007
  3. rantingrick
    Replies:
    20
    Views:
    461
    Terry Reedy
    Jul 23, 2011
  4. rantingrick
    Replies:
    0
    Views:
    169
    rantingrick
    Jul 22, 2011
  5. rantingrick
    Replies:
    7
    Views:
    340
    Chris Angelico
    Jul 30, 2011
Loading...

Share This Page