module import search path strangeness

Discussion in 'Python' started by tow, Aug 11, 2008.

  1. tow

    tow Guest

    I have a python script (part of a django application, if it makes any
    difference) which is exhibiting the following behaviour:

    import my_module # succeeds
    imp.find_module("my_module") # fails, raising ImportError

    which is completely baffling me. According to sys.path, both should
    fail; the directory containing my_module is not in sys.path (though
    the my_module directory itself is). More puzzlingly, printing out
    my_module.__file__ gives:

    /home/tow/test/my_module/../my_module/__init__.pyc

    I don't really understand what the ".." is doing in there.

    Can someone explain what I'm missing here, it's got me stumped.

    Toby
     
    tow, Aug 11, 2008
    #1
    1. Advertising

  2. En Mon, 11 Aug 2008 19:19:19 -0300, tow <>
    escribi�:

    > I have a python script (part of a django application, if it makes any
    > difference) which is exhibiting the following behaviour:
    >
    > import my_module # succeeds
    > imp.find_module("my_module") # fails, raising ImportError
    >
    > which is completely baffling me. According to sys.path, both should
    > fail; the directory containing my_module is not in sys.path (though
    > the my_module directory itself is).


    my_module is not a module but a package, right? Else I don't understand
    the above statement.

    > More puzzlingly, printing out
    > my_module.__file__ gives:
    >
    > /home/tow/test/my_module/../my_module/__init__.pyc
    >
    > I don't really understand what the ".." is doing in there.
    >
    > Can someone explain what I'm missing here, it's got me stumped.


    Perhaps you have ".." in sys.path? And the current directory happens to be
    /home/tow/test/my_module?

    --
    Gabriel Genellina
     
    Gabriel Genellina, Aug 12, 2008
    #2
    1. Advertising

  3. tow

    tow Guest

    On Aug 12, 4:59 am, "Gabriel Genellina" <>
    wrote:
    > En Mon, 11 Aug 2008 19:19:19 -0300, tow <>  
    > escribi :
    >
    > > I have a python script (part of a django application, if it makes any
    > > difference) which is exhibiting the following behaviour:

    >
    > > import my_module # succeeds
    > > imp.find_module("my_module") # fails, raising ImportError

    >
    > > which is completely baffling me. According to sys.path, both should
    > > fail; the directory containing my_module is not in sys.path (though
    > > the my_module directory itself is).

    >
    > my_module is not a module but a package, right? Else I don't understand  
    > the above statement.


    Sorry, sloppy terminology on my part, yes, my_module is a package.

    > > More puzzlingly, printing out
    > > my_module.__file__ gives:

    >
    > > /home/tow/test/my_module/../my_module/__init__.pyc

    >
    > > I don't really understand what the ".." is doing in there.

    >
    > > Can someone explain what I'm missing here, it's got me stumped.

    >
    > Perhaps you have ".." in sys.path? And the current directory happens to be  
    > /home/tow/test/my_module?


    The current directory is actually a subdirectory of /home/tow/test/
    my_module,
    but ".." is not in sys.path (nor is anything containing "..") In any
    case, if
    it were a matter of sys.path, surely the imp.find_module call above
    should succeed?

    Basically, I had thought that import and imp.find_module used exactly
    the same
    search path, but the above example shows that at least in this
    circumstance they
    don't; import is picking up additional search paths from somewhere -
    what am I missing?

    Toby
     
    tow, Aug 12, 2008
    #3
  4. tow

    Peter Otten Guest

    tow wrote:

    >>> > I have a python script (part of a django application, if it makes any
    >> > difference) which is exhibiting the following behaviour:

    >>
    >> > import my_module # succeeds
    >> > imp.find_module("my_module") # fails, raising ImportError

    >>
    >> > which is completely baffling me. According to sys.path, both should
    >> > fail; the directory containing my_module is not in sys.path (though
    >> > the my_module directory itself is).


    > The current directory is actually a subdirectory of /home/tow/test/
    > my_module,
    > but ".." is not in sys.path (nor is anything containing "..") In any


    > Basically, I had thought that import and imp.find_module used exactly
    > the same
    > search path, but the above example shows that at least in this
    > circumstance they
    > don't; import is picking up additional search paths from somewhere -
    > what am I missing?


    Grepping through the django source finds

    ../trunk/django/core/management/__init__.py:
    sys.path.append(os.path.join(project_directory, os.pardir))

    sys.path could be changed as a side effect of an import, so I ask you to
    verify (again) that import and imp.find_module() see the same sys.path.

    assert all(".." not in p for p in sys.path)
    import my_module

    assert all(".." not in p for p in sys.path)
    imp.find_module("my_module")


    Peter
     
    Peter Otten, Aug 12, 2008
    #4
  5. tow

    tow Guest

    On Aug 12, 9:56 am, Peter Otten <> wrote:
    > tow wrote:


    > > Basically, I had thought that import and imp.find_module used exactly
    > > the same
    > > search path, but the above example shows that at least in this
    > > circumstance they
    > > don't; import is picking up additional search paths from somewhere -
    > > what am I missing?

    >
    > Grepping through the django source finds
    >
    > ./trunk/django/core/management/__init__.py:  
    > sys.path.append(os.path.join(project_directory, os.pardir))


    Hmm. It turns out that that is indeed the issue, but in a way that
    wasn't immediately obvious to me. Looking at it in more context:

    sys.path.append(os.path.join(project_directory, os.pardir))
    project_module = __import__(project_name, {}, {}, [''])
    sys.path.pop()

    sys.path is extended, the project module is imported, then the
    additional path is dropped from sys.path.
    And if I comment out those sys.path manipulations, I get the result I
    expect later on.

    What I think is happening is that in that stanza, project_module
    (which is my_module in my case) is
    imported from the altered sys.path. sys.path is then put back, but
    python now knows about my_module.

    As a result when my_module is imported again elsewhere, python already
    knows about it, so no searching
    of sys.path is done (which is good, because it wouldn't be found on
    the current sys.path), and the
    import apparently succeeds, though in fact it's effectively
    (actually?) a no-op.

    However, imp.find_module("my_module") forces a search of the sys.path,
    and thus fails.

    Or at least, that is what I surmise, given that I see

    import my_module # succeeds
    imp.find_module("my_module") # fails, raising ImportError

    So, to answer my original question, the difference in search behaviour
    between "import" and "imp.find_module" is that the former might not
    look at sys.path at all if the module has already been loaded, while
    the latter will only search on the current sys.path.

    Am I right?

    Toby
     
    tow, Aug 12, 2008
    #5
  6. tow

    Peter Otten Guest

    tow wrote:

    > On Aug 12, 9:56 am, Peter Otten <> wrote:
    >> tow wrote:

    >
    >> > Basically, I had thought that import and imp.find_module used exactly
    >> > the same
    >> > search path, but the above example shows that at least in this
    >> > circumstance they
    >> > don't; import is picking up additional search paths from somewhere -
    >> > what am I missing?

    >>
    >> Grepping through the django source finds
    >>
    >> ./trunk/django/core/management/__init__.py:
    >> sys.path.append(os.path.join(project_directory, os.pardir))

    >
    > Hmm. It turns out that that is indeed the issue, but in a way that
    > wasn't immediately obvious to me. Looking at it in more context:
    >
    > sys.path.append(os.path.join(project_directory, os.pardir))
    > project_module = __import__(project_name, {}, {}, [''])
    > sys.path.pop()


    Ouch.

    > So, to answer my original question, the difference in search behaviour
    > between "import" and "imp.find_module" is that the former might not
    > look at sys.path at all if the module has already been loaded, while
    > the latter will only search on the current sys.path.
    >
    > Am I right?


    Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
    before it falls back to the more costly alternatives.

    Peter
     
    Peter Otten, Aug 12, 2008
    #6
  7. Peter Otten <> wrote:
    >tow wrote:
    >> sys.path.append(os.path.join(project_directory, os.pardir))
    >> project_module = __import__(project_name, {}, {}, [''])
    >> sys.path.pop()

    >Ouch.


    I presume that "Ouch" is in consideration of what might happen if
    the subject of the __import__ also calls sys.path.append ...?

    --
    \S -- -- http://www.chaos.org.uk/~sion/
    "Frankly I have no feelings towards penguins one way or the other"
    -- Arthur C. Clarke
    her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
     
    Sion Arrowsmith, Aug 12, 2008
    #7
  8. tow

    tow Guest

    On Aug 12, 4:59 pm, Peter Otten <> wrote:
    > tow wrote:
    > > On Aug 12, 9:56 am, Peter Otten <> wrote:
    > >> tow wrote:

    >
    > >> > Basically, I had thought that import and imp.find_module used exactly
    > >> > the same
    > >> > search path, but the above example shows that at least in this
    > >> > circumstance they
    > >> > don't; import is picking up additional search paths from somewhere -
    > >> > what am I missing?

    >
    > >> Grepping through the django source finds

    >
    > >> ./trunk/django/core/management/__init__.py:
    > >> sys.path.append(os.path.join(project_directory, os.pardir))

    >
    > > Hmm. It turns out that that is indeed the issue, but in a way that
    > > wasn't immediately obvious to me. Looking at it in more context:

    >
    > >     sys.path.append(os.path.join(project_directory, os.pardir))
    > >     project_module = __import__(project_name, {}, {}, [''])
    > >     sys.path.pop()

    >
    > Ouch.
    >
    > > So, to answer my original question, the difference in search behaviour
    > > between "import" and "imp.find_module" is that the former might not
    > > look at sys.path at all if the module has already been loaded, while
    > > the latter will only search on the current sys.path.

    >
    > > Am I right?

    >
    > Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
    > before it falls back to the more costly alternatives.
    >
    > Peter


    Thanks, at least I understand what's going on now.

    Toby
     
    tow, Aug 12, 2008
    #8
  9. tow

    Peter Otten Guest

    Sion Arrowsmith wrote:

    > Peter Otten <> wrote:
    >>tow wrote:
    >>> sys.path.append(os.path.join(project_directory, os.pardir))
    >>> project_module = __import__(project_name, {}, {}, [''])
    >>> sys.path.pop()

    >>Ouch.

    >
    > I presume that "Ouch" is in consideration of what might happen if
    > the subject of the __import__ also calls sys.path.append ...?


    Yes ;)

    Peter
     
    Peter Otten, Aug 12, 2008
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page