bug? pkgutil.walk_packages returns packages that result in anImportError

[Chris Withers]

Hi All,

From the docs of pkgutils.walk_packages:
"""
'onerror' is a function which gets called with one argument (the
name of the package which was being imported) if any exception
occurs while trying to import a package. If no onerror function is
supplied, ImportErrors are caught and ignored, while all other
exceptions are propagated, terminating the search.
"""

My expectation of this is that if onerrors is left as None, names
yielded will be importable.

However, because the yield is before the import check, you can get
packages returned that are not importable.

This feels at odds with the docs above and I think is a bug.

If the yield were dropped to befoer the import check, we wouldn't have
this problem.

what do others think?

cheers,

Chris

 

[Peter Otten]

From the docs of pkgutils.walk_packages:
"""
'onerror' is a function which gets called with one argument (the
name of the package which was being imported) if any exception
occurs while trying to import a package. If no onerror function is
supplied, ImportErrors are caught and ignored, while all other
exceptions are propagated, terminating the search.
"""

My expectation of this is that if onerrors is left as None, names
yielded will be importable.

I would infer no such promise, especially as the generator also yields
modules, and no attempt at all is made to import those.

However, because the yield is before the import check, you can get
packages returned that are not importable.

This feels at odds with the docs above and I think is a bug.

If the yield were dropped to befoer the import check, we wouldn't have
this problem.

what do others think?

I've never worked with that function; I'd like to hear more about your
usecase.

Peter

 

[Chris Withers]

I would infer no such promise, especially as the generator also yields
modules, and no attempt at all is made to import those.

Really? I thought the __import__ fired over everything?

I've never worked with that function; I'd like to hear more about your
usecase.

It's becoming a common use case for packages like martian [1] and
venutian [2] that need to scan whole packages for things. In my case,
I'm scanning for declaratively mapped SQLAlchemy models.

It would be great if this function became part of the public API and got
some more definition on what it will and won't do. When should I take
this to Python-Dev?

cheers,

Chris

[1] http://pypi.python.org/pypi/martian
[2] http://pypi.python.org/pypi/venusian

 

[Peter Otten]

Really?

Yes, and the docstring gives a hint:

"""
Note that this function must import all *packages* (NOT all
modules!) on the given path, in order to access the __path__
attribute to find submodules.
""" Emphasis *not* mine

I thought the __import__ fired over everything?

If in doubt have a look at the source code or make a little testcase.

It's becoming a common use case for packages like martian [1] and
venutian [2] that need to scan whole packages for things. In my case,
I'm scanning for declaratively mapped SQLAlchemy models.

It would be great if this function became part of the public API and got
some more definition on what it will and won't do. When should I take
this to Python-Dev?

My impression is that it was a conscious decision to import as little as
necessary. It is easy to import the modules once you have their names, but
impossible to undo an unwanted import.

If you think the approach you expected is superior you can also escalate to
python-ideas and see if you get more feedback there or make a feature
request on the bug tracker.

Peter