Error when deleting and reimporting subpackages

[Matthew Brett]

Hi,

I recently ran into this behavior:

import sys
import apkg.subpkg
del sys.modules['apkg']
import apkg.subpkg as subpkg

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'subpkg'

where 'apkg' and 'subpkg' comprise empty __init__.py files to simplify the example.

It appears then, that importing a subpackage, then deleting the containing package from sys.modules, orphans the subpackage in an unfixable state.

I ran into this because the nose testing framework does exactly this kind of thing when loading test modules, causing some very confusing errors and failures.

Is this behavior expected?

Best,

Matthew

 

[Stephen Hansen]

Hi,

I recently ran into this behavior:

import sys
import apkg.subpkg
del sys.modules['apkg']
import apkg.subpkg as subpkg

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'subpkg'

where 'apkg' and 'subpkg' comprise empty __init__.py files to simplify the example.

It appears then, that importing a subpackage, then deleting the containing package from sys.modules, orphans the subpackage in an unfixable state.

I ran into this because the nose testing framework does exactly this kind of thing when loading test modules, causing some very confusing errorsand failures.

Is this behavior expected?

Yes. Doing an import of "apkg.subpkg" results in more then just "test1"
being cached in sys.modules, and you're removing half of that so leaving
Python in a weird state.

You also want to del sys.modules["apkg.subpkg"], then you'll be able to
re-import apkg.subpkg. I.e:

Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import sys
import test1.test2
del sys.modules['test1']
del sys.modules['test1.test2']
import test1.test2 as test2

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJOUqjIAAoJEKcbwptVWx/lIfIIAK9ASWo1kRLU0khNE3tVtGFt
mqac2tHXrs/sVs/rEMr/VLDShG9UaOQWHai9W6Jb1w/Q8bUkrLcNxGUcsRphQ7rJ
7LlX/u1gJV9QlsBVVErXIY441IcoTwfbm53WOca/QKr6aXqdvSuUfRGxtl+pVpGs
/JOj0OSbtOH+V5ydjtmED0Lvqitvzis7qL+FnLOTDp5wtV2lmVrte0ftf+XpMRJx
tLxSEkqzSu0wGwpCV8spi1rsg1hFvIl4cgAasmo8X7/d0wDrvvChlFhohXhss42j
ZekKlFtibUJKWw1PP33vb5+o5ZH5h8oxD0/PTRLaWO/jk+TA9Ik4g5F4Ywxpbi0=
=X145
-----END PGP SIGNATURE-----

 

[Matthew Brett]

Hi,

I recently ran into this behavior:

import sys
import apkg.subpkg
del sys.modules['apkg']
import apkg.subpkg as subpkg

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'subpkg'

where 'apkg' and 'subpkg' comprise empty __init__.py files to simplify the example.

It appears then, that importing a subpackage, then deleting the containing package from sys.modules, orphans the subpackage in an unfixable state.

I ran into this because the nose testing framework does exactly this kind of thing when loading test modules, causing some very confusing errors and failures.

Is this behavior expected?

Yes. Doing an import of "apkg.subpkg" results in more then just "test1"
being cached in sys.modules, and you're removing half of that so leaving
Python in a weird state.

You also want to del sys.modules["apkg.subpkg"], then you'll be able to
re-import apkg.subpkg. I.e:

Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import sys
import test1.test2
del sys.modules['test1']
del sys.modules['test1.test2']
import test1.test2 as test2

Yes, sorry, I should have mentioned that I explored these kind of variations.

I think I see that there isn't an obvious way for del sys.modules['apkg'] to know to delete or modify 'apkg.subpkg', because sys.modules is just a dict.

However, I could imagine the import machinery being able to recognize that 'apkg.subpkg' is broken, and re-import it without error.

Is that reasonable?

Best,

Matthew

 

[Matthew Brett]

Hi,

I recently ran into this behavior:

import sys
import apkg.subpkg
del sys.modules['apkg']
import apkg.subpkg as subpkg

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'subpkg'

where 'apkg' and 'subpkg' comprise empty __init__.py files to simplify the example.

It appears then, that importing a subpackage, then deleting the containing package from sys.modules, orphans the subpackage in an unfixable state.

I ran into this because the nose testing framework does exactly this kind of thing when loading test modules, causing some very confusing errors and failures.

Is this behavior expected?

Yes. Doing an import of "apkg.subpkg" results in more then just "test1"
being cached in sys.modules, and you're removing half of that so leaving
Python in a weird state.

You also want to del sys.modules["apkg.subpkg"], then you'll be able to
re-import apkg.subpkg. I.e:

Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import sys
import test1.test2
del sys.modules['test1']
del sys.modules['test1.test2']
import test1.test2 as test2

Yes, sorry, I should have mentioned that I explored these kind of variations.

I think I see that there isn't an obvious way for del sys.modules['apkg'] to know to delete or modify 'apkg.subpkg', because sys.modules is just a dict.

However, I could imagine the import machinery being able to recognize that 'apkg.subpkg' is broken, and re-import it without error.

Is that reasonable?

Best,

Matthew

 

[John Nagle]

Hi,

I recently ran into this behavior:

import sys import apkg.subpkg del sys.modules['apkg'] import
apkg.subpkg as subpkg

Traceback (most recent call last): File "<stdin>", line 1,
in<module> AttributeError: 'module' object has no attribute 'subpkg'

where 'apkg' and 'subpkg' comprise empty __init__.py files to
simplify the example.

It appears then, that importing a subpackage, then deleting the
containing package from sys.modules, orphans the subpackage in an
unfixable state.

I ran into this because the nose testing framework does exactly this
kind of thing when loading test modules, causing some very confusing
errors and failures.

Is this behavior expected?

It's undefined behavior. You're dealing with CPython implementation
semantics, not Python language semantics.

John Nagle