Is there any way to unimport a library

[Gelonida N]

I wondered whether there is any way to un-import a library, such, that
it's occupied memory and the related shared libraries are released.


My usecase is following:


success = False
try:
import lib1_version1 as lib1
import lib2_version1 as lib2
success = True
except ImportError:
pass
if not success:
try:
import lib1_version2 as lib1
import lib2_version2 as lib2
success = True
except importError:
pass
if not success:
. . .



Basically if I am not amble to import lib1_version1 AND lib2_version1,
then I wanted to make sure, that lib1_version1 does not waste any memory


At this moment this is more a thought excercise than a real issue, but I
thought that perhaps somebody encountered this kind of issue and had an
idea how to deal with such situations.

One solution, that I could imagine is running the program a first time,
detect all existing libraries and write out a config file being use
the next time it is run, such, that immediately the right libs are imported.

 

[Miki Tebeka]

del sys.modules['my-module']

However if your module imported other modules, they'll still be there.
If there are references to objects your module created, they'll still be there.

A better option IMO is to use imp.find_module and then import.

 

[Miki Tebeka]

del sys.modules['my-module']

However if your module imported other modules, they'll still be there.
If there are references to objects your module created, they'll still be there.

A better option IMO is to use imp.find_module and then import.

 

[Steven D'Aprano]

I wondered whether there is any way to un-import a library, such, that
it's occupied memory and the related shared libraries are released.

Not really. Python modules are objects, like everything else in Python,
and can only be deleted when the garbage collector is certain that
nothing else is keeping a reference to it. So long as any object,
anywhere, no matter how deep in your code or in third-party code, has a
reference to the module, you can't force it to be deleted.

You can remove it from your application's namespace:

import math
result = math.sin(1.2345)
del math


This, however, is not sufficient to free the module, as it is cached. You
can (but shouldn't!) do this as well:

import sys
del sys.modules['math']

But please don't delete modules from the cache unless (1) you really need
to, and (2) you are prepared for the consequences. "Just in case the
library uses too much memory" is not a good reason for deleting the
module.

The problem is, if you delete the module from the cache, but some other
part of your application still holds onto a reference to the module
(directly, or indirectly), the next time you import the library, Python
will not use the cached version but will create a second, independent
copy of the module in memory. Instead of saving memory, you have just
DOUBLED the amount of memory used by the library. You may also run into
bizarre, hard to debug problems due to breaking the "modules are
singletons" promise.

Trust me, this opens the door to a world of pain.

success = False
try:
import lib1_version1 as lib1
import lib2_version1 as lib2
success = True
except ImportError:
pass
if not success:
try:
import lib1_version2 as lib1
import lib2_version2 as lib2
success = True
except importError:
pass

Can you mix lib1_version1 and lib2_version2? If so, the best way of doing
this would be:

try:
import lib1_version1 as lib1
except ImportError:
try:
import lib1_version2 as lib1
except ImportError:
# Last chance. If this fails, it is a fatal error.
import lib1_version3 as lib1

And then do the same for lib2.

One solution, that I could imagine is running the program a first time,
detect all existing libraries and write out a config file being use the
next time it is run, such, that immediately the right libs are imported.

What if the libs change between one run of your program and the next?

 

[Gelonida N]

Steven, Mika,

Thanks for your answers.

It's always good to know which options exist.
It makes it easier to choose the right one depending on the situation.

I wondered whether there is any way to un-import a library, such, that
it's occupied memory and the related shared libraries are released.

You can remove it from your application's namespace:

import math
result = math.sin(1.2345)
del math


This, however, is not sufficient to free the module, as it is cached. You
can (but shouldn't!) do this as well:

import sys
del sys.modules['math']


The problem is, if you delete the module from the cache, but some other
part of your application still holds onto a reference to the module
(directly, or indirectly), the next time you import the library, Python
will not use the cached version but will create a second, independent
copy of the module in memory. Instead of saving memory, you have just
DOUBLED the amount of memory used by the library. You may also run into
bizarre, hard to debug problems due to breaking the "modules are
singletons" promise.

in my case only one module (the one with the try import statements)
would import these libraries.

Can you mix lib1_version1 and lib2_version2? If so, the best way of doing
this would be:

try:
import lib1_version1 as lib1
except ImportError:
try:
import lib1_version2 as lib1
except ImportError:
# Last chance. If this fails, it is a fatal error.
import lib1_version3 as lib1

No mixing would not be possible.

So either I need the first two libs or the second two.

What if the libs change between one run of your program and the next?

Well that's why wondered whether there is a dynamic solution.
However if I'd like to force a rescan I could always just delete the
config file.


I think I'll look at imp.find_module as Miki suggested.
In my case it should probably be enough to know whether a group of
modules exist.
Theoretically a module could exist, but fail to import but if in this
rare situations one of the libraries stays in memory it wouldn't be the
biggest issue.

I'd just like to avoid that (this is not my use case, but just a (crazy)
example) TKInter, wxWidgets, and pyQt were all loaded even if due I'd
use only one of them due to the lack of other packages

 

[Chris Angelico]

No mixing would not be possible.

So either I need the first two libs or the second two.

I wonder, can you make the first one import the second one? That
automatically defines your dependency right there, and may make things
clearer - you import lib1_version1, and if that fails, you import
lib1_version2; if either succeeds, you know the matching lib2 is
available.

ChrisA

 

[Gelonida N]

I forgot to mention, that this is at the moment more a thought
experiment, than a real need.

I wonder, can you make the first one import the second one? That
automatically defines your dependency right there, and may make things
clearer - you import lib1_version1, and if that fails, you import
lib1_version2; if either succeeds, you know the matching lib2 is
available.

At the moment I will do exactly what you suggested. I will make sure,
that always the first import fails.

But I wanted to learn more what is possible and which potential can of
worms I would open if I wanted to unimport a library of which I'm sure
that nobody is currently referencing (or refering? Not sure about my
English here) to.

 

[alex23]

I wondered whether there is any way to un-import a library, such, that
it's occupied  memory and the related shared libraries are released.

My usecase is following:

success = False
try:
    import lib1_version1 as lib1
    import lib2_version1 as lib2
    success = True
except ImportError:
    pass
if not success:
    try:
        import lib1_version2 as lib1
        import lib2_version2 as lib2
        success = True
    except importError:
        pass
if not success:
    . . .

Basically if I am not amble to import lib1_version1 AND lib2_version1,
then I wanted to make sure, that lib1_version1 does not waste any memory

A simple way would be to create packages for each version that import
the two dependencies:

/version1/__init__.py:
import lib1_version1 as lib1
import lib2_version2 as lib2

/version2/__init__.py:
import lib1_version2 as lib1
import lib2_version2 as lib2

Then create a single module to handle the importing:

/libraries.py:

__all__ = ['lib1', 'lib2', 'version']

version = None
_import_errs = []

try:
from version1 import lib1, lib2
version = 1
except ImportError as (err,):
_import_errs.append(err)

if version is None:
try:
from version2 import lib1, lib2
version = 2
except ImportError as (err,):
_import_errs.append(err)

if version is None:
_format_errs = (('v%d: %s' % (ver, err)) for ver, err in
enumerate(_import_errs, 1))
raise ImportError('Unable to import libraries: %s' %
list(_format_errs))

 

[DevPlayer]

I forgot to mention, that this is at the moment more a thought
experiment, than a real need.

At the moment I will do exactly what you suggested. I will make sure,
that always the first import fails.

But I wanted to learn more what is possible and which potential can of
worms I would open if I wanted to unimport a library of which I'm sure
that nobody is currently referencing (or refering? Not sure about my
English here) to.

Get how many references to an object:
sys.getrefcount(sys)

 

[DevPlayer]

Seems so far the common way to fully unload any import is to exit the
Python session.
Only if this is true do I offer this hackish idea:

Therefore you might wish to run an os script instead of a python
script right off.
Here is my hack at it... Something like this:

file myapp.bat
--------------
python get_availble_imports.py available_imports.log
python myapp.py available_imports.log


file get_availble_imports.py
----------------------------
find_module_names = """
os sys time
sqlite lib1_verA lib2_verA
sqlite3 lib1_verB lib2_verB
"""

# other code i'm leaving out of this forum post

def find_module_names_using_pydoc( block_string ):

'''Searchs for module names, provided in a block string,
against the resultant module names list returned from pydoc. \n
Returns a list of strings, being the intersection of module names
from both lists.'''

all_wanted_modules = parse_block_for_module_names( block_string )
# use split and drop empties

module_names_found = []

# walk_packages actually imports libraries;
# so you know the import should work.
# i call em modules; but they could be packages too

# following line can take many seconds to run
package_generator = pydoc.pkgutil.walk_packages(path=None,
prefix='', onerror=error_handler)

for package_name in package_generator:
module_loader, module_name, ispkg = package_name
if module_name in all_wanted_modules:
module_names_found.append( module_name )
print repr( module_name )

return module_names_found

found = find_module_names_using_pydoc( find_module_names )

#Then with a switch statement (if/elif) create a string with to be
#saved to the log file with what module names are in usable_mods

if 'sqlite' in found and 'lib1_verA' in found and 'lib2_verA' in
found:
save('import sqlite, lib1_verA, lib2_verA')
elif 'sqlite' in found and 'lib1_verB' in found and 'lib2_verB' in
found:
save('import sqlite3, lib1_verB, lib2_verB')
else:
raise ImportError('Necessary packages not found')


file myapp.py

 

[DevPlayer]

btw if you like processing text outside of python (say using grep or
something)

python -c "help('modules')" > all_imports.log

which you might note on windows get's processed to:
python -c "help('modules')" 1> all_imports.log
on windows from within a batch file