Problem with Dynamically unloading a module

L

lordofcode

Hi All

Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.

For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)

So initially when my program starts I have to load a default module. I
do this as follows:
############################################################
And use the methods defined in "mobile_1.py" file

Now as the application continues , I may have to use the methods
defined in "mobile_2.py" or "mobile_3.py" etc instead of the
previously loaded module,which I incorrectly try to do as below:
#########################################
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll####
del sys.modules[MODULE_name] #==> does not delete the reference in namespace


1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
"reset"?
 
L

Lie Ryan

Hi All

Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.

For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)

So initially when my program starts I have to load a default module. I
do this as follows:
##############################
##############################
And use the methods defined in "mobile_1.py" file

You probably shouldn't use "from module import *", use the __import__
function:

name = __import__('one')
name.foo() # defined in `one`
name = __import__('two')
name.bar() # defined in `two`
Now as the application continues , I may have to use the methods
defined in "mobile_2.py" or "mobile_3.py" etc instead of the
previously loaded module,which I incorrectly try to do as below:
####################
#####################
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).

that will just imports mobile_2's function into the current namespace,
possibly overriding the previously defined names from mobile_1 but it
won't delete anything that doesn't get overriden.
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll####
del sys.modules[MODULE_name] #==> does not delete the reference in namespace
1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
"reset"?

You can start a python subprocess for each new module you're loading.
That restarts the whole thing, but python loads fairly quickly and it
ensures that there won't be any leftover side effects caused by the
execution of the previous module.
 
J

Jean-Michel Pichavant

lordofcode said:
Hi All

Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.

For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)

So initially when my program starts I have to load a default module. I
do this as follows:
##############################
##############################
And use the methods defined in "mobile_1.py" file

Now as the application continues , I may have to use the methods
defined in "mobile_2.py" or "mobile_3.py" etc instead of the
previously loaded module,which I incorrectly try to do as below:
####################
#####################
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll####
del sys.modules[MODULE_name] #==> does not delete the reference in namespace


1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
"reset"?
1/ Do not use from x import *
2/ How many modules would you like to import in the end ? If this number
is bound I would suggest to use a static approach:

import mod1
import mod2
import mod3
mod4 = __import__('mod4') # equivalent to import mod4

for mod in [mod1, mod2, mod3, mod4]:
# will call func1 on each modules
print "now using module %s functions " % mod.__file__
mod.func1()
mod.func2()

With this approach it's pretty much easy to call the correct
methods/attributes, as every symbols is scopped by module namespace. You
will have less hard time debugging your code.


3/ if you really need to unload the previous module, it's a little bit
tedious.

import mod1
del mod1
sys.modules['mod1'] = None # will unload mod1 assuming mod1 was the only
reference to that module.

But believe me, you don't want to mess up with the python import mechanism.

JM
 
J

Jean-Michel Pichavant

Jean-Michel Pichavant said:
lordofcode said:
Hi All

Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.

For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)

So initially when my program starts I have to load a default module. I
do this as follows:
##############################
MODULE_name = "mobile_1"
exec "from "+MODULE_name+" import *"
##############################
And use the methods defined in "mobile_1.py" file

Now as the application continues , I may have to use the methods
defined in "mobile_2.py" or "mobile_3.py" etc instead of the
previously loaded module,which I incorrectly try to do as below:
####################
MODULE_name = "mobile_2"
exec "from "+MODULE_name+" import *"
#####################
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll####
del sys.modules[MODULE_name] #==> does not delete the reference
in namespace


1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
"reset"?
1/ Do not use from x import *
2/ How many modules would you like to import in the end ? If this
number is bound I would suggest to use a static approach:

import mod1
import mod2
import mod3
mod4 = __import__('mod4') # equivalent to import mod4

for mod in [mod1, mod2, mod3, mod4]:
# will call func1 on each modules
print "now using module %s functions " % mod.__file__
mod.func1()
mod.func2()

With this approach it's pretty much easy to call the correct
methods/attributes, as every symbols is scopped by module namespace.
You will have less hard time debugging your code.


3/ if you really need to unload the previous module, it's a little bit
tedious.

import mod1
del mod1
sys.modules['mod1'] = None # will unload mod1 assuming mod1 was the
only reference to that module.

But believe me, you don't want to mess up with the python import
mechanism.

JM
You may also want to look at the imp module,
http://docs.python.org/library/imp.html
It could help you in your task.

JM
 
S

Steven D'Aprano

3/ if you really need to unload the previous module, it's a little bit
tedious.

import mod1
del mod1
sys.modules['mod1'] = None

Assigning sys.modules[name] to None is not the same as deleting the
entry. None has special meaning to imports from packages, and for modules
it is interpreted as meaning that the module doesn't exist.
import math
del math
sys.modules['math'] = None
import math
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named math


# will unload mod1 assuming mod1 was the only
reference to that module.

Which is highly unlikely. Any classes or functions from the module will
keep the module alive.

But believe me, you don't want to mess up with the python import
mechanism.

Unless you understand how it works.
 
S

Steven D'Aprano

I need to dynamically load a module and unload it and load another
module.

Why bother unloading it? Unless you're programming for an embedded
device, it's highly unlikely that you're so strapped for memory that a
module will make any real difference.

Just stop using the module and start using another. (See below for
example.)

For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)

So initially when my program starts I have to load a default module. I
do this as follows:
##############################
##############################
And use the methods defined in "mobile_1.py" file

Now as the application continues , I may have to use the methods defined
in "mobile_2.py" or "mobile_3.py" etc instead of the previously loaded
module,


Sounds like an absolute horrible design. I recommend that you re-think
the design and do something less fragile and difficult.

The obvious way would be something like this:

# Import the first module and use it.
user_module = __import__("mobile_1")
x = user_module.function(123)
y = user_module.SomeClass(123)

# Now change modules.
user_module = __import__("mobile_2")
x = user_module.function(456)
y = user_module.SomeClass(456)


Note that:

(1) The function __import__ loads the module given a name which is only
known at runtime instead of compile time.

(2) Don't use the form "from module import name" form. That injects the
objects into the current namespace, which generally leads to more trouble
than benefit.

(3) After replacing the module, you have to get rid of the old classes
that refer to the old module, and create a new objects. Classes don't
magically change their behaviour just because the name of the class
points to something different:

.... def method(self):
.... return "Something!"
........ def method(self):
.... return "Something different"
....'Something!'


I'm betting that once you understand that behaviour, you will understand
why your approach is failing.


which I incorrectly try to do as below:
####################
#####################
The above import does not have any impact and the methods called from my
application still pertain to mobile_1.py as its still in the current
namespace(?).

The most likely problem is that you have misunderstood how Python's
object model works.

Suppose you do this:

from module_1 import MyClass
x = MyClass(1, 2, 3) # create a class instance
x.method()

The object called "x" is now an instance of MyClass from module_1. So far
so good. Suppose that you do this:

from module_2 import MyClass
x.method()

I'm guessing that you expect that this will call the method from MyClass
in module_2. But that's not what happens: the object x is still the same
object it was before, and it still calls the method from module_1. You
have to throw away that object and create a new one:

from module_2 import MyClass
x = MyClass(1, 2, 3)
x.method()

(This, BTW, is exactly the same situation as the same code above with
class C and instance.method, except an import is added to the mix.)


Now, it's not *impossible* to play tricks like this:

from module_2 import MyClass
# replace the reference to the old MyClass with the new MyClass:
x.__class__ = MyClass
x.method()

which *may* work, for some classes. But better to avoid this unless you
can control the classes yourself, which you can't.


Or find a better design that doesn't rely on dynamically replacing
modules.

I tried below code with del(), reload() etc but could not figure it out.
###Code to unload a dll####

This has nothing to do with DLLs. Python modules are Python modules, not
DLLs, and they don't work the same way.

del sys.modules[MODULE_name] #==> does not delete the reference in
namespace

Deleting the module from the cache does nothing except slow Python down
the next time you import the module.

1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?

You don't have to. You just have to understand how Python actually works,
and work *with* it instead of *against* it.

2)Are there any alternative way to do the above requirement? Currently I
am working around by restarting the whole initial setup for each new
module which is unnecessary waste.

What makes you think it's unnecessary?

Can I avoid this "reset"?

I don't know, because I don't fully understand what "restarting the whole
initial setup" actually means.
 
J

Jean-Michel Pichavant

Steven said:
Unless you understand how it works.
Let me quote the OP: 'Not an expert in Python'
I was just answering the OP question, without refering to something he
could do if he was someone else.

JM
 
J

Jean-Michel Pichavant

Steven said:
3/ if you really need to unload the previous module, it's a little bit
tedious.

import mod1
del mod1
sys.modules['mod1'] = None

Assigning sys.modules[name] to None is not the same as deleting the
entry. None has special meaning to imports from packages, and for modules
it is interpreted as meaning that the module doesn't exist.
did'nt know that.

sys.modules.pop('mod1')

should then do the trick.
Which is highly unlikely. Any classes or functions from the module will
keep the module alive.

Yep.
This is why I strongly suggested the OP to use a static approach,
unloading/reloading module in python isn't really the most obvious thing
ever done. I would even dare to say that if it was possible, someone
would have already written the module for it.

JM
 
S

Steven D'Aprano

Let me quote the OP: 'Not an expert in Python' I was just answering the
OP question, without refering to something he could do if he was someone
else.

But what he *can* do is to learn how importing works. I'm not sure it's
terribly helpful to tell somebody all the things they can't do instead of
what they can.
 
C

Carl Banks

3/ if you really need to unload the previous module, it's a little bit
tedious.
import mod1
del mod1
sys.modules['mod1'] = None

Assigning sys.modules[name] to None is not the same as deleting the
entry. None has special meaning to imports from packages, and for modules
it is interpreted as meaning that the module doesn't exist.
import math
del math
sys.modules['math'] = None
import math

Traceback (most recent call last):
# will unload mod1 assuming mod1 was the only
reference to that module.

Which is highly unlikely. Any classes or functions from the module will
keep the module alive.

Actually, they won't. Neither classes nor functions directly
reference their module; classes know their module only by name, and
functions only hold references to the module's namespace, not to the
module itself. So if any references to functions defined in the
module remain, the module dict will stick around, but the module
itself may be collected.

Unless you understand how it works.



Carl Banks
 
L

lordofcode

Hi All,

Thanks you all for your replies ,cleared quiet a few doubts about
importing modules and namespace references .
Currently am going with static importing of all modules. But it may
not be efficient in future as the number of interchangeable modules
that I have to import may run in 30-40's.(Basically this Python code
is part of larger GSM Testing project where each module represents one
model of GSM mobile phone and this number keeps growing). So changing
the design will be a last resort.
But am trying to reduce the number of objects(classes,methods etc) in
each module so importing all 30-40 modules do not impact much.

Anyway am trying out all the mentioned tricks here and get back to you
with any doubts.

Thanks again
Ajay Baragur
 
S

Steven D'Aprano

Actually, they won't. Neither classes nor functions directly reference
their module; classes know their module only by name, and functions only
hold references to the module's namespace, not to the module itself. So
if any references to functions defined in the module remain, the module
dict will stick around, but the module itself may be collected.

Hmmm... so it seems. Well spotted, thanks.

It makes sense to do it that way, as it prevents circular references
(module contains a class which contains the module which...).
 
S

Steve Holden

lordofcode said:
Hi All

Not an expert in Python, so sorry if this sounds like a silly
question.
I went through other few threads in the mailing list but they are not
helping me much.
I have run into a problem related to dynamically loading and unloading
a module.
I need to dynamically load a module and unload it and load another
module.

For example I have many files(All files in Python are modules right?)
like mobile_1.py ,mobile_2.py, mobile_3.py etc.. in my project folder
which contains classes and methods with same name but different
functionality.(am afraid I cannot change this structure as these files
are generated randomly by the user)

So initially when my program starts I have to load a default module. I
do this as follows:
############################################################
And use the methods defined in "mobile_1.py" file

Now as the application continues , I may have to use the methods
defined in "mobile_2.py" or "mobile_3.py" etc instead of the
previously loaded module,which I incorrectly try to do as below:
#########################################
The above import does not have any impact and the methods called from
my application still pertain to mobile_1.py as its still in the
current namespace(?).
I tried below code with del(), reload() etc but could not figure it
out.
###Code to unload a dll####
del sys.modules[MODULE_name] #==> does not delete the reference in namespace


1)How do I unload a module dynamically and completely remove the
references in the module so a new module with same name references can
be loaded?
2)Are there any alternative way to do the above requirement?
Currently I am working around by restarting the whole initial setup
for each new module which is unnecessary waste.Can I avoid this
"reset"?
I'd argue that you don't want to unload the old code before loading the
new module, and that what you really need is an understanding that if
mod1 and mod2 both define a function called f then you can perfectly
happily reference the two functions as

mod1.f

and

mod2.f

Indeed, if x is some object that needs processing by functions f, g and
h from module mod1 then it's perfectly valid to write

mod = mod1
result = mod.f(mod.g(mod.h(x)))

for example. So you can write code that uses the appropriate module for
each input.

It's possible, of course, that I am misunderstanding your requirements,
but I hope that helps.

regards
Steve
 
J

Jean-Michel Pichavant

Steven said:
But what he *can* do is to learn how importing works. I'm not sure it's
terribly helpful to tell somebody all the things they can't do instead of
what they can.
Hacking python imports would not be in the top of the list of things I
would teach to some python newcomer, that was my point.

And to be a little more constructive, let's give the OP a litle bit of help.

How can you unload module in python ?

That I don't know, anyone knows ?

JM
 
L

Lie Ryan

Hacking python imports would not be in the top of the list of things I
would teach to some python newcomer, that was my point.

And to be a little more constructive, let's give the OP a litle bit of
help.

How can you unload module in python ?

That I don't know, anyone knows ?

Given that there is no way to ensure deconstruction of an object
instance (`del` only removes the reference and __del__ can only remove
the resources used by the instance but not the instance itself), why
would anyone expect it is possible to unload a module explicitly (which
is a deconstruction of the module)
 
A

Aahz

Thanks you all for your replies ,cleared quiet a few doubts about
importing modules and namespace references .
Currently am going with static importing of all modules. But it may
not be efficient in future as the number of interchangeable modules
that I have to import may run in 30-40's.(Basically this Python code
is part of larger GSM Testing project where each module represents one
model of GSM mobile phone and this number keeps growing). So changing
the design will be a last resort.

Worry when you have 30-40 hundreds of modules. ;-) Seriously, it's just
memory; Python's efficient dicts make lots of modules not a problem.

If/when it's time to worry, there are other techniques you can use.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top