Strange problem when using imp.load_module

[pythoncurious]

Hi,

I'm having problem when I'm trying to import modules using the
imp.load_module function.
At the end of this post there's some code I wrote to illustrate the
problem.
The code istself doesn't make much sense, but what I'm trying to do in
reality is allow people to customize an application by writing a small
python module. If I find a file in a known location with a known name,
I'll import it and use some data or function in it.
I ran into problems when writing unit tests for it.

What happens when run my code is this:

..E
======================================================================
ERROR: test_2 (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_import.py", line 30, in test_2
getattr(imported, 'y')
AttributeError: 'module' object has no attribute 'y'
----------------------------------------------------------------------
Ran 2 tests in 0.015s


So it seems that test_2 fails. It does pretty much the same thing as
test_1, but with different data.
Changing test_2 to check for 'x' instead of 'y' makes it pass, but
that's not the result I should get.
So it seems that I still have the module that I imported in test_1.

So, I'm thinking that I might be dealing with some sort of gc issue.
I add the "time.sleep(1)", (commented in the code) and the tests pass.
'y' is found in test_2.
Replacing the sleep with a gc.collect() makes the test fail again, so
garbage collection doesn't seem to help.

I've tried it on python 2.6.1 on windows, 2.5.2 in cygwin and 2.6.1 on
solaris with the same results.
Could anyone explain what I am missing?

Thanks
/Mattias

Here's the code:

import unittest
import tempfile, os, imp, time, gc

module_file_name=os.path.join(tempfile.gettempdir(), 'my_module.py')

def write_module(data):
f = open(module_file_name, 'w')
f.write(data)
f.close()

def import_from_file(path):
imported_module = imp.load_source(module_file_name, path)
return imported_module

class Test(unittest.TestCase):
def tearDown(self):
os.unlink(module_file_name)

def test_1(self):
module_data='''x=1'''
write_module(module_data)
imported=import_from_file(module_file_name)
getattr(imported, 'x')

def test_2(self):
# time.sleep(1)
module_data='''y=2'''
write_module(module_data)
imported=import_from_file(module_file_name)
getattr(imported, 'y')

if __name__ == "__main__":
unittest.main()

 

[Arnaud Delobelle]

Hi,

Hi, I have a guess at explaining the behaviour you describe - see below.

I'm having problem when I'm trying to import modules using the
imp.load_module function.
At the end of this post there's some code I wrote to illustrate the
problem.
The code istself doesn't make much sense, but what I'm trying to do in
reality is allow people to customize an application by writing a small
python module. If I find a file in a known location with a known name,
I'll import it and use some data or function in it.
I ran into problems when writing unit tests for it.

What happens when run my code is this:

.E
======================================================================
ERROR: test_2 (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_import.py", line 30, in test_2
getattr(imported, 'y')
AttributeError: 'module' object has no attribute 'y'
----------------------------------------------------------------------
Ran 2 tests in 0.015s


So it seems that test_2 fails. It does pretty much the same thing as
test_1, but with different data.
Changing test_2 to check for 'x' instead of 'y' makes it pass, but
that's not the result I should get.
So it seems that I still have the module that I imported in test_1.

So, I'm thinking that I might be dealing with some sort of gc issue.
I add the "time.sleep(1)", (commented in the code) and the tests pass.
'y' is found in test_2.
Replacing the sleep with a gc.collect() makes the test fail again, so
garbage collection doesn't seem to help.

I've tried it on python 2.6.1 on windows, 2.5.2 in cygwin and 2.6.1 on
solaris with the same results.
Could anyone explain what I am missing?

Thanks
/Mattias

Here's the code:

import unittest
import tempfile, os, imp, time, gc

module_file_name=os.path.join(tempfile.gettempdir(), 'my_module.py')

def write_module(data):
f = open(module_file_name, 'w')
f.write(data)
f.close()

def import_from_file(path):
imported_module = imp.load_source(module_file_name, path)
return imported_module

class Test(unittest.TestCase):
def tearDown(self):
os.unlink(module_file_name)

def test_1(self):
module_data='''x=1'''
write_module(module_data)
imported=import_from_file(module_file_name)

This will compile the module and create a 'my_module.pyc' file.

getattr(imported, 'x')

def test_2(self):
# time.sleep(1)
module_data='''y=2'''
write_module(module_data)
imported=import_from_file(module_file_name)

My guess is that without the sleep(1), the imp.load_source function will
use the compiled file 'my_module.pyc' created in test_1 instead of
compiling the new 'my_module.py' file.

This would be because the new 'my_module.py' file is created so soon
after the last compilation of the module that they have the same
timestamp, thus fooling imp.load_source into thinking that
'my_module.py' was not modified since its last compilation.

getattr(imported, 'y')

if __name__ == "__main__":
unittest.main()

HTH

 

[pythoncurious]

Well spotted
That does seem to be the problem. Adding removal of the .pyc file will
make the tests pass.

I guess that python doesn't use the higher resolution timestamp
you can get from at least Solaris when doing 'stat' on a file.

Thanks for the help.

/Mattias