Strange problem when using imp.load_module

Discussion in 'Python' started by pythoncurious@gmail.com, Apr 23, 2009.

  1. Guest

    Hi,

    I'm having problem when I'm trying to import modules using the
    imp.load_module function.
    At the end of this post there's some code I wrote to illustrate the
    problem.
    The code istself doesn't make much sense, but what I'm trying to do in
    reality is allow people to customize an application by writing a small
    python module. If I find a file in a known location with a known name,
    I'll import it and use some data or function in it.
    I ran into problems when writing unit tests for it.

    What happens when run my code is this:

    ..E
    ======================================================================
    ERROR: test_2 (__main__.Test)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "test_import.py", line 30, in test_2
    getattr(imported, 'y')
    AttributeError: 'module' object has no attribute 'y'
    ----------------------------------------------------------------------
    Ran 2 tests in 0.015s


    So it seems that test_2 fails. It does pretty much the same thing as
    test_1, but with different data.
    Changing test_2 to check for 'x' instead of 'y' makes it pass, but
    that's not the result I should get.
    So it seems that I still have the module that I imported in test_1.

    So, I'm thinking that I might be dealing with some sort of gc issue.
    I add the "time.sleep(1)", (commented in the code) and the tests pass.
    'y' is found in test_2.
    Replacing the sleep with a gc.collect() makes the test fail again, so
    garbage collection doesn't seem to help.

    I've tried it on python 2.6.1 on windows, 2.5.2 in cygwin and 2.6.1 on
    solaris with the same results.
    Could anyone explain what I am missing?

    Thanks
    /Mattias

    Here's the code:

    import unittest
    import tempfile, os, imp, time, gc

    module_file_name=os.path.join(tempfile.gettempdir(), 'my_module.py')

    def write_module(data):
    f = open(module_file_name, 'w')
    f.write(data)
    f.close()

    def import_from_file(path):
    imported_module = imp.load_source(module_file_name, path)
    return imported_module

    class Test(unittest.TestCase):
    def tearDown(self):
    os.unlink(module_file_name)

    def test_1(self):
    module_data='''x=1'''
    write_module(module_data)
    imported=import_from_file(module_file_name)
    getattr(imported, 'x')

    def test_2(self):
    # time.sleep(1)
    module_data='''y=2'''
    write_module(module_data)
    imported=import_from_file(module_file_name)
    getattr(imported, 'y')

    if __name__ == "__main__":
    unittest.main()
     
    , Apr 23, 2009
    #1
    1. Advertising

  2. writes:

    > Hi,


    Hi, I have a guess at explaining the behaviour you describe - see below.

    > I'm having problem when I'm trying to import modules using the
    > imp.load_module function.
    > At the end of this post there's some code I wrote to illustrate the
    > problem.
    > The code istself doesn't make much sense, but what I'm trying to do in
    > reality is allow people to customize an application by writing a small
    > python module. If I find a file in a known location with a known name,
    > I'll import it and use some data or function in it.
    > I ran into problems when writing unit tests for it.
    >
    > What happens when run my code is this:
    >
    > .E
    > ======================================================================
    > ERROR: test_2 (__main__.Test)
    > ----------------------------------------------------------------------
    > Traceback (most recent call last):
    > File "test_import.py", line 30, in test_2
    > getattr(imported, 'y')
    > AttributeError: 'module' object has no attribute 'y'
    > ----------------------------------------------------------------------
    > Ran 2 tests in 0.015s
    >
    >
    > So it seems that test_2 fails. It does pretty much the same thing as
    > test_1, but with different data.
    > Changing test_2 to check for 'x' instead of 'y' makes it pass, but
    > that's not the result I should get.
    > So it seems that I still have the module that I imported in test_1.
    >
    > So, I'm thinking that I might be dealing with some sort of gc issue.
    > I add the "time.sleep(1)", (commented in the code) and the tests pass.
    > 'y' is found in test_2.
    > Replacing the sleep with a gc.collect() makes the test fail again, so
    > garbage collection doesn't seem to help.
    >
    > I've tried it on python 2.6.1 on windows, 2.5.2 in cygwin and 2.6.1 on
    > solaris with the same results.
    > Could anyone explain what I am missing?
    >
    > Thanks
    > /Mattias
    >
    > Here's the code:
    >
    > import unittest
    > import tempfile, os, imp, time, gc
    >
    > module_file_name=os.path.join(tempfile.gettempdir(), 'my_module.py')
    >
    > def write_module(data):
    > f = open(module_file_name, 'w')
    > f.write(data)
    > f.close()
    >
    > def import_from_file(path):
    > imported_module = imp.load_source(module_file_name, path)
    > return imported_module
    >
    > class Test(unittest.TestCase):
    > def tearDown(self):
    > os.unlink(module_file_name)
    >
    > def test_1(self):
    > module_data='''x=1'''
    > write_module(module_data)
    > imported=import_from_file(module_file_name)


    This will compile the module and create a 'my_module.pyc' file.

    > getattr(imported, 'x')
    >
    > def test_2(self):
    > # time.sleep(1)
    > module_data='''y=2'''
    > write_module(module_data)
    > imported=import_from_file(module_file_name)


    My guess is that without the sleep(1), the imp.load_source function will
    use the compiled file 'my_module.pyc' created in test_1 instead of
    compiling the new 'my_module.py' file.

    This would be because the new 'my_module.py' file is created so soon
    after the last compilation of the module that they have the same
    timestamp, thus fooling imp.load_source into thinking that
    'my_module.py' was not modified since its last compilation.

    > getattr(imported, 'y')
    >
    > if __name__ == "__main__":
    > unittest.main()


    HTH

    --
    Arnaud
     
    Arnaud Delobelle, Apr 23, 2009
    #2
    1. Advertising

  3. Guest

    Well spotted :)
    That does seem to be the problem. Adding removal of the .pyc file will
    make the tests pass.

    I guess that python doesn't use the higher resolution timestamp
    you can get from at least Solaris when doing 'stat' on a file.

    Thanks for the help.

    /Mattias



    On Apr 23, 10:28 pm, Arnaud Delobelle <> wrote:
    >
    > My guess is that without the sleep(1), the imp.load_source function will
    > use the compiled file 'my_module.pyc' created in test_1 instead of
    > compiling the new 'my_module.py' file.
    >
    >
    > HTH
    >
    > --
    > Arnaud
     
    , Apr 24, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. David Hirschfield

    Help: Trouble with imp.load_module

    David Hirschfield, Dec 11, 2007, in forum: Python
    Replies:
    0
    Views:
    280
    David Hirschfield
    Dec 11, 2007
  2. Gabriel Genellina

    Re: Help: Trouble with imp.load_module

    Gabriel Genellina, Dec 17, 2007, in forum: Python
    Replies:
    0
    Views:
    385
    Gabriel Genellina
    Dec 17, 2007
  3. Paul Sijben
    Replies:
    1
    Views:
    386
    Christian Heimes
    Feb 1, 2008
  4. MRAB
    Replies:
    0
    Views:
    418
  5. Iuri
    Replies:
    0
    Views:
    179
Loading...

Share This Page