reloading code and multiprocessing

Discussion in 'Python' started by andrea crotti, Jul 19, 2012.

  1. We need to be able to reload code on a live system. This live system
    has a daemon process always running but it runs many subprocesses with
    multiprocessing, and the subprocesses might have a short life...

    Now I found a way to reload the code successfully, as you can see from
    this testcase:


    def func():
    from . import a
    print(a.ret())


    class TestMultiProc(unittest.TestCase):
    def setUp(self):
    open(path.join(cur_dir, 'a.py'), 'w').write(old_a)

    def tearDown(self):
    remove(path.join(cur_dir, 'a.py'))

    def test_reloading(self):
    """Starting a new process gives a different result
    """
    p1 = Process(target=func)
    p2 = Process(target=func)
    p1.start()
    res = p1.join()
    open(path.join(cur_dir, 'a.py'), 'w').write(new_a)
    remove(path.join(cur_dir, 'a.pyc'))

    p2.start()
    res = p2.join()


    As long as I import the code in the function and make sure to remove the
    "pyc" files everything seems to work..
    Are there any possible problems which I'm not seeing in this approach or
    it's safe?

    Any other better ways otherwise?
    andrea crotti, Jul 19, 2012
    #1
    1. Advertising

  2. andrea crottiæ–¼ 2012å¹´7月19日星期四UTC+8下åˆ6時15分11秒寫é“:
    > We need to be able to reload code on a live system. This live system
    > has a daemon process always running but it runs many subprocesses with
    > multiprocessing, and the subprocesses might have a short life...
    >
    > Now I found a way to reload the code successfully, as you can see from
    > this testcase:
    >
    >
    > def func():
    > from . import a
    > print(a.ret())
    >
    >
    > class TestMultiProc(unittest.TestCase):
    > def setUp(self):
    > open(path.join(cur_dir, 'a.py'), 'w').write(old_a)
    >
    > def tearDown(self):
    > remove(path.join(cur_dir, 'a.py'))
    >
    > def test_reloading(self):
    > """Starting a new process gives a different result
    > """
    > p1 = Process(target=func)
    > p2 = Process(target=func)
    > p1.start()
    > res = p1.join()
    > open(path.join(cur_dir, 'a.py'), 'w').write(new_a)
    > remove(path.join(cur_dir, 'a.pyc'))
    >
    > p2.start()
    > res = p2.join()
    >
    >
    > As long as I import the code in the function and make sure to remove the
    > "pyc" files everything seems to work..
    > Are there any possible problems which I'm not seeing in this approachor
    > it's safe?
    >
    > Any other better ways otherwise?


    If a byte code interpreter is embedded in the executable, then the program
    can obtain or reload code objects in the run time.

    In C/C++, unless one can swap some DLL safely or spawn in another process with
    new executables obtained from other places, or a script interpreter is embedded, otherwise it's not easy for an instance in C/C++ to get new methods dynamically.

    or
    88888 Dihedral, Jul 19, 2012
    #2
    1. Advertising

  3. andrea crottiæ–¼ 2012å¹´7月19日星期四UTC+8下åˆ6時15分11秒寫é“:
    > We need to be able to reload code on a live system. This live system
    > has a daemon process always running but it runs many subprocesses with
    > multiprocessing, and the subprocesses might have a short life...
    >
    > Now I found a way to reload the code successfully, as you can see from
    > this testcase:
    >
    >
    > def func():
    > from . import a
    > print(a.ret())
    >
    >
    > class TestMultiProc(unittest.TestCase):
    > def setUp(self):
    > open(path.join(cur_dir, 'a.py'), 'w').write(old_a)
    >
    > def tearDown(self):
    > remove(path.join(cur_dir, 'a.py'))
    >
    > def test_reloading(self):
    > """Starting a new process gives a different result
    > """
    > p1 = Process(target=func)
    > p2 = Process(target=func)
    > p1.start()
    > res = p1.join()
    > open(path.join(cur_dir, 'a.py'), 'w').write(new_a)
    > remove(path.join(cur_dir, 'a.pyc'))
    >
    > p2.start()
    > res = p2.join()
    >
    >
    > As long as I import the code in the function and make sure to remove the
    > "pyc" files everything seems to work..
    > Are there any possible problems which I'm not seeing in this approachor
    > it's safe?
    >
    > Any other better ways otherwise?


    If a byte code interpreter is embedded in the executable, then the program
    can obtain or reload code objects in the run time.

    In C/C++, unless one can swap some DLL safely or spawn in another process with
    new executables obtained from other places, or a script interpreter is embedded, otherwise it's not easy for an instance in C/C++ to get new methods dynamically.

    or
    88888 Dihedral, Jul 19, 2012
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris Angelico

    Re: reloading code and multiprocessing

    Chris Angelico, Jul 20, 2012, in forum: Python
    Replies:
    0
    Views:
    121
    Chris Angelico
    Jul 20, 2012
  2. andrea crotti

    Re: reloading code and multiprocessing

    andrea crotti, Jul 23, 2012, in forum: Python
    Replies:
    0
    Views:
    146
    andrea crotti
    Jul 23, 2012
  3. Chris Angelico

    Re: reloading code and multiprocessing

    Chris Angelico, Jul 23, 2012, in forum: Python
    Replies:
    0
    Views:
    134
    Chris Angelico
    Jul 23, 2012
  4. andrea crotti

    Re: reloading code and multiprocessing

    andrea crotti, Jul 25, 2012, in forum: Python
    Replies:
    0
    Views:
    192
    andrea crotti
    Jul 25, 2012
  5. andrea crotti

    Re: reloading code and multiprocessing

    andrea crotti, Jul 27, 2012, in forum: Python
    Replies:
    0
    Views:
    159
    andrea crotti
    Jul 27, 2012
Loading...

Share This Page