reimport module every n seconds

[Santiago Caracol]

Hello,

a server program of mine uses data which are compiled to a Python
module for efficiency reasons. In some module of the server program I
import the data:

from data import data

As the data often changes, I would like to reimport it every n (e.g.
10) seconds.

Unfortunately, it is rather difficult and probably not a good idea to
manipulate the main function of the server program, as it comes from
Django. (Already asked in the Django newsgroup, but I got no answer
there.) Is there a way to do the reimport in a module that is used by
the main program?

Santiago

 

[Jean-Michel Pichavant]

Hello,

a server program of mine uses data which are compiled to a Python
module for efficiency reasons. In some module of the server program I
import the data:

from data import data

As the data often changes, I would like to reimport it every n (e.g.
10) seconds.

Unfortunately, it is rather difficult and probably not a good idea to
manipulate the main function of the server program, as it comes from
Django. (Already asked in the Django newsgroup, but I got no answer
there.) Is there a way to do the reimport in a module that is used by
the main program?

Santiago

import data

def getUpdate():
reload(data)
return data.data

JM

WARNING : there's a lot to know about reload, the main thing is that it
cannot properly reimport a module in a sense that all objects already
created from that module are not impacted. You can search this archive
if you want to know more as this subject has already been discussed many
times.

 

[Aahz]

a server program of mine uses data which are compiled to a Python
module for efficiency reasons. In some module of the server program I
import the data:

from data import data

As the data often changes, I would like to reimport it every n (e.g.
10) seconds.

Don't do that. ;-) I suggest using exec instead. However, I would be
surprised if import worked faster than, say, JSON (more precisely, I
doubt that it's enough faster to warrnat this kludge).
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Programming language design is not a rational science. Most reasoning
about it is at best rationalization of gut feelings, and at worst plain
wrong." --GvR, python-ideas, 2009-03-01

 

[Roy Smith]

Don't do that. ;-) I suggest using exec instead. However, I would be
surprised if import worked faster than, say, JSON (more precisely, I
doubt that it's enough faster to warrnat this kludge).

I'm with Aahz. Don't do that.

I don't know what you're doing, but I suspect an even better solution
would be to have your program run a "reconfigure" thread which listens
on a UDP socket and reads a JSON object from it. Or, at the very least,
which listens for a signal which kicks it to re-read some JSON from a
disk file. This will be more responsive when the data changes quickly
and more efficient when it changes slowly. As opposed to just polling
for changes every 10 seconds.

 

[Santiago Caracol]

Don't do that.  ;-)  I suggest using exec instead.  However, I would be

I'm with Aahz.  Don't do that.

I don't know what you're doing, but I suspect an even better solution
would be to have your program run a "reconfigure" thread which listens
on a UDP socket and reads a JSON object from it.  Or, at the very least,
which listens for a signal which kicks it to re-read some JSON from a
disk file.  This will be more responsive when the data changes quickly
and more efficient when it changes slowly.  As opposed to just polling
for changes every 10 seconds.

Somehow I guessed that I would be told not to do it. But it's my
fault. I should have been more explicit. The data is not just data. It
is a large set of Django objects I call "ContentClassifiers" that have
each certain methods that calculate from user input very large regular
expressions. They they have a method "classify" that is applied to
messages and uses the very large regular expressions. To classify a
message I simply apply the classify method of all ContentClassifiers.
There are very many Contentclassifiers. I compile the
ContentClassifiers, which are Django objects, to pure Python objects
in order to not have to fetch them from the database each time I need
them and in order to be able to compile the large regular expressions
offline. In Django, I generated and compiled each regular expressions
of each ContentClassifier each time I needed it to classify a message.
I didn't find a good way to calculate and compile the regular
expressions once and store them.

I already tested the automatically generated module: Before,
classifying a message took about 10 seconds, now it takes miliseconds.

My only problem is reloading the module: At the time being, I simply
restart the web server manually from time to time in order to load the
freshly compiled module.

 

[Aahz]

Somehow I guessed that I would be told not to do it. But it's my
fault. I should have been more explicit. The data is not just data. It
is a large set of Django objects I call "ContentClassifiers" that have
each certain methods that calculate from user input very large regular
expressions. They they have a method "classify" that is applied to
messages and uses the very large regular expressions. To classify a
message I simply apply the classify method of all ContentClassifiers.
There are very many Contentclassifiers. I compile the
ContentClassifiers, which are Django objects, to pure Python objects
in order to not have to fetch them from the database each time I need
them and in order to be able to compile the large regular expressions
offline. In Django, I generated and compiled each regular expressions
of each ContentClassifier each time I needed it to classify a message.
I didn't find a good way to calculate and compile the regular
expressions once and store them.

I already tested the automatically generated module: Before,
classifying a message took about 10 seconds, now it takes miliseconds.

My only problem is reloading the module: At the time being, I simply
restart the web server manually from time to time in order to load the
freshly compiled module.

Why not just create a second process and send the messages there? Then
restart the second process each time you need to reload. That can be
easily automated.

 

[John Nagle]

Hm. Regular expression compilation in CPython happens when
you call "re.compile", not when .pyc files are generated.
You can call "re.compile" on strings read from an external
source without loading a new module.

John Nagle