reimport module every n seconds

Discussion in 'Python' started by Santiago Caracol, Feb 17, 2011.

  1. Hello,

    a server program of mine uses data which are compiled to a Python
    module for efficiency reasons. In some module of the server program I
    import the data:

    from data import data

    As the data often changes, I would like to reimport it every n (e.g.
    10) seconds.

    Unfortunately, it is rather difficult and probably not a good idea to
    manipulate the main function of the server program, as it comes from
    Django. (Already asked in the Django newsgroup, but I got no answer
    there.) Is there a way to do the reimport in a module that is used by
    the main program?

    Santiago
     
    Santiago Caracol, Feb 17, 2011
    #1
    1. Advertising

  2. Santiago Caracol wrote:
    > Hello,
    >
    > a server program of mine uses data which are compiled to a Python
    > module for efficiency reasons. In some module of the server program I
    > import the data:
    >
    > from data import data
    >
    > As the data often changes, I would like to reimport it every n (e.g.
    > 10) seconds.
    >
    > Unfortunately, it is rather difficult and probably not a good idea to
    > manipulate the main function of the server program, as it comes from
    > Django. (Already asked in the Django newsgroup, but I got no answer
    > there.) Is there a way to do the reimport in a module that is used by
    > the main program?
    >
    > Santiago
    >

    import data

    def getUpdate():
    reload(data)
    return data.data

    JM

    WARNING : there's a lot to know about reload, the main thing is that it
    cannot properly reimport a module in a sense that all objects already
    created from that module are not impacted. You can search this archive
    if you want to know more as this subject has already been discussed many
    times.
     
    Jean-Michel Pichavant, Feb 17, 2011
    #2
    1. Advertising

  3. Santiago Caracol

    Aahz Guest

    In article <>,
    Santiago Caracol <> wrote:
    >
    >a server program of mine uses data which are compiled to a Python
    >module for efficiency reasons. In some module of the server program I
    >import the data:
    >
    >from data import data
    >
    >As the data often changes, I would like to reimport it every n (e.g.
    >10) seconds.


    Don't do that. ;-) I suggest using exec instead. However, I would be
    surprised if import worked faster than, say, JSON (more precisely, I
    doubt that it's enough faster to warrnat this kludge).
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Programming language design is not a rational science. Most reasoning
    about it is at best rationalization of gut feelings, and at worst plain
    wrong." --GvR, python-ideas, 2009-03-01
     
    Aahz, Feb 17, 2011
    #3
  4. Santiago Caracol

    Roy Smith Guest

    In article <ijjo0p$3dm$>, (Aahz)
    wrote:

    > In article
    > <>,
    > Santiago Caracol <> wrote:
    > >
    > >a server program of mine uses data which are compiled to a Python
    > >module for efficiency reasons. In some module of the server program I
    > >import the data:
    > >
    > >from data import data
    > >
    > >As the data often changes, I would like to reimport it every n (e.g.
    > >10) seconds.

    >
    > Don't do that. ;-) I suggest using exec instead. However, I would be
    > surprised if import worked faster than, say, JSON (more precisely, I
    > doubt that it's enough faster to warrnat this kludge).


    I'm with Aahz. Don't do that.

    I don't know what you're doing, but I suspect an even better solution
    would be to have your program run a "reconfigure" thread which listens
    on a UDP socket and reads a JSON object from it. Or, at the very least,
    which listens for a signal which kicks it to re-read some JSON from a
    disk file. This will be more responsive when the data changes quickly
    and more efficient when it changes slowly. As opposed to just polling
    for changes every 10 seconds.
     
    Roy Smith, Feb 17, 2011
    #4
  5. > > Don't do that.  ;-)  I suggest using exec instead.  However, I would be
    > > surprised if import worked faster than, say, JSON (more precisely, I
    > > doubt that it's enough faster to warrnat this kludge).

    >
    > I'm with Aahz.  Don't do that.
    >
    > I don't know what you're doing, but I suspect an even better solution
    > would be to have your program run a "reconfigure" thread which listens
    > on a UDP socket and reads a JSON object from it.  Or, at the very least,
    > which listens for a signal which kicks it to re-read some JSON from a
    > disk file.  This will be more responsive when the data changes quickly
    > and more efficient when it changes slowly.  As opposed to just polling
    > for changes every 10 seconds.


    Somehow I guessed that I would be told not to do it. But it's my
    fault. I should have been more explicit. The data is not just data. It
    is a large set of Django objects I call "ContentClassifiers" that have
    each certain methods that calculate from user input very large regular
    expressions. They they have a method "classify" that is applied to
    messages and uses the very large regular expressions. To classify a
    message I simply apply the classify method of all ContentClassifiers.
    There are very many Contentclassifiers. I compile the
    ContentClassifiers, which are Django objects, to pure Python objects
    in order to not have to fetch them from the database each time I need
    them and in order to be able to compile the large regular expressions
    offline. In Django, I generated and compiled each regular expressions
    of each ContentClassifier each time I needed it to classify a message.
    I didn't find a good way to calculate and compile the regular
    expressions once and store them.

    I already tested the automatically generated module: Before,
    classifying a message took about 10 seconds, now it takes miliseconds.

    My only problem is reloading the module: At the time being, I simply
    restart the web server manually from time to time in order to load the
    freshly compiled module.
     
    Santiago Caracol, Feb 18, 2011
    #5
  6. Santiago Caracol

    Aahz Guest

    In article <>,
    Santiago Caracol <> wrote:
    >>> Don't do that. =A0;-) =A0I suggest using exec instead. =A0However, I wo=

    >uld be
    >>> surprised if import worked faster than, say, JSON (more precisely, I
    >>> doubt that it's enough faster to warrnat this kludge).

    >>
    >> I'm with Aahz. =A0Don't do that.
    >>
    >> I don't know what you're doing, but I suspect an even better solution
    >> would be to have your program run a "reconfigure" thread which listens
    >> on a UDP socket and reads a JSON object from it. =A0Or, at the very least=

    >,
    >> which listens for a signal which kicks it to re-read some JSON from a
    >> disk file. =A0This will be more responsive when the data changes quickly
    >> and more efficient when it changes slowly. =A0As opposed to just polling
    >> for changes every 10 seconds.

    >
    >Somehow I guessed that I would be told not to do it. But it's my
    >fault. I should have been more explicit. The data is not just data. It
    >is a large set of Django objects I call "ContentClassifiers" that have
    >each certain methods that calculate from user input very large regular
    >expressions. They they have a method "classify" that is applied to
    >messages and uses the very large regular expressions. To classify a
    >message I simply apply the classify method of all ContentClassifiers.
    >There are very many Contentclassifiers. I compile the
    >ContentClassifiers, which are Django objects, to pure Python objects
    >in order to not have to fetch them from the database each time I need
    >them and in order to be able to compile the large regular expressions
    >offline. In Django, I generated and compiled each regular expressions
    >of each ContentClassifier each time I needed it to classify a message.
    >I didn't find a good way to calculate and compile the regular
    >expressions once and store them.
    >
    >I already tested the automatically generated module: Before,
    >classifying a message took about 10 seconds, now it takes miliseconds.
    >
    >My only problem is reloading the module: At the time being, I simply
    >restart the web server manually from time to time in order to load the
    >freshly compiled module.


    Why not just create a second process and send the messages there? Then
    restart the second process each time you need to reload. That can be
    easily automated.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Beware of companies that claim to be like a family. They might not be
    lying." --Jill Lundquist
     
    Aahz, Mar 17, 2011
    #6
  7. Santiago Caracol

    John Nagle Guest

    On 3/17/2011 3:53 PM, Aahz wrote:
    > In article<>,
    > Santiago Caracol<> wrote:
    >>>> Don't do that. =A0;-) =A0I suggest using exec instead. =A0However, I wo=

    >> uld be
    >>>> surprised if import worked faster than, say, JSON (more precisely, I
    >>>> doubt that it's enough faster to warrnat this kludge).
    >>>
    >>> I'm with Aahz. =A0Don't do that.

    >>
    >> I didn't find a good way to calculate and compile the regular
    >> expressions once and store them.


    Hm. Regular expression compilation in CPython happens when
    you call "re.compile", not when .pyc files are generated.
    You can call "re.compile" on strings read from an external
    source without loading a new module.

    John Nagle
     
    John Nagle, Mar 21, 2011
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. tshad
    Replies:
    7
    Views:
    16,993
    tshad
    Mar 11, 2005
  2. 00_CP_D12
    Replies:
    3
    Views:
    8,923
    dilipv
    Feb 22, 2008
  3. Stu
    Replies:
    7
    Views:
    1,766
    Dave Thompson
    Mar 7, 2005
  4. peteshinners

    Python reimport

    peteshinners, Jun 3, 2009, in forum: Python
    Replies:
    1
    Views:
    859
    peteshinners
    Jun 3, 2009
  5. Michael Tan
    Replies:
    32
    Views:
    989
    Ara.T.Howard
    Jul 21, 2005
Loading...

Share This Page