Use a Thread to reload a Module?

Discussion in 'Python' started by =?ISO-8859-1?Q?Gregory_Pi=F1ero?=, Dec 22, 2006.

  1. Hi Python Experts,

    I hope I can explain this right. I'll try.

    Background:
    I have a module that I leave running in a server role. It has a
    module which has data in it that can change. So every nth time a
    function in the server gets called, I want to reload the module so it
    has the freshest data. But there's a lot of data so it takes 5-10
    seconds to do a reload.

    My question is:
    Would I be able to launch a seperate thread to reload the module while
    the server does it work? Hopefully it would be using the old module
    data right up until the thread was finished reloading.

    Thanks in advance,

    Greg

    Here's some psuedo code that might illustrate what I'm talking about:

    import lotsa_data

    def serve():
    reload(lotsa_data) #can this launch a thread so "do_stuff" runs right away?
    do_stuff(lotsa_data.data)

    while 1:
    listen_for_requests()
     
    =?ISO-8859-1?Q?Gregory_Pi=F1ero?=, Dec 22, 2006
    #1
    1. Advertising

  2. =?ISO-8859-1?Q?Gregory_Pi=F1ero?=

    Carl Banks Guest

    Gregory PiƱero wrote:
    > Hi Python Experts,
    >
    > I hope I can explain this right. I'll try.
    >
    > Background:
    > I have a module that I leave running in a server role. It has a
    > module which has data in it that can change. So every nth time a
    > function in the server gets called, I want to reload the module so it
    > has the freshest data. But there's a lot of data so it takes 5-10
    > seconds to do a reload.
    >
    > My question is:
    > Would I be able to launch a seperate thread to reload the module while
    > the server does it work? Hopefully it would be using the old module
    > data right up until the thread was finished reloading.
    >
    > Thanks in advance,
    >
    > Greg
    >
    > Here's some psuedo code that might illustrate what I'm talking about:
    >
    > import lotsa_data
    >
    > def serve():
    > reload(lotsa_data) #can this launch a thread so "do_stuff" runs right away?
    > do_stuff(lotsa_data.data)
    >
    > while 1:
    > listen_for_requests()


    Using a thread for this purpose is no problem. Using a module: yep,
    that's a problem. (I'd say using a module in this way, to update data,
    is very far from best practice, but its convenience justifies simple
    uses. You are going beyond simple now, though.)

    Not knowing more about your program, I'd say the simplest way is:

    1. exec, don't reload, your data file (with the standard warning that
    exec should only be used on carefully contructed code, or to
    deliberately give the user the power to input python code--of course
    the same warning applies when reloading a dynamically generated
    module).

    2. Store the new data somewhere (such as a Queue) waiting for a good
    time to update.

    3. At a convenient time, overwrite the old data in the module.

    I'm going to assume that your server has heretofore been
    single-threaded; therefore you don't need locks or queues or semaphores
    in your main code. Here, then, is a very improvable example to
    consider. Notice that the lotsa_data module is empty. Instead, you
    call load_data() to exec the file where the data really is, and it puts
    the loaded data into a queue. Next time you wait for a new request, it
    checks to see if there are any data updates in the queue, and updates
    the date in lotsa_data module if so.

    import Queue
    import threading
    import lotsa_data ## empty!

    data_update_queue = Queue.Queue()

    def serve():
    if request_count % n:
    threading.Thread(target=load_data).start()
    do_stuff(lotsa_data.data)
    request_count += 1

    def load_data()
    d = {}
    exec "/path/to/data/file" in d
    data_update_queue.put(d)

    load_data() # run once in main thread to load data initially
    while True:
    try:
    d = data_update_queue.get_nowait(d)
    except Queue.Empty:
    pass
    else:
    for key,value in d:
    setattr(losta_data,key,value)
    listen_for_requests()


    There is much room for improvement. For example, can requests come in
    fast enough to spawn another load_data before the first had ended? You
    should consider trying to acquire a threading.Lock in load_data and
    waiting and/or returning immediately if you can't. Other things can go
    wrong, too. Using threads requires care. But hopefully this'll get
    you started.


    Carl Banks
     
    Carl Banks, Dec 23, 2006
    #2
    1. Advertising

  3. On 22 Dec 2006 20:02:31 -0800, Carl Banks <> wrote:
    ....
    > There is much room for improvement. For example, can requests come in
    > fast enough to spawn another load_data before the first had ended? You
    > should consider trying to acquire a threading.Lock in load_data and
    > waiting and/or returning immediately if you can't. Other things can go
    > wrong, too. Using threads requires care. But hopefully this'll get
    > you started.
    >


    Thanks, Carl. That gives me a lot to consider. I won't be able to
    work on this project more until Wed. but I'll probably run into some
    follow up questions then.

    A few more details if it helps ...

    That module deals with accessing data from QuickBooks, marshaling it,
    and providing methods to access the marshaled data. Having the server
    program keep that module and all of its data in memory makes the
    server run really fast, but yeah, it does get complicated now that
    it's storing 100's of MB of data. I guess most people go to a
    database at this point?
    It's just so easy to store the data in the Python objects I already
    need them in instead of converting to tables in a DB and then
    converting back. So maybe this threading will work for me.

    Thanks again,

    Greg
     
    =?ISO-8859-1?Q?Gregory_Pi=F1ero?=, Dec 23, 2006
    #3
  4. =?ISO-8859-1?Q?Gregory_Pi=F1ero?=

    Aahz Guest

    In article <>,
    =?ISO-8859-1?Q?Gregory_Pi=F1ero?= <> wrote:
    >
    >That module deals with accessing data from QuickBooks, marshaling it,
    >and providing methods to access the marshaled data. Having the server
    >program keep that module and all of its data in memory makes the
    >server run really fast, but yeah, it does get complicated now that
    >it's storing 100's of MB of data. I guess most people go to a
    >database at this point?


    Very, very yes. Try using a SQLite in-memory database, maybe?

    >It's just so easy to store the data in the Python objects I already
    >need them in instead of converting to tables in a DB and then
    >converting back. So maybe this threading will work for me.


    You might well run into problems with the import lock. I strongly
    advise against your current approach.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "I support family values -- Addams family values" --www.nancybuttons.com
     
    Aahz, Dec 24, 2006
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Joshua Beall
    Replies:
    26
    Views:
    220,672
    AndreiKaa
    Jun 29, 2007
  2. Lonnie Princehouse

    reload fails if module not in sys.path

    Lonnie Princehouse, Oct 21, 2005, in forum: Python
    Replies:
    4
    Views:
    539
    Lonnie Princehouse
    Oct 21, 2005
  3. gen_tricomi
    Replies:
    2
    Views:
    746
    gen_tricomi
    May 23, 2006
  4. Stefan Mueller
    Replies:
    4
    Views:
    298
    Stefan Mueller
    Nov 5, 2005
  5. Gildor Oronar
    Replies:
    4
    Views:
    85
Loading...

Share This Page