Use a Thread to reload a Module?

  • Thread starter =?ISO-8859-1?Q?Gregory_Pi=F1ero?=
  • Start date
?

=?ISO-8859-1?Q?Gregory_Pi=F1ero?=

Hi Python Experts,

I hope I can explain this right. I'll try.

Background:
I have a module that I leave running in a server role. It has a
module which has data in it that can change. So every nth time a
function in the server gets called, I want to reload the module so it
has the freshest data. But there's a lot of data so it takes 5-10
seconds to do a reload.

My question is:
Would I be able to launch a seperate thread to reload the module while
the server does it work? Hopefully it would be using the old module
data right up until the thread was finished reloading.

Thanks in advance,

Greg

Here's some psuedo code that might illustrate what I'm talking about:

import lotsa_data

def serve():
reload(lotsa_data) #can this launch a thread so "do_stuff" runs right away?
do_stuff(lotsa_data.data)

while 1:
listen_for_requests()
 
C

Carl Banks

Gregory said:
Hi Python Experts,

I hope I can explain this right. I'll try.

Background:
I have a module that I leave running in a server role. It has a
module which has data in it that can change. So every nth time a
function in the server gets called, I want to reload the module so it
has the freshest data. But there's a lot of data so it takes 5-10
seconds to do a reload.

My question is:
Would I be able to launch a seperate thread to reload the module while
the server does it work? Hopefully it would be using the old module
data right up until the thread was finished reloading.

Thanks in advance,

Greg

Here's some psuedo code that might illustrate what I'm talking about:

import lotsa_data

def serve():
reload(lotsa_data) #can this launch a thread so "do_stuff" runs right away?
do_stuff(lotsa_data.data)

while 1:
listen_for_requests()

Using a thread for this purpose is no problem. Using a module: yep,
that's a problem. (I'd say using a module in this way, to update data,
is very far from best practice, but its convenience justifies simple
uses. You are going beyond simple now, though.)

Not knowing more about your program, I'd say the simplest way is:

1. exec, don't reload, your data file (with the standard warning that
exec should only be used on carefully contructed code, or to
deliberately give the user the power to input python code--of course
the same warning applies when reloading a dynamically generated
module).

2. Store the new data somewhere (such as a Queue) waiting for a good
time to update.

3. At a convenient time, overwrite the old data in the module.

I'm going to assume that your server has heretofore been
single-threaded; therefore you don't need locks or queues or semaphores
in your main code. Here, then, is a very improvable example to
consider. Notice that the lotsa_data module is empty. Instead, you
call load_data() to exec the file where the data really is, and it puts
the loaded data into a queue. Next time you wait for a new request, it
checks to see if there are any data updates in the queue, and updates
the date in lotsa_data module if so.

import Queue
import threading
import lotsa_data ## empty!

data_update_queue = Queue.Queue()

def serve():
if request_count % n:
threading.Thread(target=load_data).start()
do_stuff(lotsa_data.data)
request_count += 1

def load_data()
d = {}
exec "/path/to/data/file" in d
data_update_queue.put(d)

load_data() # run once in main thread to load data initially
while True:
try:
d = data_update_queue.get_nowait(d)
except Queue.Empty:
pass
else:
for key,value in d:
setattr(losta_data,key,value)
listen_for_requests()


There is much room for improvement. For example, can requests come in
fast enough to spawn another load_data before the first had ended? You
should consider trying to acquire a threading.Lock in load_data and
waiting and/or returning immediately if you can't. Other things can go
wrong, too. Using threads requires care. But hopefully this'll get
you started.


Carl Banks
 
?

=?ISO-8859-1?Q?Gregory_Pi=F1ero?=

On 22 Dec 2006 20:02:31 -0800 said:
There is much room for improvement. For example, can requests come in
fast enough to spawn another load_data before the first had ended? You
should consider trying to acquire a threading.Lock in load_data and
waiting and/or returning immediately if you can't. Other things can go
wrong, too. Using threads requires care. But hopefully this'll get
you started.

Thanks, Carl. That gives me a lot to consider. I won't be able to
work on this project more until Wed. but I'll probably run into some
follow up questions then.

A few more details if it helps ...

That module deals with accessing data from QuickBooks, marshaling it,
and providing methods to access the marshaled data. Having the server
program keep that module and all of its data in memory makes the
server run really fast, but yeah, it does get complicated now that
it's storing 100's of MB of data. I guess most people go to a
database at this point?
It's just so easy to store the data in the Python objects I already
need them in instead of converting to tables in a DB and then
converting back. So maybe this threading will work for me.

Thanks again,

Greg
 
A

Aahz

That module deals with accessing data from QuickBooks, marshaling it,
and providing methods to access the marshaled data. Having the server
program keep that module and all of its data in memory makes the
server run really fast, but yeah, it does get complicated now that
it's storing 100's of MB of data. I guess most people go to a
database at this point?

Very, very yes. Try using a SQLite in-memory database, maybe?
It's just so easy to store the data in the Python objects I already
need them in instead of converting to tables in a DB and then
converting back. So maybe this threading will work for me.

You might well run into problems with the import lock. I strongly
advise against your current approach.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,902
Latest member
Elena68X5

Latest Threads

Top