Forking SocketServer daemon -- updating state

[Reid Priedhorsky]

Hi folks,

I am implementing a forking SocketServer daemon that maintains significant
internal state (a graph that takes ~30s to build by fetching from a SQL
database, and eventually further state that may take up to an hour to
build).

I would like to be able to notify the daemon that it needs to update its
state. Because it forks for each new request, a request handler can't
update the state because then only the child would have the new state.

One idea I had was to use signals. Is it safe to write a signal handler
that does extensive work (several seconds)? Seems like even so, it might
be tricky to do this without race conditions.

Another possibility is that the signal handler simply sets a needs_update
flag, which I could check for in a handle_request() loop. The disadvantage
here is that the update wouldn't happen until after the next request is
handled, and I would like the state to be available for that next request.
A solution might be to send a signal followed by a dummy request, which
seems a bit awkward.

Any other ideas or suggestions?

Thanks in advance,

Reid

p.s. This group's help on A* search was very much appreciated -- just the
ticket!

 

[Irmen de Jong]

Another possibility is that the signal handler simply sets a needs_update
flag, which I could check for in a handle_request() loop. The disadvantage
here is that the update wouldn't happen until after the next request is
handled, and I would like the state to be available for that next request.
A solution might be to send a signal followed by a dummy request, which
seems a bit awkward.

Any other ideas or suggestions?

Just send a special type of request that signifies that an update is
wanted, and act on that?

Basically you need to find a way to let two processes talk to each
other. There are a lot of possibilities here (IPC). The simplest
would be to reuse the one you already have (the request handler!).
Another solution might be to use Pyro.
Or just open a custom socket yourself to send messages.
Or stick with your idea of using a signal handler. But I don't know
if you can let a signal handler run for a long time (as you already
asked yourself), and it won't be portable to Windows for instance.

You could maybe use threads instead, and then use some form of
thread synchronization primitives such as threading.Event
(threads would remove the need to do IPC).

Hope this helps a bit,

--Irmen

 

[Nick Craig-Wood]

I am implementing a forking SocketServer daemon that maintains significant
internal state (a graph that takes ~30s to build by fetching from a SQL
database, and eventually further state that may take up to an hour to
build).

I would like to be able to notify the daemon that it needs to update its
state. Because it forks for each new request, a request handler can't
update the state because then only the child would have the new state.

One idea I had was to use signals. Is it safe to write a signal handler
that does extensive work (several seconds)? Seems like even so, it might
be tricky to do this without race conditions.

In general no it isn't safe.

However due to the way python handles its interrupts (it sets a flag
in its own signal handler which the bytecode interpreter examines and
acts upon when it is safe to do so) you won't corrupt python if you do
this.

I'd still recommend the set a flag and do it in your mainloop approach
though if you can.

Another possibility is that the signal handler simply sets a
needs_update flag, which I could check for in a handle_request()
loop. The disadvantage here is that the update wouldn't happen
until after the next request is handled, and I would like the state
to be available for that next request. A solution might be to send
a signal followed by a dummy request, which seems a bit awkward.

Can't you get SocketServer to timeout and return you control
regularly?