Timing out arbitrary functions

[Steven D'Aprano]

I have a problem and I don't know where to start looking for a solution.

I have a class that needs to call an arbitrary function and wait for a
result. The function, being completely arbitrary and not under my control,
may be very time consuming and possibly may not even halt. My class needs
to be able to give up waiting for a result after a specified amount of
time.

I'm thinking something conceptually like this:

# pseudo-code:
set time out to 30 seconds
try:
result = somefunction()
except TimeOut:
# 30 second time out happened
print "somefunction() timed out without returning"
else:
print "somefunction() returned %s" % result


The easy (for some definition of easy) solution would be to code
somefunction() so that it raised an exception if it hadn't returned a
result within a certain time. Unfortunately, I can't do rely on that -- I
only have control over the calling code, not the called somefunction(),
which may be any arbitrary function.

How do others handle something like this? What should I be looking for?
I'm after a lightweight solution, if any such thing exists.

Thanks,

 

[Paul Rubin]

How do others handle something like this? What should I be looking for?
I'm after a lightweight solution, if any such thing exists.

Is something stopping you from using sigalarm?

 

[David Wahler]

I have a problem and I don't know where to start looking for a solution.

I have a class that needs to call an arbitrary function and wait for a
result. The function, being completely arbitrary and not under my control,
may be very time consuming and possibly may not even halt. My class needs
to be able to give up waiting for a result after a specified amount of
time.

I'm thinking something conceptually like this:

# pseudo-code:
set time out to 30 seconds
try:
result = somefunction()
except TimeOut:
# 30 second time out happened
print "somefunction() timed out without returning"
else:
print "somefunction() returned %s" % result


The easy (for some definition of easy) solution would be to code
somefunction() so that it raised an exception if it hadn't returned a
result within a certain time. Unfortunately, I can't do rely on that -- I
only have control over the calling code, not the called somefunction(),
which may be any arbitrary function.

How do others handle something like this? What should I be looking for?
I'm after a lightweight solution, if any such thing exists.

For simple cases, I would use signal.alarm() with a SIGALARM handler
that raises a TimeOut exception. However, this is by no means
foolproof; you have to rely on the called function not to mess with
your signal handler. Plus, if your alarm occurs within a try-except
block that catches the TimeOut, it'll still be dropped. And to the best
of my knowledge, you can't otherwise forcibly terminate the execution
of a Python thread or block of code.

If you're going to be running untrusted code, I would use the
subprocess module to invoke a separate Python instance which takes the
code to be executed on stdin, and returns a pickled copy of the return
value on stdout. Then you can start it running, wait 30 seconds, and
then kill it if it hasn't already returned.

-- David

 

[Steven D'Aprano]

Is something stopping you from using sigalarm?

Pure ignorance of its existence.


Thanks, I'll check it out.

 

[Paul Rubin]

Pure ignorance of its existence.
Thanks, I'll check it out.

Two things to keep in mind:

- You can have only ONE alarm pending for the whole process. If
different things in the program need timeouts of their own, you have
to manage that yourself, maybe with heapq. And the thing you're
trying to time out may itself mess with the alarm or its handler.

- The alarm raises an exception in the main thread. If you want a
timeout in some other thread, you're more or less out of luck. Antoon
Pardon has posted a couple times about a ctypes-dependent hack that
raises asynchronous exceptions in arbitrary threads, that might be
worth looking into if you have to. I haven't done so for now.

Besides sigalarm you might be able to concoct something with SIGIO
(have another thread sleep til the timeout then send a character back
to the main process through a pipe) or some other signal (use
os.kill). The same issues would apply as with sigalarm.

 

[antti kervinen]

AOP would be a quite elegant way set timeouts for functions, in my
opinion. The nice thing in it is that, in principle, you can write a
single timeout advice code and then wrap it over any function you want
to timeout.

I wrote timeout_advice.py to demonstrate this a couple of years ago
(see http://www.cs.tut.fi/~ask/aspects/aspects.html). It may not
directly solve the problem at hand because it is thought to be used
with a wrap_around implementation that wraps methods in classes rather
than ordinary functions in modules. urllib.URLopener.open is used as an
example in the code. Unfortunately, I still have not implemented the
wrapping for ordinary functions, although it should be straight-forward
with the same idea that is explained in the web page.

Of course, in the real life, timeouts are tricky and dangerous. The
consequences of interrupting a function that is not designed to be
interrupted, or leaving it running in the background after the timeout
(which is what timeout_advice.py does) may be surprising.

-- Antti Kervinen