Thread#raise, Thread#kill, and timeout.rb are unsafe

  • Thread starter Charles Oliver Nutter
  • Start date
C

Charles Oliver Nutter

I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
I hope can start making the rounds. Long story short, neither
Thread#raise nor Thread#kill is safe to use, and as a result all
libraries that call them are also unsafe (including timeout.rb, net/*,
and many other libraries in the wider world).

Have a look, add comments, discuss. Hopefully we can find a way to fix
this, because it's going to hold Ruby back until we do.

http://headius.blogspot.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html

- Charlie
 
R

Robert Klemme

2008/2/25 said:
I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
I hope can start making the rounds. Long story short, neither
Thread#raise nor Thread#kill is safe to use, and as a result all
libraries that call them are also unsafe (including timeout.rb, net/*,
and many other libraries in the wider world).

Have a look, add comments, discuss. Hopefully we can find a way to fix
this, because it's going to hold Ruby back until we do.

http://headius.blogspot.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html

There is a typo: your second example misses the "main." before the raise. :)

I have to think a bit more about this, but one remark: IMHO the
locking should be outside the begin end block. Reason is, that if
locking fails for a reason you would not want / need to unlock.

At the moment I'm pondering implications of these options:

main = Thread.current
timer = Thread.new { sleep 5; main.raise }
lock(some_resource)
begin
do_some_work
ensure
unlock_some_resource
timer.kill # moved downwards
end


main = Thread.current
timer = Thread.new { sleep 5; main.raise }
begin
lock(some_resource)
begin
do_some_work
ensure
unlock_some_resource
end
ensure
timer.kill # separate ensure
end

The last one has the added benefit that you can implement it in a
method with a block because timer code is not interleaved with work
code.

But both have the disadvantage that you risk getting a timeout
exception during unlock - which would be unfortunate. OTOH, killing
the timer before the lock release can violate the timing constraint if
the unlocking takes longer - not very likely.

Nasty issues; we had some similar issues with changed timeout handling
of a TX manager in JBoss. Basically the old TX manager was able to
bust an otherwise perfect transaction by timing out in the 2PC
process. :))

Kind regards

robert
 
P

Paul Brannan

Unlike Java, Thread#raise etc. should be treated similar to
KeyboardInterrupt in Ruby. No realtime exception posting is expected.
If you handle KeyboardInterrupt safely, do same thing for Thread#raise
etc., e.g. just turning on flags to reserve exception, then raise
exception at the safe place, as MRI does. Nothing more is required.

I'm not convinced KeyboardInterrupt can be handled safely, either, in a
script. What happens if a KeyboardInterrupt exception is raised inside
an ensure block? It could easily result in resource leaks.

I think the right way to handle signals and timeouts is either in an
event loop or a dedicated thread. Asynchronous exceptions are
convenient but error-prone.

Paul
 
P

Paul Brannan

Hi,

In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"

|I'm not convinced KeyboardInterrupt can be handled safely, either, in a
|script. What happens if a KeyboardInterrupt exception is raised inside
|an ensure block? It could easily result in resource leaks.

You are right, but keyboard interrupt has been there so long in the
history, and we hardly see such leak problems, so I assume it's safer
than you might think. At least, we cannot not removed the interrupts
from the language.

For simple scripts, KeyboardInterrupt is probably the right behavior.

For applications that use an event loop, we can use trap('INT') to
disable KeyboardInterrupt.

I wonder if there should be an equivalent trap mechanism for other
asynchronous exceptions?

Paul
 
R

Roger Pack

Charles said:
I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
I hope can start making the rounds. Long story short, neither
Thread#raise nor Thread#kill is safe to use, and as a result all
libraries that call them are also unsafe (including timeout.rb, net/*,
and many other libraries in the wider world).

Have a look, add comments, discuss. Hopefully we can find a way to fix
this, because it's going to hold Ruby back until we do.

http://headius.blogspot.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html

- Charlie

It was the problems with timeout that make me wish we had an
EnsureCritical (an Ensure that gets Thread.critical=true set to it
before entering it), so that timeouts can end properly.
Just my $.02
 
T

Tanaka Akira

Yukihiro Matsumoto said:
Probably. Does anyone have any idea?

The problem is, where is safe point.

You said "just turning on flags to reserve exception, then
raise exception at the safe place, as MRI does."

Your "safe" is for interpreter. Ruby shouldn't SEGV, etc.

But Charles's "safe" is for application. acquired lock
should be released later, etc.

Your "safe" is not enough for application.

We need a mechanism to delay asynchronous exceptions until a
safe point.

trap is bad way to do it. trap is similar to Unix signal
handler. Unix signal handler makes it difficult to avoid
race conditions around blocking operations such as I/O.
With trap, applications need to re-implement the race
condition handling in Ruby level. It is very difficult if
it is possible.

It is preferable to have a way to declare safe points
directly.

The safe points vary according to applications. Even in a
application, different kinds of exceptions may have
different safe points.

For example, there are applications KeyboardInterrupt is
safe as you said. I think a filter, such as grep, is a kind
of the applications. It means that any points in the
applications are safe points.

Another example: movemail is a program to move mbox in a
mail spool. It needs to lock a mbox. Assume we implement
movemail in Ruby and the lock scheme is file lock. The lock
file is removed in ensure clause. But asynchronous
exception in the ensure block may cause to fail removing the
lock file. So the ensure block is not safe points.

Yet another example: Apache can be stopped with apachectl
"stop" and "graceful-stop". "stop" aborts open connections
but "graceful-stop" doesn't. Assume we implement a
threading http server with keep-alive and similar stopping
feature in Ruby. "stop" can be implemented by killing
threads for each connections by (dangerous) Thread.kill. It
means that any points in the threads are safe points.
"graceful-stop" need to wait the threads until they waits
requests. Since keep-alive is assumed, the threads
doesn't terminate after the first request on a connection.
This means the safe points are the request waiting points of
the threads.

I think the safe points declaration mechanism should able to
define safe points for each exception and easy to handle
blocking operations without race conditions.
 
M

MenTaLguY

It was the problems with timeout that make me wish we had an
EnsureCritical (an Ensure that gets Thread.critical=true set to it
before entering it), so that timeouts can end properly.

Anything using Thread.critical has its own problems, particularly
when native threads are involved. It can easily result in deadlocks
if the running thread inadvertently tries to use something which another
stopped thread is using (imagine a thread stopped halfway through
releasing a lock that the ensure block needs to acquire, for example
-- something which would have worked fine with normal use of locks, but
becomes a problem once threads can "stop the world").

-mental
 
R

Roger Pack

What should safe point declaration mechanism look like?

matz.

Tongue in cheek I would say something like

uninterruptible do

end

or

ensureUninterruptible

end

or

uninterruptible do
# some stuff
handle_exceptions_that_have_been_thrown
end
# where handle_exceptions is where they can all get handled.
Kind of scary still, though.

:)

Anyway my latest thought on the whole timeout difficulty is that there
are a few difficulties with it that might be overcomeable:

Unfortunately, as a previous poster noted, using raise seems itself
almost always inherently dangerous, so these would probably all be
band-aids to a bigger problem.

1) if you have two nested timeouts then "you don't know where the
timeout came from"
timeout 5 do
timeout 5 do
begin
sleep
rescue Exception
# which one is it? What did I just rescue?
end
end
end


You could overcome this by just ensuring that it throws distinct classes
per timeout (i.e. if different timeouts).
timeout 5 do # will by default throw class Timeout::ErrorDepth1 <
Timeout::Error
timeout 5 do # will by default throw class Timeout::ErrorDepth2 <
Timeout::Error

I suppose this would help people decipher what is happening when
timeouts are thrown, though not be helpful otherwise.

1.5) It creates a new thread once per call to 'timeout' which is less
than efficient.

While I typically don't care about efficiency when using timeout, it
might be possible to just have a "single timer thread" that handles all
the raise'ings on the others, to save on thread creation
(theoretically).


So something like this

(sorry if the syntax is wrong--this is just pseudo-code for now)

$timer_thread = nil # just so I remember the name--you wouldn't have to
use a global, of course

class TimerThread < Thread
def handle_any_requests
# if any thread requests a timeout, add it to some list
if some_thread_is_waiting
sleep_until_should_wake_next_thread
if waiting_thread_is_still_running_its_timeout_block
waiting_thread.raise Timeout::Error
end
else
sleep -1
end
end

def TimerThread.add_me_to_the_queue_to_interrupt_me_in seconds, &block
$timer_thread.add_it_to_queue Thread.current, seconds
$timer_thread.wakeup # might need to synchronize this call so that
we don't get confusion
block.call # if we get interrupted between this line and the next we
are DEAD. Exactly the same problem that we had before--an exception
could be raised "far in the future" to interrupt it, when it has already
left and it shouldn't be raised.
$timer_thread.take_me_off_the_queue_I_am_done Thread.current # might
need to synchronize this call, too
end

end



# begin program. Note that you could just fire this up the first time
the script calls Timeout::timeout
$timer_thread = Thread.new {
loop do
handle_any_requests
end
}


Now when you want timeouts you do something like
TimerThread.add_me_to_the_queue_to_interrupt_me_in 5 do # seconds

# do some stuff for a long time

end

That would help with efficiency, since you wouldn't have to spawn a new
thread for each call (though it would, sigh, leave one running always in
the background). It would alleviate some concurrency problems, and
still leave at least one in there. Just throwing it out there.

2) nested timeouts can interrupt each other and leave and 'errant
thread' running which raises a

timeout 5 do
timeout 50 do
begin
sleep
end
end

This causes two threads to be generated, each with a "time out bomb"
ticking, waiting to happen. The outer loops' thread could (as Charles
Nutter noted) interrupt the inner thread during its ensure block, so it
ends up not killing its ticking "time out bomb" thread. Which means
that 45 seconds later it will go off and randomly raise on the parent
thread.

To avoid this I "think" you could wrap the things with mutexes.
However, this also doesn't work because the mutex "synchronize" stuff
itself uses an ensure block, which could be interrupted by the same
confusion (and does, in my experience).

<<Sigh>>

Never mind. I give up. The only safe way to generate asynchronous
calls seems to be to use EventMachine or rev (which have asynchronous
timers) and have your own "exception handling" blocks (i.e. blocks where
it is designated to handle those exceptions).

Thanks.

-R
 
M

MenTaLguY

Tongue in cheek I would say something like

uninterruptible do

end

or

ensureUninterruptible

end

or

uninterruptible do
# some stuff
handle_exceptions_that_have_been_thrown
end
# where handle_exceptions is where they can all get handled.
Kind of scary still, though.

In a language with side-effects, I've become convinced that the only safe
way to handle asynchronous exceptions is for code to be uninterruptible by
default, and only explicitly allow interrupts at specific points (not broad
regions of code).

This gets back to the fundamental principle that, in order to write a
correct multithreaded program, any interaction between threads needs to be
by explicit mutual agreement at specific places. One thread needs to
explicitly initiate the interaction (in this case, by calling Thread#raise),
and the other thread needs to explicitly accept it (e.g. by calling a method
which is known to permit the delivery of any pending interrupts when it is
called). Otherwise, the resulting nondeterminism is impossible to control.

-mental
 
T

Tanaka Akira

Yukihiro Matsumoto said:
What should safe point declaration mechanism look like?

Basically, asynchronous events should be queued.

* make a queue for each thread.
* Thread#raise(exc) enqueue exc to the queue.
* Thread.check_interrupt(klass) checks an exception which is
a kind of klass in the queue of the current thread. If it
exists, the exception is raised.
* other queue examining mechanism, such as sigpending, etc,
may be useful.

Thread.check_interrupt is a safe point.

However safe points is not only Thread.check_interrupt.
Blocking operations, such as I/O, may or may not be safe
points. It is because Thread.check_interrupt with blocking
operations causes race conditions. So application must
choose that make a blocking operation interruption safe or
uninterruptible.

* Thread.blocking_interruptible(klass) { ... }
* Thread.blocking_uninterruptible(klass) { ... }

Blocking operations in Thread.blocking_interruptible are
safe points.

If a thread has no resources to release, it is safe to kill
asynchronously. Other than that, safe points should be
declared explicitly. When a resource is acquired,
application should declare it.

* Thread.delay_interrupt(klass) { ... }

It is safe points that out of outermost
Thread.delay_interrupt.

Another idea is about ensure clause. Since an ensure
clause is used to release a resource, it may delay
asynchronous events as in Thread.delay_interrupt.
 
J

James Gray

Basically, asynchronous events should be queued.

* make a queue for each thread.
* Thread#raise(exc) enqueue exc to the queue.
* Thread.check_interrupt(klass) checks an exception which is
a kind of klass in the queue of the current thread. If it
exists, the exception is raised.
* other queue examining mechanism, such as sigpending, etc,
may be useful.

Perl went kind-of along these lines around version 5.8, if I recall
correctly. They added "safe signals" which were signals that only
interrupted at times when it was safe for them to do so.

This did have a few weird side-effects though. For example, running a
complex regex would shutoff signal handling for the duration. That
meant you could no longer timeout a regex.

Perhaps they've fixes these issues now, but I remember the change was
not as smooth as they had hoped it would be.

James Edward Gray II
 
T

Tanaka Akira

James Gray said:
Perl went kind-of along these lines around version 5.8, if I recall
correctly. They added "safe signals" which were signals that only
interrupted at times when it was safe for them to do so.

It's from 5.7.3.
http://search.cpan.org/dist/perl/pod/perlipc.pod#Deferred_Signals_(Safe_Signals)
This did have a few weird side-effects though. For example, running a
complex regex would shutoff signal handling for the duration. That
meant you could no longer timeout a regex.

Ruby already defer signal handling same as Perl.
 
M

MenTaLguY

Basically, asynchronous events should be queued.

* make a queue for each thread.
* Thread#raise(exc) enqueue exc to the queue.
* Thread.check_interrupt(klass) checks an exception which is
a kind of klass in the queue of the current thread. If it
exists, the exception is raised.
* other queue examining mechanism, such as sigpending, etc,
may be useful.

Thread.check_interrupt is a safe point.

However safe points is not only Thread.check_interrupt.
Blocking operations, such as I/O, may or may not be safe
points. It is because Thread.check_interrupt with blocking
operations causes race conditions. So application must
choose that make a blocking operation interruption safe or
uninterruptible.

* Thread.blocking_interruptible(klass) { ... }
* Thread.blocking_uninterruptible(klass) { ... }

Blocking operations in Thread.blocking_interruptible are
safe points.

This sounds very good! I hadn't considered anything like
blocking_interruptible, but it seems useful.

-mental
 
R

Roger Pack

Thread.current.raises do |exception|
queue.push exception
end

....

event_loop do
long_running_stuff
if exception = queue.pop
...
end
end

seems like it would address a ton of issued without overly
complicating ruby's internals - just provide an 'on raise' handler.

This looks nice!

I wish that a handler like this could be "unhandled," as well, as you
have the flexibility to handle exceptions only in parts of the code.
Say we call your first block of code as queue_future_exceptions

Then you could have code like

Thread.current.queue_future_exceptions
# code in here is uninterruptible
Thread.current.no_longer_queue_future_exceptions # so I don't have to
worry about them EVERYWHERE in my code--only at the end of certain
blocks.
Thread.current.raises do |exception| ; end

So basically it would allow for uninterruptible blocks. That's what I
wish we had.


In response to Mental's questions of concurrency problems for raises
injected whilst you are in the middle of handling raises, some options
that might help:

1) protect the queue with mutexes--add them to the end of the queue
atomically.
2) dare I say it, require the exception handling to be either within or
without of an "uninterruptible block" --that is, allow the exception
handling to be interrupted (itself), or prevent it from being
interrupted, always, to provide for sanity.

The first option (require handling outside of an uninterruptible block)
would allow for us to "not have to worry" about concurrency of adding to
the exception queue, since we no longer add to the queue but just raise
on the thread. It would allow it to be interrupted, which might be
annoying, however, and isn't our purpose to "avoid" those pesky
asynchronous interrupts?

The second option (requiring handling within an uninterruptible block)
avoids handling's being interrupted, however, it could "miss" an
interrupt or two, should they be raised between when we handle them and
when an "uninterruptible block" ends.

So I guess the best way to deal with it would be to surround queue
addition with mutexes, and handle them both right before an
uninterruptible block ends, and again right after it ends.

Ok so not such a great idea.
Thoughts?
 
M

MenTaLguY

In response to Mental's questions of concurrency problems for raises
injected whilst you are in the middle of handling raises, some options
that might help:

1) protect the queue with mutexes--add them to the end of the queue
atomically.

This would deadlock if the handler happened to fire while the queue's
lock was held. Generally, locking doesn't mix with anything that can
interrupt or suspend the execution of a thread at an arbitrary time.
2) dare I say it, require the exception handling to be either within or
without of an "uninterruptible block" --that is, allow the exception
handling to be interrupted (itself), or prevent it from being
interrupted, always, to provide for sanity.

That's a good point -- besides conflicts between the handler and whatever
the recipient thread is running at the time, you do also need to be
concerned with conflicts between multiple handler instances (if that
is allowed).
So I guess the best way to deal with it would be to surround queue
addition with mutexes, and handle them both right before an
uninterruptible block ends, and again right after it ends.

Ok so not such a great idea.

In terms of an API for signal queuing, I think Tanaka Akira's idea
is elegant and reliable.

-mental
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
474,432
Messages
2,571,681
Members
48,796
Latest member
Greg L.

Latest Threads

Top