Everyone's favorite flow control: retry

[Charles Oliver Nutter]

Today I was thinking about retry support in JRuby, and figured we've
reached a decision point.

retry is a tricky little bugger. It works the way it does in MRI largely
because of the way the interpreter works; so in that sense it's very
implementation specific. So here's some example code, see if you can
figure out what it prints.

$count = 0
class Foo
def initialize
$count += 1
@value = $count
end

def do_retry(i)
puts "value is #{@value} and i is #{i}"
retry
end
end

a = 0
Foo.new.do_retry(a += 1) {}

Give up? Here's the answer:

value is 1 and i is 1
value is 2 and i is 2
value is 3 and i is 3

The trick here is the retry. A retry can only be used in a rescue block
(which is ok), inside a block, or inside a method that's been passed a
block. Excluding rescue, retry causes the original call to the method
with the block to reevaluate. And I don't just mean re-execute with the
same values, I mean re-evaluate both the receiver and the arguments and
make the call again.

So for example, this method invocation

def foo(a); puts 3; retry; end

(print 1; self).foo((print 2; nil)) {}

Prints out "123" forever, because the receiver (the (print 1; self) bit)
and the arguments (the (print 2; nil) bit) get executed again. Goofy, right?

This also affects retry within a block

def foo(a); puts 3; yield; end

(print 1; self).foo((print 2; nil)) { retry }

This also prints "123" repeatedly.

The problem I've run into is that it's really cumbersome to implement
this correctly in the compiler, cumbersome enough I'm debating whether
we'll support it. There's a few reasons for this:

- retry is handled using exception; so every method call that takes a
block would have to be wrapped with exception-handling for retry to
work. Exception-handling is expensive, even if no exceptions are
actually raised. It would also add a significant amount of code.
- there's no way to know that a method will be reevaluated by looking at
it, which makes it both dangerous and impossible to predict.
- nobody understands retry, and nobody uses it.

As I understand it, retry in a method body is going away in 1.9, so it
should be considered deprecated behavior. So far, retry in a block is
not yet going away, though I'd prefer it did. Actually, I wish this
behavior never existed, because it makes a lot of assumptions about the
way Ruby is implemented. Ask if you want to know more about that.

I'm looking for input on this. If there's a way to implement it I have
missed, please tell me. If you have a concern with this feature being
disabled, please tell me (it won't affect retry within rescue blocks).
If you want to talk about it more, please tell me.

- Charlie

 

[Gregory Seidman]

Today I was thinking about retry support in JRuby, and figured we've
reached a decision point. [...]
The problem I've run into is that it's really cumbersome to implement this
correctly in the compiler, cumbersome enough I'm debating whether we'll
support it. There's a few reasons for this:

- retry is handled using exception; so every method call that takes a block
would have to be wrapped with exception-handling for retry to work.
Exception-handling is expensive, even if no exceptions are actually raised.
It would also add a significant amount of code.
- there's no way to know that a method will be reevaluated by looking at
it, which makes it both dangerous and impossible to predict.
- nobody understands retry, and nobody uses it.

Okay, I disagree with that last point. Retry is very handy when
appropriate, and I have, indeed, used it.

As I understand it, retry in a method body is going away in 1.9, so it
should be considered deprecated behavior. So far, retry in a block is not
yet going away, though I'd prefer it did. Actually, I wish this behavior
never existed, because it makes a lot of assumptions about the way Ruby is
implemented. Ask if you want to know more about that.

The semantics of retry outside the context of a rescue makes no sense to
me.

I'm looking for input on this. If there's a way to implement it I have
missed, please tell me. If you have a concern with this feature being
disabled, please tell me (it won't affect retry within rescue blocks). If
you want to talk about it more, please tell me.

I recommend an RCR for retry to be available only in a rescue and nowhere
else. Furthermore, unless this is already the case with a rescue in a
method (i.e. using the method's implicit exception handling context), retry
should only rerun the code in the exception handling context and nothing
outside it (e.g. the method call itself). Unless someone chimes in with a
really good use case for the other forms of retry, I think they are worse
than hard to implement -- they are confusing to both the programmer writing
the code and the programmers reading the code.

- Charlie

--Greg

 

[MonkeeSage]

Today I was thinking about retry support in JRuby, and figured we've
reached a decision point. [...]
The problem I've run into is that it's really cumbersome to implement this
correctly in the compiler, cumbersome enough I'm debating whether we'll
support it. There's a few reasons for this:

- retry is handled using exception; so every method call that takes a block
would have to be wrapped with exception-handling for retry to work.
Exception-handling is expensive, even if no exceptions are actually raised.
It would also add a significant amount of code.
- there's no way to know that a method will be reevaluated by looking at
it, which makes it both dangerous and impossible to predict.
- nobody understands retry, and nobody uses it.

Okay, I disagree with that last point. Retry is very handy when
appropriate, and I have, indeed, used it.

As I understand it, retry in a method body is going away in 1.9, so it
should be considered deprecated behavior. So far, retry in a block is not
yet going away, though I'd prefer it did. Actually, I wish this behavior
never existed, because it makes a lot of assumptions about the way Ruby is
implemented. Ask if you want to know more about that.

The semantics of retry outside the context of a rescue makes no sense to
me.

As I understand, it's like a label in C. Charles' first example is
equivalent to something like this...

#include <stdio.h>
void foo(int (*pf)(const char *f, ...), char *a) {
(*pf)("%s\n", a);
}
int main() {
retry:
foo(&printf, "123");
goto retry;
return 0;
}

 

[Charles Oliver Nutter]

As I understand, it's like a label in C. Charles' first example is
equivalent to something like this...

#include <stdio.h>
void foo(int (*pf)(const char *f, ...), char *a) {
(*pf)("%s\n", a);
}
int main() {
retry:
foo(&printf, "123");
goto retry;
return 0;
}

It's more than that...here's some pseudo C code:

void foo() { goto retry; }
int main() {
retry:
foo();
return 0;
}

Retry outside a rescue actually allows you to jump back across call
boundaries and start an invocation again. Scary.

- Charlie

 

[MonkeeSage]

It's more than that...here's some pseudo C code:

void foo() { goto retry; }
int main() {
retry:
foo();
return 0;

}

Retry outside a rescue actually allows you to jump back across call
boundaries and start an invocation again. Scary.

- Charlie

Ah. How does it unwind the stack like that? Is that what you meant
about implementation specific stuff?

Ps. I don't think anyone will miss retry outside of a rescue. I won't
at least.

Regards,
Jordan

 

[Charles Oliver Nutter]

Ah. How does it unwind the stack like that? Is that what you meant
about implementation specific stuff?

It raises an exception, a LocalJumpError (which is how all non-local
flow control works) which is then handled further up the stack.

The reason it's implementation-specific is because the behavior ends up
being a side effect of MRI's interpreter.

The following call:

foo {}

Parses into an AST like this:

Iter # this represents the block
FCall :foo # this represents the call receiving the block
nil # no args to the call

Outside of a rescue, retry bubbles out to the nearest Iter. If there is
none, it bubbles all the way out and terminates execution. The
implementation-specific bit here is the fact that Iter contains FCall,
rather than the more intuitive reverse; this causes the retry to go all
the way back above the FCall and try it completely anew, reevaluating
its receiver and arguments in the process. In compiled implementations
or implementations with a different AST, this doesn't map well. It's
very specific to the way MRI represents Iter/Call in the AST.

Ps. I don't think anyone will miss retry outside of a rescue. I won't
at least.

I've just disabled it outside of rescue in JRuby trunk; we'll see how
that goes. So far, nothing seems breaks.

- Charlie

 

[Alex Gutteridge]

Today I was thinking about retry support in JRuby, and figured we've
reached a decision point. [...]
The problem I've run into is that it's really cumbersome to
implement this
correctly in the compiler, cumbersome enough I'm debating whether
we'll
support it. There's a few reasons for this:

- retry is handled using exception; so every method call that takes
a block
would have to be wrapped with exception-handling for retry to work.
Exception-handling is expensive, even if no exceptions are actually
raised.
It would also add a significant amount of code.
- there's no way to know that a method will be reevaluated by
looking at
it, which makes it both dangerous and impossible to predict.
- nobody understands retry, and nobody uses it.

Okay, I disagree with that last point. Retry is very handy when
appropriate, and I have, indeed, used it.

I'm pretty sure Charles meant that nobody uses or understands retry
*outside a rescue* there. Clearly it's very useful in the context of a
rescue.

This works for me in irb:

irb(main):004:0> def foo(a); puts 3; yield; end
=> nil
irb(main):005:0> (print 1; self).foo((print 2; nil)) { retry }
123
123
123
123
123
123
......Ad infinitum

But not in 1.8.6:

[alexg@powerbook]/Users/alexg/Desktop(27): cat retry2.rb
def foo(a); puts 3; yield; end

(print 1; self).foo((print 2; nil)) { retry }
[alexg@powerbook]/Users/alexg/Desktop(28): ruby retry2.rb
retry2.rb:3: private method `foo' called for main:Object (NoMethodError)

Though this does:

[alexg@powerbook]/Users/alexg/Desktop(31): cat retry.rb
class Foo
def initialize
print 1
end
def foo(a); puts 3; yield; end
end

Foo.new.foo((print 2; nil)) { retry }
[alexg@powerbook]/Users/alexg/Desktop(32): ruby retry.rb
123
123
123
123
123
......Ad infinitum

Color me confused, and one extra vote for removing retry outside of
rescue.

Alex Gutteridge

Bioinformatics Center
Kyoto University

 

[MonkeeSage]

It raises an exception, a LocalJumpError (which is how all non-local
flow control works) which is then handled further up the stack.

The reason it's implementation-specific is because the behavior ends up
being a side effect of MRI's interpreter.

The following call:

foo {}

Parses into an AST like this:

Iter # this represents the block
FCall :foo # this represents the call receiving the block
nil # no args to the call

Outside of a rescue, retry bubbles out to the nearest Iter. If there is
none, it bubbles all the way out and terminates execution. The
implementation-specific bit here is the fact that Iter contains FCall,
rather than the more intuitive reverse; this causes the retry to go all
the way back above the FCall and try it completely anew, reevaluating
its receiver and arguments in the process. In compiled implementations
or implementations with a different AST, this doesn't map well. It's
very specific to the way MRI represents Iter/Call in the AST.

I see. Thanks for explaining.

I've just disabled it outside of rescue in JRuby trunk; we'll see how
that goes. So far, nothing seems breaks.

- Charlie

Regards,
Jordan

 

[Alex Gutteridge]

Today I was thinking about retry support in JRuby, and figured we've
reached a decision point. [...]
The problem I've run into is that it's really cumbersome to
implement this
correctly in the compiler, cumbersome enough I'm debating whether
we'll
support it. There's a few reasons for this:

- retry is handled using exception; so every method call that
takes a block
would have to be wrapped with exception-handling for retry to work.
Exception-handling is expensive, even if no exceptions are
actually raised.
It would also add a significant amount of code.
- there's no way to know that a method will be reevaluated by
looking at
it, which makes it both dangerous and impossible to predict.
- nobody understands retry, and nobody uses it.

Okay, I disagree with that last point. Retry is very handy when
appropriate, and I have, indeed, used it.

I'm pretty sure Charles meant that nobody uses or understands retry
*outside a rescue* there. Clearly it's very useful in the context of
a rescue.

This works for me in irb:

irb(main):004:0> def foo(a); puts 3; yield; end
=> nil
irb(main):005:0> (print 1; self).foo((print 2; nil)) { retry }
123
123
123
123
123
123
......Ad infinitum

But not in 1.8.6:

[alexg@powerbook]/Users/alexg/Desktop(27): cat retry2.rb
def foo(a); puts 3; yield; end

(print 1; self).foo((print 2; nil)) { retry }
[alexg@powerbook]/Users/alexg/Desktop(28): ruby retry2.rb
retry2.rb:3: private method `foo' called for main:Object
(NoMethodError)

Sorry, I didn't cut and paste that correctly. The real output is:

[alexg@powerbook]/Users/alexg/Desktop(33): ruby retry2.rb
retry2.rb:3: private method `foo' called for main:Object (NoMethodError)
12[alexg@powerbook]/Users/alexg/Desktop(34):

So one 'iteration' gets called, but then 'foo' somehow becomes
undefined? Can anyone explain this behaviour?

Though this does:

[alexg@powerbook]/Users/alexg/Desktop(31): cat retry.rb
class Foo
def initialize
print 1
end
def foo(a); puts 3; yield; end
end

Foo.new.foo((print 2; nil)) { retry }
[alexg@powerbook]/Users/alexg/Desktop(32): ruby retry.rb
123
123
123
123
123
......Ad infinitum

Color me confused, and one extra vote for removing retry outside of
rescue.

Alex Gutteridge

Bioinformatics Center
Kyoto University

Alex Gutteridge

Bioinformatics Center
Kyoto University

 

[Alex Gutteridge]

Today I was thinking about retry support in JRuby, and figured
we've
reached a decision point.
[...]
The problem I've run into is that it's really cumbersome to
implement this
correctly in the compiler, cumbersome enough I'm debating whether
we'll
support it. There's a few reasons for this:

- retry is handled using exception; so every method call that
takes a block
would have to be wrapped with exception-handling for retry to work.
Exception-handling is expensive, even if no exceptions are
actually raised.
It would also add a significant amount of code.
- there's no way to know that a method will be reevaluated by
looking at
it, which makes it both dangerous and impossible to predict.
- nobody understands retry, and nobody uses it.

Okay, I disagree with that last point. Retry is very handy when
appropriate, and I have, indeed, used it.

I'm pretty sure Charles meant that nobody uses or understands retry
*outside a rescue* there. Clearly it's very useful in the context
of a rescue.

This also affects retry within a block

def foo(a); puts 3; yield; end

(print 1; self).foo((print 2; nil)) { retry }

This also prints "123" repeatedly.

This works for me in irb:

irb(main):004:0> def foo(a); puts 3; yield; end
=> nil
irb(main):005:0> (print 1; self).foo((print 2; nil)) { retry }
123
123
123
123
123
123
......Ad infinitum

But not in 1.8.6:

[alexg@powerbook]/Users/alexg/Desktop(27): cat retry2.rb
def foo(a); puts 3; yield; end

(print 1; self).foo((print 2; nil)) { retry }
[alexg@powerbook]/Users/alexg/Desktop(28): ruby retry2.rb
retry2.rb:3: private method `foo' called for main:Object
(NoMethodError)

Sorry, I didn't cut and paste that correctly. The real output is:

[alexg@powerbook]/Users/alexg/Desktop(33): ruby retry2.rb
retry2.rb:3: private method `foo' called for main:Object
(NoMethodError)
12[alexg@powerbook]/Users/alexg/Desktop(34):

So one 'iteration' gets called, but then 'foo' somehow becomes
undefined? Can anyone explain this behaviour?

Ignore that last message, I see now it's not even getting through one
iteration. You can't call 'foo' in that way.

Alex Gutteridge

Bioinformatics Center
Kyoto University

 

[Charles Oliver Nutter]

Ignore that last message, I see now it's not even getting through one
iteration. You can't call 'foo' in that way.

It's not callable because it's at the top level. It works if it's in a
class (as you saw) or if you public :foo after defining it.

- Charlie

 

[MonkeeSage]

It raises an exception, a LocalJumpError (which is how all non-local
flow control works) which is then handled further up the stack.

The reason it's implementation-specific is because the behavior ends up
being a side effect of MRI's interpreter.

The following call:

foo {}

Parses into an AST like this:

Iter # this represents the block
FCall :foo # this represents the call receiving the block
nil # no args to the call

Outside of a rescue, retry bubbles out to the nearest Iter. If there is
none, it bubbles all the way out and terminates execution. The
implementation-specific bit here is the fact that Iter contains FCall,
rather than the more intuitive reverse; this causes the retry to go all
the way back above the FCall and try it completely anew, reevaluating
its receiver and arguments in the process. In compiled implementations
or implementations with a different AST, this doesn't map well. It's
very specific to the way MRI represents Iter/Call in the AST.


I've just disabled it outside of rescue in JRuby trunk; we'll see how
that goes. So far, nothing seems breaks.

- Charlie

Just out of curiousity...is there a reason why MRI has Iter closing on
the FCall rather than the other way 'round? Or is it just incidental?

Regards,
Jordan

 

[Thomas Enebo]

Just out of curiousity...is there a reason why MRI has Iter closing on
the FCall rather than the other way 'round? Or is it just incidental?

Regards,
Jordan

If Charlie did not mention this earlier in the thread we have in fact
reversed that relationship in JRuby. I suspect it was reversed in MRI
to make it easier to set/unset (setup/teardown) some stuff when a method
call needs to pass a block. Consider:

doIter {
set something block related
doCall
unset something block related
}

I don't remember specifically what they set/unset but I remember this
appearing to be a possible reason for it.

Perhaps an MRI coder could comment since I have always wondered about
this reversed relationship.

-Tom

 

[Gregory Seidman]

Hi,

In message "Re: Everyone's favorite flow control: retry"

|Today I was thinking about retry support in JRuby, and figured we've
|reached a decision point.

OK, even though I am not fully agreed with you in some edge corner, we
see no specific benefit to keep retry outside of rescue. I and Koichi
talked about this issue, and agreed to remove it in 1.9. I guess he
would do it in a day or two, hopefully.

Thank you, Matz! Even without it being an issue for JRuby, I would be in
favor of this change due to the (IMO) confusing and weird semantics it
presents.

matz.

--Greg