Revival of RubyInRuby?

[Michael Neumann]

Dear Rubyists,

Lately (as I was dissatified with Ruby's current continuation
implementation), I remembered that we had a Ruby-In-Ruby project some
years ago. Python has PyPy, which got even fund by EU. Squeak in mostly
written in itself, too.

So the question is, do we get enough interested people together that
would work on such a project? I've noticed that PyPy and StacklessPython
develop much code in so-called Sprints, where the developers meet and
hack a weekend or a whole week on that project. Ruby developers are
probably more distributed that Python's, but maybe it would work if we
build small teams, which are then connected via internet to other teams,
worldwide... just dreaming ;-) Or isn't that exactly the idea of the
Ruby Codefest by Ruby Central, Inc? Maybe we could even get funded ;-)

So who is really interested in this project and would spend a little bit
of his/her free time (of course not on christmas or silvester .

= First steps?

1. setup a project... discuss, discuss, discuss... agree on some goals,
most important is that we stand united and bundle our forces ;-)
2. write a Ruby parser in Ruby (do we already have one, Robert Feldt?)
we could also omit this step, and use the node-tree directly, and
implement the parser later.
3. Ruby AST-to-bytecode generator (YARV bytecode?)
4. Implement the Ruby VM (in Ruby): object model, memory management (GC),
interface to RubyC, port core-libraries.
5. Implement the RubyVM in C (either manually or by using a MicroRuby
approach, by translating a subset of Ruby into C)

Steps 2. and 3. could be reused by future RubyC implementations. 4. and
5. not, this are the steps that would bring us some new features.

What I'm especially dreaming of is the ability to dump any object to
disk... threads, continuations, and to have an accurate
(non-conservative) garbage collector, so that Ruby will under no
circumstances (except bugs) leak memory (that's especially important in
the case of continuations). If we can dump everything, the GC-thing
would of course be optional, as we could stop and restart at any point.

Well, we could also implement a Ruby-like language first... maybe that's
simpler... start small. And parts of this project could flow into
Rite/Ruby2 if matz agrees.

Enough for today...tell me what you think about this (old) idea...

Regards,

Michael

 

[Ryan Davis]

Lately (as I was dissatified with Ruby's current continuation
implementation), I remembered that we had a Ruby-In-Ruby project some
years ago. Python has PyPy, which got even fund by EU. Squeak in
mostly written in itself, too.

So the question is, do we get enough interested people together that
would work on such a project?

We've got a fairly good start w/ our ruby2c translator. The long term
goal is to reimplement ruby in ruby and translate it to C in much the
same way that Squeak smalltalk does. More info:

IRC: #ruby2c
rubyforge: http://rubyforge.org/projects/ruby2c/
Propaganda: http://www.zenspider.com/~ryand/Ruby2C.pdf

 

[David Garamond]

Lately (as I was dissatified with Ruby's current continuation
implementation), I remembered that we had a Ruby-In-Ruby project some
years ago. Python has PyPy, which got even fund by EU. Squeak in mostly
written in itself, too.

So the question is, do we get enough interested people together that
would work on such a project? I've noticed that PyPy and StacklessPython

[snip]

Btw, I'm not following PyPy or the Stackless project. Have they produced
something "tangible" or "real"? It seems Stackless have existed for
years (back in the day I was beginning to learn Python) but I don't
recall the main Python implementation being stackless even today?

Regards,
dave

 

[gabriele renzi]

David Garamond ha scritto:

[snip]

Btw, I'm not following PyPy or the Stackless project. Have they produced
something "tangible" or "real"? It seems Stackless have existed for
years (back in the day I was beginning to learn Python) but I don't
recall the main Python implementation being stackless even today?

the main implementation is not stackless, but stackless is working and
used in some projects, IIUC. PyPy is not usable, but it seem to be
shaping fine, they recently ained a found from the EU, too.

 

[Michael Neumann]

We've got a fairly good start w/ our ruby2c translator. The long term
goal is to reimplement ruby in ruby and translate it to C in much the
same way that Squeak smalltalk does. More info:

That's great news. Very similar to what I thought. But I don't see a
reason for implementing the parser in a subset of Ruby (except for
performance). Once you've bootstrapped the parser to bytecode with
real-Ruby, you can load it into the VM.

How active is the project? Who else is working on it?

Parser: I'm currently hacking a bit on my packrat project and try to get
a TPDL (Top Down Parsing Language) generator working. It's pretty slow
(~1000 lines per second), maybe if I implement memoizing it will become
faster. Nevertheless, it would be fast enough if you precompile the
whole libraries to .rbc, and use the parser only for "eval" at runtime.
I don't think that this would be a big performance penalty.

Garbage Collector?
Any concrete plans, which strategy to use? Using a non conservating GC
would break with existing C extensions. But if they'd be rewritten in
Ruby2C language, the generator could automatically generate code for a
conservating GC or a non conservating... I'm for a non-conservative due
to the reasons I said in the last email.

My idea for the Ruby2C translator is very similar, but more C oriented
than yours (it's not only for extensions). Example:

class MyStruct < CStruct

def_type :initialize, {:a => 'int', :b => 'int', :return => 'void'}
def initialize(a, b)
@a = a
@b = b
end

def_type :inspect, {:return => 'void'}
def inspect
printf("%d\n%d\n", @a, @b)
end

...
end

which would then be translated into C:

typedef struct MyStruct {
int a;
int b;
};

void MyStruct_initialize(MyStruct* self, int a, int b) {
self->a = a;
self->b = b;
}

void MyStruct_inspect(MyStruct *self) {
printf("%d\n%d\n", self->a, self->b);
}

The types of the instance variables are automatically infered. Types or
local, argument and return values have to be specified via def_type.
Special methods like deref(value) would map to *(value) in the C output.
Constants map to MACROS, so they are simply replaced with their name
in the C output:

def a
printf("%d\n", SQL_BIT)
end

becomes:

printf("%d\n", SQL_BIT)

And of course method calls map to function calls in C, with the
difference that, if the method is defined in the class, then the self
argument is passed, otherwise a real function call (as for printf) is
performed.

I think with this kind of translator, it's pretty simple to write C
programs cleanly in Ruby. And it's still valid Ruby, unlike Pyrex for
example (Pyrex is a Python-dialect for implementing extensions, but it's
not valid Python, AFAIK).


One could extend that a bit, and e.g. if your class inherits from
T_DATA, then you are specifying a Ruby-extensions, which gets
automatically wrapped and unwrapped, and provides the attached C struct
to you without hassle.

And of course class methods map to "self"-less functions:

type_def :"MyStruct.new", {:i => int, :j => int, a => 'MyStruct*',
:return => 'MyStruct*'}
def MyStruct.new(i, j)
obj = cast(malloc(sizeof(MyStruct)), 'MyStruct*')
if obj == NULL
# error
else
MyStruct_initialize(obj, i, j) # or obj.initialize(i,j)
return obj
end
end


Indeed, what I propose is some kind of C with classes ;-)
Maybe I'm missing something important.

 

[Ryan Davis]

That's great news. Very similar to what I thought. But I don't see a
reason for implementing the parser in a subset of Ruby (except for
performance). Once you've bootstrapped the parser to bytecode with
real-Ruby, you can load it into the VM.

I wasn't planning on making a bytecode step. The system is architected
in such a way that it is possible (and easy--eric has done a
prototype), but it wasn't in the plans.

How active is the project? Who else is working on it?

Taking a minor vacation this month, but it is Eric and me w/ some
close-by projects that may merge in the future.

Parser: I'm currently hacking a bit on my packrat project and try to
get a TPDL (Top Down Parsing Language) generator working. It's pretty
slow (~1000 lines per second), maybe if I implement memoizing it will
become faster. Nevertheless, it would be fast enough if you precompile
the whole libraries to .rbc, and use the parser only for "eval" at
runtime. I don't think that this would be a big performance penalty.

You might want to check out coco/r(uby) (on rubyforge as cocor). It is
the coco/r RDP framework ported to ruby (and generating pure ruby). I
haven't benchmarked it at all, but I suspect it does pretty well.

Garbage Collector?
Any concrete plans, which strategy to use? Using a non conservating GC
would break with existing C extensions. But if they'd be rewritten in
Ruby2C language, the generator could automatically generate code for a
conservating GC or a non conservating... I'm for a non-conservative
due to the reasons I said in the last email.

non-conservative won't necessarily break C extensions as far as I
understand. It'll just have to jump through some extra hoops.

My idea for the Ruby2C translator is very similar, but more C oriented
than yours (it's not only for extensions). Example:

ruby2c isn't for extensions at all. If you could tell me what gave you
that impression (like a slide or something, I'll fix it).

The types of the instance variables are automatically infered. Types
or local, argument and return values have to be specified via
def_type. Special methods like deref(value) would map to *(value) in
the C output. Constants map to MACROS, so they are simply replaced
with their name in the C output:

We do type inference already, without a map.

And of course method calls map to function calls in C, with the
difference that, if the method is defined in the class, then the self
argument is passed, otherwise a real function call (as for printf) is
performed.

Here is where we have maps, but only for mapping into the standard C
library.

 

[Michael Neumann]

We've got a fairly good start w/ our ruby2c translator. The long term
goal is to reimplement ruby in ruby and translate it to C in much the
same way that Squeak smalltalk does. More info:

IRC: #ruby2c
rubyforge: http://rubyforge.org/projects/ruby2c/
Propaganda: http://www.zenspider.com/~ryand/Ruby2C.pdf

Is the source somewhere available? I can't find it on the rubyforge
project page.

Regards,

Michael

 

[gabriele renzi]

Michael Neumann ha scritto:

My idea for the Ruby2C translator is very similar, but more C oriented
than yours (it's not only for extensions). Example:

<snip>
note that also koichi sasada (the YARV guy) is planning to provide a c
translation engine. There was some stuff about it in the rubyconf
slides, but I can't remember anything :/

 

[Michael Neumann]

Michael Neumann ha scritto:



<snip>
note that also koichi sasada (the YARV guy) is planning to provide a c
translation engine. There was some stuff about it in the rubyconf
slides, but I can't remember anything :/

But remember that koichi's ahead-of-time compiler is quite different.
This code still depends on the runtime+libraries, whereas we aim at
writing the runtime and core libraries (Array, Hash, String...) in a
Ruby-subset. Both makes sense!

Regards,

Michael

 

[Ryan Davis]

Is the source somewhere available? I can't find it on the rubyforge
project page.

Not yet no... but SOON! We have one more round of private review
that'll happen in a day or two and then it'll go public after we
incorporate the feedback. I'm also working on expanding its
capabilities as much as possible before it goes to the next review, but
that isn't critical.