Kind of ParsTree for 1.9.1

[Macario Ortega]

Hi, I've been missing ParseTree very badly in 1.9.1. So in my naivité I
decided to try to convert YARV iseq's to Sexp's similar of those
produced by ParseTree. So far I can convert simple expressions, method
calls, nested method calls, arrays, hashes, I can extract method
parameter and default values provided the expression is not too
complicated.
At first it seemed easy until I hitted control structures.

Is someone aware of a similar effort?

Does anyone else craves AST manipulation in 1.9.1 as bad as me?

Anyone care to join?

Any sugested reads appart from YARV wiki?

My attempt is hosted at http://github.com/maca/iseq_parser/tree/master


Thanks
Macario

 

[Thomas Chust]

[...]
Hi, I've been missing ParseTree very badly in 1.9.1. So in my naivit=E9 I
decided to try to convert YARV iseq's to Sexp's similar of those
produced by ParseTree.
[...]
Is someone aware of a similar effort?
[...]

Hello,

the Ruby 1.9.1p243 that I installed from source came with a library
called 'ripper' that can produce S-expression parse trees from Ruby
code, like this:

$ cat foo.rb
require 'ripper'
require 'pp'

pp Ripper.sexp %{
def foo
:bar
end
}

$ ruby foo.rb
[rogram,
[[:def,
[ident, "foo", [2, 6]],
[arams, nil, nil, nil, nil, nil],
[:body_stmt,
[[:symbol_literal, [:symbol, [ident, "bar", [3, 5]]]]],
nil,
nil,
nil]]]]

I played around with that a little, but didn't try to code anything
serious. Also, since I'm new to Ruby and haven't used ParseTree
before, I don't know how compatible the abstract syntax tree
representations are.

cu,
Thomas


--=20
When C++ is your hammer, every problem looks like your thumb.

 

[Caleb Clausen]

the Ruby 1.9.1p243 that I installed from source came with a library
called 'ripper' that can produce S-expression parse trees from Ruby
code, like this:

$ cat foo.rb
require 'ripper'
require 'pp'

pp Ripper.sexp %{
def foo
:bar
end
}

Hey, I've been wondering how one gets a tree out of ripper, or even if
it can be done at all. Thanks!

$ ruby foo.rb
[rogram,
[[:def,
[ident, "foo", [2, 6]],
[arams, nil, nil, nil, nil, nil],
[:body_stmt,
[[:symbol_literal, [:symbol, [ident, "bar", [3, 5]]]]],
nil,
nil,
nil]]]]

I played around with that a little, but didn't try to code anything
serious. Also, since I'm new to Ruby and haven't used ParseTree
before, I don't know how compatible the abstract syntax tree
representations are.

That's definitely not compatible with any of the ParseTree formats.

 

[Macario Ortega]

Thomas, Thanks for you reply. I wish there was more info on ripper,
definitelly useful Allthough nice thing about ParseTree is that it
works with "live" objects, meanining there is no need to resort to the
source you can extract the AST for a Class even it was defined with eval
or redefined later.

[...]
Hi, I've been missing ParseTree very badly in 1.9.1. So in my naivit� I
decided to try to convert YARV iseq's to Sexp's similar of those
produced by ParseTree.
[...]
Is someone aware of a similar effort?
[...]

Hello,

the Ruby 1.9.1p243 that I installed from source came with a library
called 'ripper' that can produce S-expression parse trees from Ruby
code, like this:

$ cat foo.rb
require 'ripper'
require 'pp'

pp Ripper.sexp %{
def foo
:bar
end
}

$ ruby foo.rb
[rogram,
[[:def,
[ident, "foo", [2, 6]],
[arams, nil, nil, nil, nil, nil],
[:body_stmt,
[[:symbol_literal, [:symbol, [ident, "bar", [3, 5]]]]],
nil,
nil,
nil]]]]

I played around with that a little, but didn't try to code anything
serious. Also, since I'm new to Ruby and haven't used ParseTree
before, I don't know how compatible the abstract syntax tree
representations are.

cu,
Thomas

 

[Ryan Davis]

Hi, I've been missing ParseTree very badly in 1.9.1. So in my =20
naivit=E9 I
decided to try to convert YARV iseq's to Sexp's similar of those
produced by ParseTree. So far I can convert simple expressions, method
calls, nested method calls, arrays, hashes, I can extract method
parameter and default values provided the expression is not too
complicated.
At first it seemed easy until I hitted control structures.

Is someone aware of a similar effort?

Does anyone else craves AST manipulation in 1.9.1 as bad as me?

Anyone care to join?

This is pretty awesome. It is gonna only get worse tho.

We talked about doing this a while back and came to the conclusion =20
that it was more work than we were interested in investing. Please =20
prove us wrong and you'll get a least a couple beers at the next =20
rubyconf.

A suggestion: Use test/pt_testcase.rb from the ParseTree project and =20
you'll be a LOT happier. See the test cases from either ruby2c or =20
ruby_parser as examples of test reuse.

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

Does anyone else craves AST manipulation in 1.9.1 as bad as me?

Yes


Any sugested reads appart from YARV wiki?

Does ruby_parser work on 1.9?

http://parsetree.rubyforge.org/

 

[Macario Ortega]

Tony Arcieri wrote:

Does ruby_parser work on 1.9?

Yeah it does but you need to pass it ruby source code not live objects.

 

[Macario Ortega]

On Aug 3, 2009, at 23:20 , Macario Ortega wrote:


A suggestion: Use test/pt_testcase.rb from the ParseTree project and
you'll be a LOT happier. See the test cases from either ruby2c or
ruby_parser as examples of test reuse.

Thanks for the suggestion Ryan! I don't have much time to hack nowadays
but I love ParseTree and there is definitelly a gap in 1.9.1. As I told
I started out of naivite.

 

[Charles Oliver Nutter]

Tony Arcieri wrote:



Yeah it does but you need to pass it ruby source code not live objects.

That's how Ripper works too. Notice in this example:

require 'ripper'
require 'pp'

pp Ripper.sexp %{
def foo
:bar
end
}

The %{ .. } is a string.

- Charlie

 

[Kevin Rutherford]

Hi Macario,

Does anyone else craves AST manipulation in 1.9.1 as bad as me?

Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
in 1.9 too, because it means that some of its Rspec matchers no longer
work. Without ParseTree (or equivalent) it's not possible to write:

MyClass.should_not reek

Anyone care to join?

I don't have much time available, but I'll be happy to help.
As Ryan suggests, the starting point will be a decent suite of regression tests.
Cheers,
Kevin
--
m: +44 (0) 797 356 3521
w: http://www.kevinrutherford.co.uk
e: (e-mail address removed)
t: @kevinrutherford
l: http://www.linkedin.com/in/kevinrutherford

 

[Brian Candler]

That's how Ripper works too. Notice in this example:

require 'ripper'
require 'pp'

pp Ripper.sexp %{
def foo
:bar
end
}

The %{ .. } is a string.

I think that's the point You *have* to pass it a String source for
it to parse. You can't take an object which Ruby has already parsed and
installed, such as methodfoo) in this case, and get the sexp from the
object.

 

[Roger Pack]

Hi Macario,


Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
in 1.9 too, because it means that some of its Rspec matchers no longer
work. Without ParseTree (or equivalent) it's not possible to write:

MyClass.should_not reek

You can still pull the code for a method using #source_location [though
it's definitely not as nice as using parse tree was, and not as stable].

A crude example of extracting a method can be found in the method
describer gem [1].

What problems are you running up against exactly? Learning lessons from
python decompyle might be helpful, too.
I also remember that Paul Brannan had some type of iseq -> c translator,
that might be related.
GL!

-r
[1] http://github.com/rogerdpack/method_describer/tree/master

 

[Macario Ortega]

Hi Roger. nice to read you.

I've allready done that with our named arguments gem but I wan't to have
AST for live objects. The project is going dormant for a while
(allthough is not gonna die) because lack of time but I found great
advice in this thread:

Having a tight and organized set of tests and recycling tests from other
proyects such as PT and checking other proyects that work with bytecode.

Cheers

Macario

Hi Macario,


Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
in 1.9 too, because it means that some of its Rspec matchers no longer
work. Without ParseTree (or equivalent) it's not possible to write:

MyClass.should_not reek

You can still pull the code for a method using #source_location [though
it's definitely not as nice as using parse tree was, and not as stable].

A crude example of extracting a method can be found in the method
describer gem [1].

What problems are you running up against exactly? Learning lessons from
python decompyle might be helpful, too.
I also remember that Paul Brannan had some type of iseq -> c translator,
that might be related.
GL!

-r
[1] http://github.com/rogerdpack/method_describer/tree/master

 

[Charles Oliver Nutter]

I think that's the point You *have* to pass it a String source for
it to parse. You can't take an object which Ruby has already parsed and
installed, such as methodfoo) in this case, and get the sexp from the
object.

Yeah, what's wrong with that?

Ignoring for the moment that different impls (and even different revs
of MRI) have (often wildly) different AST structures...

The problem with expecting you can get an AST at runtime is that it
means impls have to hold on to the AST for you. Guess what one of the
largest startup memory costs is in MRI and JRuby...that's right, it's
the AST from loading all that code into a big in-memory data
structure. Ruby 1.9 doesn't have the AST available anymore for a good
reason: it doesn't need it. JRuby won't need it soon either once our
new compiler lands, and already doesn't have it available when you
precompile code. So I guess it's a question of which is
better...getting rid of that memory use or having the AST around all
the time.

And again, that ignores that the AST is ever-changing and usually very
different across impls.

- Charlie

 

[Macario Ortega]

Yeah, what's wrong with that?

Ignoring for the moment that different impls (and even different revs
of MRI) have (often wildly) different AST structures...

The problem with expecting you can get an AST at runtime is that it
means impls have to hold on to the AST for you. Guess what one of the
largest startup memory costs is in MRI and JRuby...that's right, it's
the AST from loading all that code into a big in-memory data
structure. Ruby 1.9 doesn't have the AST available anymore for a good
reason: it doesn't need it. JRuby won't need it soon either once our
new compiler lands, and already doesn't have it available when you
precompile code. So I guess it's a question of which is
better...getting rid of that memory use or having the AST around all
the time.

And again, that ignores that the AST is ever-changing and usually very
different across impls.

- Charlie

I see the AST representation spitted by ParseTree as some sort of lisp
like meta programming language in its own right (I've never used lisp
but I am increasingly interested in it). I really don't care if it
corresponds to Ruby internal representation as long as I can use it to
generate Ruby code (or not Ruby code) and it provides a level of
introspection not available without access to it.

I think I once saw a discussion between Ola Bini, Mentalguy and others
about the need to have a common sexp representation across diferent
implementations, some sort or Risp.

I think there is a gap in 1.9.x that RubyParser fills partialy as it
only works with static code.

Thanks

Macario

 

[Robert Dober]

And again, that ignores that the AST is ever-changing and usually very
different across impls.

I guess if I could really chose I would like a switch, and would like
the VM to keep the AST iff that switch is set (-a|--ast).
Goo{df} idea?

Cheers
Robert
--=20
module Kernel
alias_method :=EB, :lambda
end

 

[Lui Core]

...
I think that's the point You *have* to pass it a String source for
it to parse. You can't take an object which Ruby has already parsed and
installed, such as methodfoo) in this case, and get the sexp from the
object.

One way to generate AST from live object is to use method_missing, which
is applicable in 1.9 with pure ruby code.

see
http://rubyquiz.com/quiz95.html

 

[Roger Pack]

One way to generate AST from live object is to use method_missing, which
is applicable in 1.9 with pure ruby code.

see
http://rubyquiz.com/quiz95.html

That is fascinating. Maybe I should read the ruby quizzes
-r

 

[Jörg W Mittag]

I guess if I could really chose I would like a switch, and would like
the VM to keep the AST iff that switch is set (-a|--ast).
Goo{df} idea?

Unfortunately, by then it will already be too late. There's nothing
the VM can do about it. That's exactly the reason why ParseTree
doesn't work on YARV: the only component that needs the parsetree is
the compiler. By the time the compiler hands off the code to the VM,
the parsetree is long gone. The same applies to XRuby and Ruby.NET as
well as the AOT modes of Rubinius, JRuby, IronRuby and MacRuby.

And remember that the compiler and the VM need not be run by the same
person, on the same machine or at the same time. In HotRuby, for
example, the compiler is generally run by the web developer on the web
server at development time whereas the VM is run by the user inside
the browser on the client, maybe months after the code was compiled.

jwm

 

[Brian Candler]

One way to generate AST from live object is to use method_missing, which
is applicable in 1.9 with pure ruby code.

see
http://rubyquiz.com/quiz95.html

That's not proper Ruby; I don't see any loops or conditional branches. I
suspect this approach requires you to solve the halting problem. e.g.

sxp {
for i in 3..1.0/0
for j in 3..i
for k in j..i
for p in 3..i
if i**p + j**p == k**p
puts "fermat disproved: #{i}, #{j}, #{k}, #{p}"
break
end
end
end
end
end
}