Kind of ParsTree for 1.9.1

Discussion in 'Ruby' started by Macario Ortega, Aug 4, 2009.

  1. Hi, I've been missing ParseTree very badly in 1.9.1. So in my naivité I
    decided to try to convert YARV iseq's to Sexp's similar of those
    produced by ParseTree. So far I can convert simple expressions, method
    calls, nested method calls, arrays, hashes, I can extract method
    parameter and default values provided the expression is not too
    complicated.
    At first it seemed easy until I hitted control structures.

    Is someone aware of a similar effort?

    Does anyone else craves AST manipulation in 1.9.1 as bad as me?

    Anyone care to join?

    Any sugested reads appart from YARV wiki?

    My attempt is hosted at http://github.com/maca/iseq_parser/tree/master


    Thanks
    Macario
    --
    Posted via http://www.ruby-forum.com/.
    Macario Ortega, Aug 4, 2009
    #1
    1. Advertising

  2. Macario Ortega

    Thomas Chust Guest

    2009/8/4 Macario Ortega <>:
    > [...]
    > Hi, I've been missing ParseTree very badly in 1.9.1. So in my naivit=E9 I
    > decided to try to convert YARV iseq's to Sexp's similar of those
    > produced by ParseTree.
    > [...]
    > Is someone aware of a similar effort?
    > [...]


    Hello,

    the Ruby 1.9.1p243 that I installed from source came with a library
    called 'ripper' that can produce S-expression parse trees from Ruby
    code, like this:

    $ cat foo.rb
    require 'ripper'
    require 'pp'

    pp Ripper.sexp %{
    def foo
    :bar
    end
    }

    $ ruby foo.rb
    [:program,
    [[:def,
    [:mad:ident, "foo", [2, 6]],
    [:params, nil, nil, nil, nil, nil],
    [:body_stmt,
    [[:symbol_literal, [:symbol, [:mad:ident, "bar", [3, 5]]]]],
    nil,
    nil,
    nil]]]]

    I played around with that a little, but didn't try to code anything
    serious. Also, since I'm new to Ruby and haven't used ParseTree
    before, I don't know how compatible the abstract syntax tree
    representations are.

    cu,
    Thomas


    --=20
    When C++ is your hammer, every problem looks like your thumb.
    Thomas Chust, Aug 4, 2009
    #2
    1. Advertising

  3. On 8/4/09, Thomas Chust <> wrote:
    > the Ruby 1.9.1p243 that I installed from source came with a library
    > called 'ripper' that can produce S-expression parse trees from Ruby
    > code, like this:
    >
    > $ cat foo.rb
    > require 'ripper'
    > require 'pp'
    >
    > pp Ripper.sexp %{
    > def foo
    > :bar
    > end
    > }


    Hey, I've been wondering how one gets a tree out of ripper, or even if
    it can be done at all. Thanks!

    > $ ruby foo.rb
    > [:program,
    > [[:def,
    > [:mad:ident, "foo", [2, 6]],
    > [:params, nil, nil, nil, nil, nil],
    > [:body_stmt,
    > [[:symbol_literal, [:symbol, [:mad:ident, "bar", [3, 5]]]]],
    > nil,
    > nil,
    > nil]]]]
    >
    > I played around with that a little, but didn't try to code anything
    > serious. Also, since I'm new to Ruby and haven't used ParseTree
    > before, I don't know how compatible the abstract syntax tree
    > representations are.


    That's definitely not compatible with any of the ParseTree formats.
    Caleb Clausen, Aug 4, 2009
    #3
  4. Thomas, Thanks for you reply. I wish there was more info on ripper,
    definitelly useful :) Allthough nice thing about ParseTree is that it
    works with "live" objects, meanining there is no need to resort to the
    source you can extract the AST for a Class even it was defined with eval
    or redefined later.



    Thomas Chust wrote:
    > 2009/8/4 Macario Ortega <>:
    >> [...]
    >> Hi, I've been missing ParseTree very badly in 1.9.1. So in my naivit� I
    >> decided to try to convert YARV iseq's to Sexp's similar of those
    >> produced by ParseTree.
    >> [...]
    >> Is someone aware of a similar effort?
    >> [...]

    >
    > Hello,
    >
    > the Ruby 1.9.1p243 that I installed from source came with a library
    > called 'ripper' that can produce S-expression parse trees from Ruby
    > code, like this:
    >
    > $ cat foo.rb
    > require 'ripper'
    > require 'pp'
    >
    > pp Ripper.sexp %{
    > def foo
    > :bar
    > end
    > }
    >
    > $ ruby foo.rb
    > [:program,
    > [[:def,
    > [:mad:ident, "foo", [2, 6]],
    > [:params, nil, nil, nil, nil, nil],
    > [:body_stmt,
    > [[:symbol_literal, [:symbol, [:mad:ident, "bar", [3, 5]]]]],
    > nil,
    > nil,
    > nil]]]]
    >
    > I played around with that a little, but didn't try to code anything
    > serious. Also, since I'm new to Ruby and haven't used ParseTree
    > before, I don't know how compatible the abstract syntax tree
    > representations are.
    >
    > cu,
    > Thomas


    --
    Posted via http://www.ruby-forum.com/.
    Macario Ortega, Aug 4, 2009
    #4
  5. Macario Ortega

    Ryan Davis Guest

    On Aug 3, 2009, at 23:20 , Macario Ortega wrote:

    > Hi, I've been missing ParseTree very badly in 1.9.1. So in my =20
    > naivit=E9 I
    > decided to try to convert YARV iseq's to Sexp's similar of those
    > produced by ParseTree. So far I can convert simple expressions, method
    > calls, nested method calls, arrays, hashes, I can extract method
    > parameter and default values provided the expression is not too
    > complicated.
    > At first it seemed easy until I hitted control structures.
    >
    > Is someone aware of a similar effort?
    >
    > Does anyone else craves AST manipulation in 1.9.1 as bad as me?
    >
    > Anyone care to join?


    This is pretty awesome. It is gonna only get worse tho. :)

    We talked about doing this a while back and came to the conclusion =20
    that it was more work than we were interested in investing. Please =20
    prove us wrong and you'll get a least a couple beers at the next =20
    rubyconf.

    A suggestion: Use test/pt_testcase.rb from the ParseTree project and =20
    you'll be a LOT happier. See the test cases from either ruby2c or =20
    ruby_parser as examples of test reuse.
    Ryan Davis, Aug 4, 2009
    #5
  6. Macario Ortega

    Tony Arcieri Guest

    [Note: parts of this message were removed to make it a legal post.]

    On Tue, Aug 4, 2009 at 12:20 AM, Macario Ortega <> wrote:

    > Does anyone else craves AST manipulation in 1.9.1 as bad as me?
    >


    Yes


    > Any sugested reads appart from YARV wiki?
    >


    Does ruby_parser work on 1.9?

    http://parsetree.rubyforge.org/

    --
    Tony Arcieri
    Medioh/Nagravision
    Tony Arcieri, Aug 4, 2009
    #6
  7. Tony Arcieri wrote:


    >
    > Does ruby_parser work on 1.9?
    >


    Yeah it does but you need to pass it ruby source code not live objects.

    --
    Posted via http://www.ruby-forum.com/.
    Macario Ortega, Aug 5, 2009
    #7
  8. Ryan Davis wrote:
    > On Aug 3, 2009, at 23:20 , Macario Ortega wrote:



    >
    > A suggestion: Use test/pt_testcase.rb from the ParseTree project and
    > you'll be a LOT happier. See the test cases from either ruby2c or
    > ruby_parser as examples of test reuse.



    Thanks for the suggestion Ryan! I don't have much time to hack nowadays
    but I love ParseTree and there is definitelly a gap in 1.9.1. As I told
    I started out of naivite. :)
    --
    Posted via http://www.ruby-forum.com/.
    Macario Ortega, Aug 5, 2009
    #8
  9. On Tue, Aug 4, 2009 at 7:56 PM, Macario Ortega<> wrote:
    > Tony Arcieri wrote:
    >
    >
    >>
    >> Does ruby_parser work on 1.9?
    >>

    >
    > Yeah it does but you need to pass it ruby source code not live objects.


    That's how Ripper works too. Notice in this example:

    require 'ripper'
    require 'pp'

    pp Ripper.sexp %{
    def foo
    :bar
    end
    }

    The %{ .. } is a string.

    - Charlie
    Charles Oliver Nutter, Aug 5, 2009
    #9
  10. Hi Macario,

    > Does anyone else craves AST manipulation in 1.9.1 as bad as me?


    Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
    in 1.9 too, because it means that some of its Rspec matchers no longer
    work. Without ParseTree (or equivalent) it's not possible to write:

    MyClass.should_not reek

    > Anyone care to join?


    I don't have much time available, but I'll be happy to help.
    As Ryan suggests, the starting point will be a decent suite of regression tests.
    Cheers,
    Kevin
    --
    m: +44 (0) 797 356 3521
    w: http://www.kevinrutherford.co.uk
    e:
    t: @kevinrutherford
    l: http://www.linkedin.com/in/kevinrutherford
    Kevin Rutherford, Aug 5, 2009
    #10
  11. Charles Nutter wrote:
    >>> Does ruby_parser work on 1.9?
    >>>

    >>
    >> Yeah it does but you need to pass it ruby source code not live objects.

    >
    > That's how Ripper works too. Notice in this example:
    >
    > require 'ripper'
    > require 'pp'
    >
    > pp Ripper.sexp %{
    > def foo
    > :bar
    > end
    > }
    >
    > The %{ .. } is a string.


    I think that's the point :) You *have* to pass it a String source for
    it to parse. You can't take an object which Ruby has already parsed and
    installed, such as method:)foo) in this case, and get the sexp from the
    object.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Aug 5, 2009
    #11
  12. Macario Ortega

    Roger Pack Guest

    Kevin Rutherford wrote:
    > Hi Macario,
    >
    >> Does anyone else craves AST manipulation in 1.9.1 as bad as me?

    >
    > Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
    > in 1.9 too, because it means that some of its Rspec matchers no longer
    > work. Without ParseTree (or equivalent) it's not possible to write:
    >
    > MyClass.should_not reek


    You can still pull the code for a method using #source_location [though
    it's definitely not as nice as using parse tree was, and not as stable].

    A crude example of extracting a method can be found in the method
    describer gem [1].

    What problems are you running up against exactly? Learning lessons from
    python decompyle might be helpful, too.
    I also remember that Paul Brannan had some type of iseq -> c translator,
    that might be related.
    GL!

    -r
    [1] http://github.com/rogerdpack/method_describer/tree/master
    --
    Posted via http://www.ruby-forum.com/.
    Roger Pack, Aug 8, 2009
    #12
  13. Hi Roger. nice to read you.

    I've allready done that with our named arguments gem but I wan't to have
    AST for live objects. The project is going dormant for a while
    (allthough is not gonna die) because lack of time but I found great
    advice in this thread:

    Having a tight and organized set of tests and recycling tests from other
    proyects such as PT and checking other proyects that work with bytecode.

    Cheers

    Macario


    Roger Pack wrote:
    > Kevin Rutherford wrote:
    >> Hi Macario,
    >>
    >>> Does anyone else craves AST manipulation in 1.9.1 as bad as me?

    >>
    >> Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
    >> in 1.9 too, because it means that some of its Rspec matchers no longer
    >> work. Without ParseTree (or equivalent) it's not possible to write:
    >>
    >> MyClass.should_not reek

    >
    > You can still pull the code for a method using #source_location [though
    > it's definitely not as nice as using parse tree was, and not as stable].
    >
    > A crude example of extracting a method can be found in the method
    > describer gem [1].
    >
    > What problems are you running up against exactly? Learning lessons from
    > python decompyle might be helpful, too.
    > I also remember that Paul Brannan had some type of iseq -> c translator,
    > that might be related.
    > GL!
    >
    > -r
    > [1] http://github.com/rogerdpack/method_describer/tree/master


    --
    Posted via http://www.ruby-forum.com/.
    Macario Ortega, Aug 8, 2009
    #13
  14. On Wed, Aug 5, 2009 at 9:56 AM, Brian Candler<> wrote:
    > I think that's the point :) You *have* to pass it a String source for
    > it to parse. You can't take an object which Ruby has already parsed and
    > installed, such as method:)foo) in this case, and get the sexp from the
    > object.


    Yeah, what's wrong with that?

    Ignoring for the moment that different impls (and even different revs
    of MRI) have (often wildly) different AST structures...

    The problem with expecting you can get an AST at runtime is that it
    means impls have to hold on to the AST for you. Guess what one of the
    largest startup memory costs is in MRI and JRuby...that's right, it's
    the AST from loading all that code into a big in-memory data
    structure. Ruby 1.9 doesn't have the AST available anymore for a good
    reason: it doesn't need it. JRuby won't need it soon either once our
    new compiler lands, and already doesn't have it available when you
    precompile code. So I guess it's a question of which is
    better...getting rid of that memory use or having the AST around all
    the time.

    And again, that ignores that the AST is ever-changing and usually very
    different across impls.

    - Charlie
    Charles Oliver Nutter, Aug 9, 2009
    #14
  15. Charles Nutter wrote:
    > On Wed, Aug 5, 2009 at 9:56 AM, Brian Candler<>
    > wrote:
    >> I think that's the point :) You *have* to pass it a String source for
    >> it to parse. You can't take an object which Ruby has already parsed and
    >> installed, such as method:)foo) in this case, and get the sexp from the
    >> object.

    >
    > Yeah, what's wrong with that?
    >
    > Ignoring for the moment that different impls (and even different revs
    > of MRI) have (often wildly) different AST structures...
    >
    > The problem with expecting you can get an AST at runtime is that it
    > means impls have to hold on to the AST for you. Guess what one of the
    > largest startup memory costs is in MRI and JRuby...that's right, it's
    > the AST from loading all that code into a big in-memory data
    > structure. Ruby 1.9 doesn't have the AST available anymore for a good
    > reason: it doesn't need it. JRuby won't need it soon either once our
    > new compiler lands, and already doesn't have it available when you
    > precompile code. So I guess it's a question of which is
    > better...getting rid of that memory use or having the AST around all
    > the time.
    >
    > And again, that ignores that the AST is ever-changing and usually very
    > different across impls.
    >
    > - Charlie


    I see the AST representation spitted by ParseTree as some sort of lisp
    like meta programming language in its own right (I've never used lisp
    but I am increasingly interested in it). I really don't care if it
    corresponds to Ruby internal representation as long as I can use it to
    generate Ruby code (or not Ruby code) and it provides a level of
    introspection not available without access to it.

    I think I once saw a discussion between Ola Bini, Mentalguy and others
    about the need to have a common sexp representation across diferent
    implementations, some sort or Risp.

    I think there is a gap in 1.9.x that RubyParser fills partialy as it
    only works with static code.

    Thanks

    Macario

    --
    Posted via http://www.ruby-forum.com/.
    Macario Ortega, Aug 9, 2009
    #15
  16. Macario Ortega

    Robert Dober Guest

    On Sun, Aug 9, 2009 at 3:24 AM, Charles Oliver
    Nutter<> wrote:
    > On Wed, Aug 5, 2009 at 9:56 AM, Brian Candler<> wrote:
    >
    > And again, that ignores that the AST is ever-changing and usually very
    > different across impls.

    I guess if I could really chose I would like a switch, and would like
    the VM to keep the AST iff that switch is set (-a|--ast).
    Goo{df} idea?

    Cheers
    Robert
    --=20
    module Kernel
    alias_method :=EB, :lambda
    end
    Robert Dober, Aug 9, 2009
    #16
  17. Macario Ortega

    Lui Core Guest

    Brian Candler wrote:
    > ...
    > I think that's the point :) You *have* to pass it a String source for
    > it to parse. You can't take an object which Ruby has already parsed and
    > installed, such as method:)foo) in this case, and get the sexp from the
    > object.


    One way to generate AST from live object is to use method_missing, which
    is applicable in 1.9 with pure ruby code.

    see
    http://rubyquiz.com/quiz95.html
    --
    Posted via http://www.ruby-forum.com/.
    Lui Core, Aug 9, 2009
    #17
  18. Macario Ortega

    Roger Pack Guest

    Roger Pack, Aug 10, 2009
    #18
  19. Robert Dober wrote:
    > On Sun, Aug 9, 2009 at 3:24 AM, Charles Oliver
    > Nutter<> wrote:
    >> On Wed, Aug 5, 2009 at 9:56 AM, Brian Candler<> wrote:
    >>
    >> And again, that ignores that the AST is ever-changing and usually very
    >> different across impls.

    > I guess if I could really chose I would like a switch, and would like
    > the VM to keep the AST iff that switch is set (-a|--ast).
    > Goo{df} idea?


    Unfortunately, by then it will already be too late. There's nothing
    the VM can do about it. That's exactly the reason why ParseTree
    doesn't work on YARV: the only component that needs the parsetree is
    the compiler. By the time the compiler hands off the code to the VM,
    the parsetree is long gone. The same applies to XRuby and Ruby.NET as
    well as the AOT modes of Rubinius, JRuby, IronRuby and MacRuby.

    And remember that the compiler and the VM need not be run by the same
    person, on the same machine or at the same time. In HotRuby, for
    example, the compiler is generally run by the web developer on the web
    server at development time whereas the VM is run by the user inside
    the browser on the client, maybe months after the code was compiled.

    jwm
    Jörg W Mittag, Aug 10, 2009
    #19
  20. Lui Core wrote:
    >> I think that's the point :) You *have* to pass it a String source for
    >> it to parse. You can't take an object which Ruby has already parsed and
    >> installed, such as method:)foo) in this case, and get the sexp from the
    >> object.

    >
    > One way to generate AST from live object is to use method_missing, which
    > is applicable in 1.9 with pure ruby code.
    >
    > see
    > http://rubyquiz.com/quiz95.html


    That's not proper Ruby; I don't see any loops or conditional branches. I
    suspect this approach requires you to solve the halting problem. e.g.

    sxp {
    for i in 3..1.0/0
    for j in 3..i
    for k in j..i
    for p in 3..i
    if i**p + j**p == k**p
    puts "fermat disproved: #{i}, #{j}, #{k}, #{p}"
    break
    end
    end
    end
    end
    end
    }
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Aug 10, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Nicolas Matringe

    Synthesizable (kind of) dual-edge FF

    Nicolas Matringe, Sep 27, 2004, in forum: VHDL
    Replies:
    1
    Views:
    927
    Ralf Hildebrandt
    Oct 12, 2004
  2. nb
    Replies:
    0
    Views:
    472
  3. Daniel Walzenbach
    Replies:
    5
    Views:
    405
    =?Utf-8?B?RGFuaWVsIFdhbHplbmJhY2g=?=
    Feb 3, 2004
  4. Guoqi Zheng

    how to convert this kind of date?

    Guoqi Zheng, Jan 27, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    282
    Ollie Riches
    Jan 27, 2005
  5. Elmo Watson

    Different kind of Checkbox question

    Elmo Watson, Apr 13, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    339
    Elmo Watson
    Apr 13, 2005
Loading...

Share This Page