[ANN] rubylexer 0.7.3 Released

Discussion in 'Ruby' started by Caleb Clausen, May 1, 2009.

  1. Rubylexer version 0.7.3 has been released!

    * <http://rubylexer.rubyforge.org/>
    * <http://rubyforge.org/projects/rubylexer/>

    RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant
    as a lexer for Ruby that's complete and correct; all legal Ruby
    code should be lexed correctly by RubyLexer as well. Just enough parsing
    capability is included to give RubyLexer enough context to tokenize correctly
    in all cases. (This turned out to be more parsing than I had thought or
    wanted to take on at first.) RubyLexer handles the hard things like
    complicated strings, the ambiguous nature of some punctuation characters and
    keywords in ruby, and distinguishing methods and local variables.



    Changes:

    ### 0.7.3/4-19-2009
    * 9 Bugfixes:
    * remember whether comma was seen in paren context
    * reducing the warning load
    * remember whether we're in a method def when starting a recursive lexer
    * (so that class vars can be marked as being in a method if in #{})
    * make sure Token#to_s always outputs _something_ halfway sensible
    * make sure funclike keywords come out as a keyword
    * break/next/return method after . was not detected properly
    * need a NoWsToken before :: in names of compound modules
    * abort implicit paren contexts when elsif seen
    * all files should be world-readable now

    * 9 Minor Enhancements:
    * move test data into lib/ dir so I can get to it better from redparse
    * split parse_keywords into separate methods for each keyword
    * (I had hoped to speed up keywords, but keyword_def is still slow)
    * more of the lists of keywords are now available as arrays as well
    * (parenthesis) KeywordToken needs to know if it's for grouping or call
    * minor speedups in newline and ident
    * don't panic if rubygems not available
    * make linenum publicly available in RubyCode
    * various other little helper methods needed by redparse in Tokens
    * hack Rakefile so 'rake test' will stay in 1 process (keeps netbeans happy)

    * <http://rubylexer.rubyforge.org/>
    * <http://rubyforge.org/projects/rubylexer/>
     
    Caleb Clausen, May 1, 2009
    #1
    1. Advertising

  2. Caleb Clausen

    Marc Heiler Guest

    Re: rubylexer 0.7.3 Released

    > RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is
    > meant as a lexer for Ruby that's complete and correct; all legal Ruby
    > code should be lexed correctly by RubyLexer as well.


    Would this allow one to create a "mini-irb", i.e. something that can
    interprete ruby code like irb does, but without the need to reuse that
    undercommented irb code?
    --
    Posted via http://www.ruby-forum.com/.
     
    Marc Heiler, May 2, 2009
    #2
    1. Advertising

  3. Re: rubylexer 0.7.3 Released

    On 5/2/09, Marc Heiler <> wrote:
    >> RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is
    >> meant as a lexer for Ruby that's complete and correct; all legal Ruby
    >> code should be lexed correctly by RubyLexer as well.

    >
    > Would this allow one to create a "mini-irb", i.e. something that can
    > interprete ruby code like irb does, but without the need to reuse that
    > undercommented irb code?


    So, irb has its own little lexer it uses internally, mostly so it can
    tell whether you've typed a complete line of code or not. I presume
    that's what you're referring to...? Irb's lexer could fairly easily be
    replaced with RubyLexer, or you could write a whole new implementation
    based on it.

    I know irb's lexer is incomplete in various ways, but it seems to be
    much better than it was
    when I started writing RubyLexer. RubyLexer isn't totally perfect yet
    either... it certainly isn't very readable... so now you've replaced
    undercommented irb code with undercommented RubyLexer code... ;)

    Is this answering your question? I'm not sure.
     
    Caleb Clausen, May 3, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Caleb Clausen

    [ANN] rubylexer 0.7.0 Released

    Caleb Clausen, Feb 21, 2008, in forum: Ruby
    Replies:
    0
    Views:
    99
    Caleb Clausen
    Feb 21, 2008
  2. Caleb Clausen

    [ANN] rubylexer 0.7.1 Released

    Caleb Clausen, Sep 2, 2008, in forum: Ruby
    Replies:
    1
    Views:
    78
    Roger Pack
    Oct 8, 2008
  3. Caleb Clausen

    [ANN] RubyLexer 0.7.4 Released

    Caleb Clausen, May 22, 2009, in forum: Ruby
    Replies:
    0
    Views:
    90
    Caleb Clausen
    May 22, 2009
  4. Caleb Clausen

    [ANN] rubylexer 0.7.6 Released

    Caleb Clausen, Jul 7, 2009, in forum: Ruby
    Replies:
    0
    Views:
    78
    Caleb Clausen
    Jul 7, 2009
  5. Caleb Clausen

    [ANN] RubyLexer 0.7.7 Released

    Caleb Clausen, Jan 4, 2010, in forum: Ruby
    Replies:
    0
    Views:
    103
    Caleb Clausen
    Jan 4, 2010
Loading...

Share This Page