C
Caleb Clausen
[Note: parts of this message were removed to make it a legal post.]
rubylexer version 0.7.0 has been released!
http://rubyforge.org/projects/rubylexer
RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant
as a lexer for Ruby that's complete and correct; all legal Ruby
code should be lexed correctly by RubyLexer as well. Just enough parsing
capability is included to give RubyLexer enough context to tokenize
correctly
in all cases. (This turned out to be more parsing than I had thought or
wanted to take on at first.) RubyLexer handles the hard things like
complicated strings, the ambiguous nature of some punctuation characters and
keywords in ruby, and distinguishing methods and local variables.
This version (long overdue) fixes most known minor problems with the
previous release. The output is now more suitable for use by a parser.
The next major goal is the Big String Reform which will remove the
tokens-inside-a-token hack needed to support string inclusions
currently.
Changes:
### 0.7.0/2-15-2008
* implicit tokens are now emitted at the right times (need more test code)
* local variables are now temporarily hidden by class, module, and def
* line numbers should always be correct now (=begin...=end caused this) (??)
* fixed meth/var-name confusion in idents after 'def' but before params
* FileAndLineToken now emitted on all new lines (0.8)
* offset of __END__ now works(??)
* put files into lib/* eof in unusual cases is better handled
* improvements in parsing unary * and & (??)
* input can now come from a string
* local vars (defs and uses) are recognized in string inclusions
* explicitly decimal numbers, eg: 0d123456789
* eof in unusual cases is better handled
* __END__ is not treated as a keyword
* '|' as goalpost is now better handled
* a number of things have been renamed internally
* no more implicit parens for setter method calls
* '{' after return, break, and next is now the start of a hash.
* ambiguous cases of '::','(',':',and '?' are now better handled.
* more start/end hint tokens (for 'when', 'rescue', and 'for')
* bugfixes in rhs hint tokens
* parsing of def headers for singleton methods is improved
* rescue as operator is now handled
* block param list lookahead is simplified
* unary ops (including * and &) can be easily distinguished in output
* here document bodies better handled, esp after escaped newline
* symbols like %s{symbol} now actually work
* implicit parens around => better handled...
* different cases of '{' can now be easily distinguished
* ImplicitParamList Start and End are now Keyword, not Ignore tokens.
To get rubylexer, type:
gem install rubylexer
Or download:
http://rubyforge.org/frs/download.php/32769/rubylexer-0.7.0.tgz
project homepage:
http://rubyforge.org/projects/rubylexer
rubylexer version 0.7.0 has been released!
http://rubyforge.org/projects/rubylexer
RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant
as a lexer for Ruby that's complete and correct; all legal Ruby
code should be lexed correctly by RubyLexer as well. Just enough parsing
capability is included to give RubyLexer enough context to tokenize
correctly
in all cases. (This turned out to be more parsing than I had thought or
wanted to take on at first.) RubyLexer handles the hard things like
complicated strings, the ambiguous nature of some punctuation characters and
keywords in ruby, and distinguishing methods and local variables.
This version (long overdue) fixes most known minor problems with the
previous release. The output is now more suitable for use by a parser.
The next major goal is the Big String Reform which will remove the
tokens-inside-a-token hack needed to support string inclusions
currently.
Changes:
### 0.7.0/2-15-2008
* implicit tokens are now emitted at the right times (need more test code)
* local variables are now temporarily hidden by class, module, and def
* line numbers should always be correct now (=begin...=end caused this) (??)
* fixed meth/var-name confusion in idents after 'def' but before params
* FileAndLineToken now emitted on all new lines (0.8)
* offset of __END__ now works(??)
* put files into lib/* eof in unusual cases is better handled
* improvements in parsing unary * and & (??)
* input can now come from a string
* local vars (defs and uses) are recognized in string inclusions
* explicitly decimal numbers, eg: 0d123456789
* eof in unusual cases is better handled
* __END__ is not treated as a keyword
* '|' as goalpost is now better handled
* a number of things have been renamed internally
* no more implicit parens for setter method calls
* '{' after return, break, and next is now the start of a hash.
* ambiguous cases of '::','(',':',and '?' are now better handled.
* more start/end hint tokens (for 'when', 'rescue', and 'for')
* bugfixes in rhs hint tokens
* parsing of def headers for singleton methods is improved
* rescue as operator is now handled
* block param list lookahead is simplified
* unary ops (including * and &) can be easily distinguished in output
* here document bodies better handled, esp after escaped newline
* symbols like %s{symbol} now actually work
* implicit parens around => better handled...
* different cases of '{' can now be easily distinguished
* ImplicitParamList Start and End are now Keyword, not Ignore tokens.
To get rubylexer, type:
gem install rubylexer
Or download:
http://rubyforge.org/frs/download.php/32769/rubylexer-0.7.0.tgz
project homepage:
http://rubyforge.org/projects/rubylexer