Ruby Code Parsing

[Jonathan Bale]

I have a Perl friend asking me questions about how ruby parses its code.
Things like, how does it know where a statement ends and where can you
and can't you put new lines -- basically trying to understand how Ruby
differs from Perl and from Python in regard to parsing Ruby code.

I have a good /practical/ understanding of how it works, but have
difficulty delineating in one e-mail all the rules of how it works. Is
there a particular document (please give specific section) on the
Internet that I could link him to that explains all the rules?

 

[Ryan Davis]

I have a Perl friend asking me questions about how ruby parses its = code.
Things like, how does it know where a statement ends and where can you
and can't you put new lines -- basically trying to understand how Ruby
differs from Perl and from Python in regard to parsing Ruby code.
=20
I have a good /practical/ understanding of how it works, but have
difficulty delineating in one e-mail all the rules of how it works. Is
there a particular document (please give specific section) on the
Internet that I could link him to that explains all the rules?

Well... You can look at ruby_parser.y in the ruby_parser gem. It is =
"only" ~1800 lines long. I wouldn't recommend it tho, except as evidence =
that it is statically parseable.

The long and the short of it is that it is somewhere between perl (which =
basically can't be statically parsed) and python (which I can't mentally =
parse (because of things like list comprehensions are anything but =
comprehensible to me), but which is highly regular).

 

[Caleb Clausen]

I have a Perl friend asking me questions about how ruby parses its code.
Things like, how does it know where a statement ends and where can you
and can't you put new lines -- basically trying to understand how Ruby
differs from Perl and from Python in regard to parsing Ruby code.

Statements generally end with a semicolon or newline... but not all
newlines are statement terminators. Newlines are considered whitespace
if preceded by an operator or some other token (eg: '(', '[', '{' )
which requires an expression following it. There are some other
special cases.

I have a good /practical/ understanding of how it works, but have
difficulty delineating in one e-mail all the rules of how it works. Is
there a particular document (please give specific section) on the
Internet that I could link him to that explains all the rules?

Ruby is designed to be easy for humans to read, which means it is not
easy for computers to parse. As opposed to say, perl, which is hard
for both humans and computers to parse. If you were to write down in
one email a complete set of rules, it would be a pretty long email.

You might take a look at the ruby draft standard, which is fairly
complete. For a lot of stuff, it just gives bnf, tho, with no verbal
description.

 

[Ryan Davis]

=20

=20
I find Perl pretty easy to parse. Maybe yours is a personal problem.

No, you do not find perl pretty easy to parse. Last time I checked, you =
can NOT statically parse perl. You have to evaluate it in order to get a =
proper parse. So, you find it pretty easy to evaluate. No easy task for =
me, I'll admit, but there is no reason for you to be an ass about it.=20

 

[Mark Thomas]

No, you do not find perl pretty easy to parse. Last time I checked, you can NOT statically parse perl. You have to evaluate it in order to get a proper parse.

There's a saying in the Perl community... "Only perl can parse
Perl." (capitalization intentional--that is, "perl" is the executable
and "Perl" is the language).

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

Regardless of nitpicky phrasing, I found your comment about Perl kind of
ass-ish. It seems kind of ironic to be called an ass for pointing out
that your problems with Perl may not be others' problems, when you make a
categorical deprecating statement about Perl.

I think Ryan was pointing out Perl can't be parsed:

http://www.perlmonks.org/?node_id=663393

 

[Caleb Clausen]

Regardless of nitpicky phrasing, I found your comment about Perl kind of
ass-ish. It seems kind of ironic to be called an ass for pointing out
that your problems with Perl may not be others' problems, when you make a
categorical deprecating statement about Perl.

First of all, do not confuse me with Ryan.

My statement about perl was perfectly accurate, in addition to being
deprecatory. Although perl has been an interesting experiment in
language design, it can at best be seen as a rough draft for ruby. No
one, not even its mother, has ever claimed perl is beautiful. Why do
you think the symbol of perl is a camel?

What I mostly find unreadable about perl code are the many uses of
sigils, particularly dollar signs which interrupt the flow of the
narrative in perl programs. The special funnily-named global variables
($_, $\, etc) are pretty ugly too. Of course, ruby has these as well,
but they just don't seem to be needed as much. The fact that some
variables sometimes start with a $ and other times with a @ is also
disturbing. None of these are the reasons that make perl difficult for
programs to parse; those are entirely different problems.

 

[Suraj Kurapati]

I have a Perl friend asking me questions about how ruby parses its code.

Is there a particular document (please give specific section) on the
Internet that I could link him to that explains all the rules?

How about the "To Ruby From Perl" mini-guide on the Ruby website?

http://www.ruby-lang.org/en/documentation/ruby-from-other-languages/to-ruby-from-perl/

It doesn't explain *all* the rules, but it's a worthy starting point.

Cheers.

 

[Ryan Davis]

=20
Regardless of nitpicky phrasing, I found your comment about Perl kind = of
ass-ish. It seems kind of ironic to be called an ass for pointing out
that your problems with Perl may not be others' problems, when you = make a
categorical deprecating statement about Perl.

There is nothing nitpicky about it. YOU _can't_ parse perl, yet you =
claim you can and that it is pretty easy.=20

If you think it is so easy, here is a challenge: write a parser for perl =
in ruby. It can't be that hard, right? I have one in ruby, and it "only" =
took 145 commits (so far) to do. I don't care what type of parser style =
you choose (LALR, RDP, PEG, etc), and I don't care if you choose to port =
someone else's parser to ruby... but you can NOT evaluate/interpret the =
code in order to determine your parse trees. It has to be a 100% static =
parse.

P.S. Even tho it was Caleb who made the "deprecating statement" about =
perl (nice double-entendre considering we're all still waiting for perl =
6... 10 years later), I totally agree with him on this one.=

 

[Ryan Davis]

=20
On Aug 23, 2010, at 08:10 , Chad Perrin wrote:
=20

12 , Chad Perrin wrote:
=20
On Mon, Aug 23, 2010 at 08:38:40AM +0900, Caleb Clausen wrote:
=20
Ruby is designed to be easy for humans to read, which means it is = not
easy for computers to parse. As opposed to say, perl, which is = hard
for both humans and computers to parse. If you were to write down = in
one email a complete set of rules, it would be a pretty long = email.
=20
I find Perl pretty easy to parse. Maybe yours is a personal = problem.
=20
No, you do not find perl pretty easy to parse. [...]=20
=20
Regardless of nitpicky phrasing, I found your comment about Perl = kind of
ass-ish. It seems kind of ironic to be called an ass for pointing = out
that your problems with Perl may not be others' problems, when you = make a
categorical deprecating statement about Perl.

=20
There is nothing nitpicky about it. YOU _can't_ parse perl, yet you
claim you can and that it is pretty easy. [...]
=20

You apparently think I'm saying my brain has an implementation of the
perl runtime installed on it. By that standard, none of us can parse
Perl, or Ruby, or Python, or C, or Scheme, or any other nontrivial
computer programming language.
=20
I'm just saying that I can look at some Perl code and figure out what
it's doing -- a necessary task if you're going to try to maintain some
Perl code.

And I'm "just saying" that you were being an ass to Caleb when you =
suggested his comment about perl to be a personal problem. His comment =
is a fact. Perl is difficult for both man and machine to parse. I'm also =
"just saying" that you're wrong when you said that you found perl pretty =
easy to parse. You don't. You can't. That's been pointed out several =
times and you dance around it. You look at perl and "figure out what =
it's doing", meaning: you EVALUATE it in your head in order to determine =
what the code is doing. If you don't, then you're not as good as you =
think you are and perl is a lot more difficult than you realize. (or as =
a possible alternative, you're only dealing with kindergarten-level =
perl--which is a good thing really.)

By the standard you think I'm setting, MANY of us can mentally =
statically parse ruby, python, most C, and especially scheme, just like =
MANY of us learned how to diagram natural languages like English in =
grade school.

In increasing order of difficulty to parse: scheme, python, C, ruby.

Notice that perl isn't on that list.