Open letter to anyone developing a Ruby IDE

[Adelle Hartley]

It has been said that features like "intellisense" or "autocomplete" are
difficult to implement for dynamic languages such as Ruby.

I'd be happy with an IDE that let me choose a .rb file per project, let's
call it "environment.rb" which would usually be a list of "require"
statements which would establish a run-time state that would be a fair
approximation of what classes and methods would be available to 90% of my
application.

I mean, sure, dynamic languages let us add/remove classes variable and
methods anytime we like, but during the first moments of my application's
startup I usually set up 90% of the classes that I will need during the
remainder of the program's execution.

Adelle.

 

[Lothar Scholz]

Hello Adelle,


AH> I mean, sure, dynamic languages let us add/remove classes variable and
AH> methods anytime we like, but during the first moments of my application's
AH> startup I usually set up 90% of the classes that I will need during the
AH> remainder of the program's execution.

The difficult thing is not really gathering of information. It is the
way how do you want to present the thousands of methods
found in your files and the runtime library.

Since type interference is quite impossible the only solution i came
up is some kind of a smalltalk class browser and a very clever
interface to find the right method with as less as possible shortcuts.
So any ideas here would be very helpful.

 

[Vincent Foley]

I don't know if people who develop FreeRIDE, ArachnoRuby or other
IDEs/text editors that want to include intellisense know Smalltalk, but
the eCompletion
<http://homepage.mac.com/monique_bakker/squeak/eCompletion.html>
"add-on" (for lack of a better word) available in Squeak is quite nice.
Maybe it could be a source of inspiration for Ruby IDE developers.

Vincent

P.S: Lothar, I'm interested by your choice of language for ArachnoRuby;
it's the first Eiffel software (that I know of) that I've used. Any
reason for going with Eiffel instead of something more dynamic like
Ruby|Python|Smalltalk|Lisp? How's developing with Eiffel too? Thanks
for the insights.

 

[Eric Hodel]

Hello Adelle,


The difficult thing is not really gathering of information. It is the
way how do you want to present the thousands of methods
found in your files and the runtime library.

Since type interference is quite impossible the only solution i came
up is some kind of a smalltalk class browser and a very clever
interface to find the right method with as less as possible shortcuts.
So any ideas here would be very helpful.

Uh....

Where exactly does type inference come in for an auto-completion
feature for an IDE?

Something so heavyweight as type inferencing is unnecessary for doing
90% of the work of figuring out what the likely completions for a
particular chunk of code you're writing.

It would be far easier to just run etags over all the files you require
and use those for auto-completion purposes.

vim has a workable solution for this (and I'm told emacs' solution is
nearly, if not actually, identical), it simply auto-completes words
that have the same beginning above your current location in the text
before switching to words from other buffers. (So if I typed
b-<TAB>-<TAB>-<TAB> here, I would get 'buffers' then 'before' then
'beginning' as possible completions.)

Extending this by reading files on require lines into etags would be
pretty easy, and you could use some weighted sorting to keep commonly
completed words only a few presses of tab away.

While this won't get an accurate completion every time, you really
don't care. You'll find that 90% of the time the word you want is less
than 3 presses of tab away.

 

[Ville Mattila]

While this won't get an accurate completion every time, you really
don't care. You'll find that 90% of the time the word you want is less
than 3 presses of tab away.

Lothar probably forgot that arachno ruby has this feature already 8).
Although it only uses tokens from current buffer only. But you still
get 80% accuracy with that. Autocomplete is bound to control-space.
Speeds up typing considerably.

- Ville

 

[Vincent Foley]

Unless I'm misunderstanding what Lothar is saying, there's no way to
give out only the methods that a particular object responds to. In
VisualWorks Smalltalk, when you use completion, you just get a list of
_ALL_ methods defined in the system beginning with your pattern.
There's nothinig in the code to let you know what class a particular
object is.

 

[Lothar Scholz]

Hello Eric,


EH> Where exactly does type inference come in for an auto-completion
EH> feature for an IDE?

EH> Something so heavyweight as type inferencing is unnecessary for doing
EH> 90% of the work of figuring out what the likely completions for a
EH> particular chunk of code you're writing.

EH> While this won't get an accurate completion every time, you really
EH> don't care. You'll find that 90% of the time the word you want is less
EH> than 3 presses of tab away.

Unfortunately this is not what people expect, at least when they come
from the Java world. Code Completition is a learning tool not
primarily to save some keystrokes. You slowly learn the available
messages for a class while writing your code. Therefore a much better
way then showing a huge list with hundrets of strings list should be
available.

And methods have the heuristic that they share a common
prefix and differ only the last characters. This makes your
(emacs/vi) completition even more useless.

Thats why for example Wing IDE for python is so successful. They even
encourage their users to use type asserts, which is IMHO a good thing.
As a result they can show some precise information.

 

[Adelle Hartley]

Hi,

The difficult thing is not really gathering of information.
It is the way how do you want to present the thousands of
methods found in your files and the runtime library.

Since type interference is quite impossible the only solution
i came up is some kind of a smalltalk class browser and a
very clever interface to find the right method with as less
as possible shortcuts.
So any ideas here would be very helpful.

I'm writing code against an ORM layer, so the attributes of my object
variables are defined in my database schema, not in my source code.

I think the IDE needs to be running a Ruby debugger and capture it's state
(which variables, constants, classes, methods, etc are 'visible' at a
particular point).

The problem with this approach is that when examining a particular line of
code, there may be several routes by which one might reach that line of code
at run-time.

My suggestion presupposes the use of a debugger (which would pick up classes
and attributes that my ORM layer gets from the database), but rather than
having it execute my application, having it execute the first part of my
application (where 90% of my classes are defined) and then stop before my
application actually does anything.

The IDE would take its table of classes+attributes from state of the
debugger, and then combine that with any local definitions in the
file/module/class that I'm working on.

Adelle.

 

[Eric Hodel]

It would be far easier to just run etags over all the files you
require and use those for auto-completion purposes.

vim has a workable solution for this (and I'm told emacs' solution is
nearly, if not actually, identical), it simply auto-completes words
that have the same beginning above your current location in the text
before switching to words from other buffers. (So if I typed
b-<TAB>-<TAB>-<TAB> here, I would get 'buffers' then 'before' then
'beginning' as possible completions.)

Extending this by reading files on require lines into etags would be
pretty easy, and you could use some weighted sorting to keep commonly
completed words only a few presses of tab away.

While this won't get an accurate completion every time, you really
don't care. You'll find that 90% of the time the word you want is
less than 3 presses of tab away.

Also, this[1] or something similar could probably get you those hints
you need, and again be accurate 90% of the time.

http://www.chadfowler.com/index.cgi/Computing/Programming/Ruby/
TypeWatching.rdoc,v

 

[Eric Hodel]

Hello Eric,


EH> Where exactly does type inference come in for an auto-completion
EH> feature for an IDE?

EH> Something so heavyweight as type inferencing is unnecessary for
doing
EH> 90% of the work of figuring out what the likely completions for a
EH> particular chunk of code you're writing.

EH> While this won't get an accurate completion every time, you really
EH> don't care. You'll find that 90% of the time the word you want is
less
EH> than 3 presses of tab away.

Unfortunately this is not what people expect, at least when they come
from the Java world. Code Completition is a learning tool not
primarily to save some keystrokes. You slowly learn the available
messages for a class while writing your code. Therefore a much better
way then showing a huge list with hundrets of strings list should be
available.

Have you used Readline tab completion in irb? It is far from useful,
giving you too many potential completions from too far up the
inheritance tree.

And methods have the heuristic that they share a common
prefix and differ only the last characters. This makes your
(emacs/vi) completition even more useless.

In the cases where I've found methods that share a common prefix, I've
found it rare that I'm actually calling those methods from regular
code. Test::Unit is the best example of this. I have many, many
methods starting with test_ but I never call them.

In the cases in Ruby where there are methods that share a common
prefix, #each comes to mind, I don't see how this is useless at all.

I type ea<tab> and see "each" then I hit tab again "each_line" and
again "each_with_index" which is the one I wanted. I don't see how
that's less useful than the ones I've seen in MS' VB editors.
(Besides, I'd never tab-complete 'each', all but the 'c' is on the home
row of a Dvorak keyboard.)

If you still think its totally useless I'd like to see a series of
keystrokes that show how it falls down.

FWIW, ri says there's the following possible each completions:
Core: #each #each_line, #each_with_index, #each_value, #each_pair,
#each_key, #each_byte, #each_object, #each_pair
Pathname: #each_entry, #each_filename
YAML: #each_document, #each_node
Vector: #each2

You can only get those last 5 by requiring extra libraries.

 

[Guest]

http://www.chadfowler.com/index.cgi/Computing/Programming/Ruby/
TypeWatching.rdoc,v

It would be nice to see this somehow integrated in IDE's, but this still
means the code needs to be executed somehow before the typehints are
available, not?

Regards,

Peter

 

[Eric Hodel]

It would be nice to see this somehow integrated in IDE's, but this
still means the code needs to be executed somehow before the typehints
are available, not?

From Chad's description, I believe that is correct.

... unit tests work great for this ...

 

[Lothar Scholz]

Hello Eric,


EH> From Chad's description, I believe that is correct.

EH> ... unit tests work great for this ...

But unit tests sometimes use other objects (Mock classes) then the
"normal" classes. So the results can be misinterpreted.

But even with unit tests the application under development will
normally run a thousand times during a development week.

 

[Jussi Jumppanen]

Extending this by reading files on require lines into etags
would be pretty easy

Zeus for Windows uses ctags as it's source of tags information
and this is used to provide a limited form of auto-completion.

But unfortunately providing good intellisensing is far from
simple. The reason is the information produced buy ctags is
class based and looks very nice in a class browser window.

But the programmer works with instances of those classes and
the hardest part of implementing good intellisensing is
correctly identifying to which class the current instance
belongs.

This is not as simple as it may seem as it depends a lot on
the current scope. For example a local variable, a global
variable, and a member variable can all have the same name
yet all be of different class. But of the three only one will
be correct and which one depends on the current scope.

Jussi Jumppanen
Author of: Zeus for Windows (New version 3.94 out now)
"The C/C++, Cobol, Java, HTML, Python, PHP, Ruby folding editor"
Home Page: http://www.zeusedit.com

 

[Tim Sutherland]

It has been said that features like "intellisense" or "autocomplete" are
difficult to implement for dynamic languages such as Ruby.

I'd be happy with an IDE that let me choose a .rb file per project, let's
call it "environment.rb" which would usually be a list of "require"
statements which would establish a run-time state that would be a fair
approximation of what classes and methods would be available to 90% of my
application.

I mean, sure, dynamic languages let us add/remove classes variable and
methods anytime we like, but during the first moments of my application's
startup I usually set up 90% of the classes that I will need during the
remainder of the program's execution.

How do you know which class is associated with a particular variable?

a = somemethod(1, 2, 3)
a.<TAB> # what methods are available? We have an approximation of
# the classes/methods in the system, but no idea what
# `a' corresponds to.
#
# You could use "Foo.new is an 'atomic' type constraint,
# everything else bubbles up from these kind of constraints",
# but how reliable would that be?

.... although if you had decent unit test coverage, you could use the unit
tests to guess what variables probably correspond to.

"When we ran the unit tests, `a' was an instance of Foo, so assume for code
completion that `a' is a Foo."

+ do some simple inference from e.g. Foo.new.

 

[Rob .]

It has been said that features like "intellisense" or "autocomplete" are
difficult to implement for dynamic languages such as Ruby.

I've got *naive* method completion working in a development version of
the jEdit Ruby Editor Plugin. Not that difficult to implement after
all ...

In jEdit you can create file projects using the Project Viewer. I use
JRuby to parse the static contents of Ruby files in all projects and
create a cache of classes/modules/methods.

For completion I infer the possible classes of a variable, based on
what methods have already been called on it. Then I provide a popup of
the combined set of methods available for those classes, i.e. if it
quacks like a duck, it will probably waddle like one two.

Early days yet, but naive type based method completion is on the way.
I hope to have a working version out in a week's time; in the meantime
version 0.1 of the Ruby plugin for jEdit is available at:
http://www.jedit.org/ruby/

Rob

 

[Shajith]

Hi!

a = somemethod(1, 2, 3)
a.<TAB> # what methods are available? We have an approximation of
# the classes/methods in the system, but no idea what
# `a' corresponds to.
#
# You could use "Foo.new is an 'atomic' type constraint,
# everything else bubbles up from these kind of constraints",
# but how reliable would that be?

What about the IDE doing eval("a.methods")? Won't that cover the
general <expr>.<TAB> case?

Shajith

 

[Rob .]

What a nifty and simple heuristic! How well has it worked so far?

It works. Obviously the more unique method names are, the better the
type matching will be. Now I have a class/method cache implemented in
Java the performance issue has gone away. Previously I was doing
multiple calls out to ri from Java, which was very slow.

Outstanding technical issues include keeping the cache uptodate as the
code is being edited, and recognizing parent class methods and mixins.

As to whether it's useful, not sure, haven't done any Ruby development
with such a feature available before. I tend to think that navigation
features like, "find usages" and "find declarations" can prove more
beneficial during development.

Will your completion list also show parameter names (just display, not
insert into buffer)? That would also be a nice touch.

Yes, the completion list shows parameter names - it's inserting them
too at the moment.

 

[Florian Gross]

What about the IDE doing eval("a.methods")? Won't that cover the
general <expr>.<TAB> case?

What if a is `rm -rf /`? It might not always be possible to execute code
as it might have side effects which I think is also the reason for IRB's
rather simplistic method completion...

 

[Zed A. Shaw]

Weird, I was just working on this problem this morning in the index
feature for fastcst. I have some code right now which indexes the
source using suffix arrays and then lets you quickly search for stuff in
the source. It could easily be "stolen" and turned into a code
completion thing I think. Here's some of the results of doing
"index .join" on my source:

/lib/fastcst/operation.rb:
"File.join(target, path)"
/lib/fastcst/command/finish.rb:
"File.join(@repo.work_dir, UNDO_JOURNAL)"
/lib/fastcst/command/list.rb:
"\#{readable_path.join('/')} ]\""
/lib/fastcst/command/disp.rb:
"File.join(@repo.work_dir, MetaData::META_DATA_FILE)"
/lib/fastcst/command/merge.rb:
"File.join(@repo.work_dir, MetaData::META_DATA_FILE)"

Which is basically all the places where a .join "thing" is used, and
probably enough information to figure out what possible parameters
the .join takes.

The "completion" could then simply be when you start to type a certain
sequence (say "." followed by another char) it would do a search like
this, collapse "similar" stuff and present that as the completion. Neat
thing is this works with any source, not just Ruby, and requires no
parsing.

This is also fairly quick even though I'm storing the index in a
serialized hash table.

Anyway, is this kind of what you're looking for?

Zed

Where exactly does type inference come in for an auto-completion
feature for an IDE?

<snip>