Leo + Python: the ultimate scripting tool

[Edward K. Ream]

how different would Leo be if the outline were just directories and text
files? Would it be a set of file utilities and a kind of super-editor?

I'm glad you asked this question. There are several kinds of replies. I
suspect point 3 answers your question most directly, but points 1 and 2 seem
like necessary background:

1. Yes, you could simulate the effects I talk about in various ways.
Because directories are hierarchical, one can no doubt simulate almost any
kind of hierarchical operation using files and directories. Whether that
simulation is easy, safe, powerful and natural is another matter. It is
vastly easier (and safer) to create, delete and reorganize nodes in Leo
outlines than it would be to do the corresponding operations on files and
directories.

The main topic of my postings has been the happy interaction between Leo
outlines and scripts. These interactions would not be nearly so strong if
all hierarchies had to simulated via the file system. I used the term
"momentum" in one of the postings. Intellectual momentum is really
important, and we don't want to slow that momentum by simulating things
indirectly that can much more easily be done directly.

For example, I presume unit test would typically be organized in folders to
make it possible to create unit tests dynamically. And yes, you could use
something like grep to organize search and replace operations. However,
grep is just another wimpy re tool, so the simulation is weak.

2. Leo itself has many file-oriented features. In particular, Leo provides
a way of associating parts of outlines with so-called "derived files". Leo
can change derived files based on changes to Leo outlines. Very
importantly, Leo can do the reverse: Leo can change outlines based on
changes to derived files.

3. I have devoted a _huge_ amount of wondering, over a period of at least 10
years, about whether one could eliminate .leo files entirely, somehow
replacing .leo files with the union of all derived files. It turns out that
the answer must be "no". Without going into details, trying to organize
derived files without a .leo file would be like trying to work in an IDE
without a project file. The analogy isn't exact, but the conclusion is
inescapable: Leo needs .leo files. Moreover, Leo needs information _not_
contained in derived files in order to make derived files friendly to cvs.
In the 4.0 code base this extra information is the tnodeList attribute of
<v> elements of .leo files. Finally, it is just plain more convenient to
squirrel away information in an outline than it would be to create separate
files. In other words, we wouldn't want to give up .leo files even if it
were possible.

In short, there are extremely strong reasons why Leo works the way it does.
If you want more details, look at the detailed design discussions that have
appeared on Leo's SF forums over the past year or so. I'm always glad to
talk about Leo, so feel free to ask more questions if you like.

Edward

P.S. You question got me thinking along a new and amusing line: .leo files
simulate (in quite a few ways) a file system, with clones, for example,
acting much like shortcuts or links. Actually, implementing Leo's outline
operations is quite a bit more complex than implementing the core of a
unix-like file system. I can say this because I've done both.

EKR

 

[Aahz]

I do so for hours every day with no problem.

That's not what I said.

That's not to say that an ascii-only version of Leo might not be a
good thing, but rather that in this day and age, not having one should
hardly be a show-stopper.

That's your opinion. I disagree, at least to the extent that I won't
use any editing environment that doesn't have a text-only version.

 

[Edward K. Ream]

I won't use any editing environment that doesn't have a text-only version.

Your call, of course.

I just found the following at:

http://www.amk.ca/python/howto/curses/

In a world of X displays, one might ask ``why bother''? It's true that
character-cell display terminals are an obsolete technology, but there are
niches in which being able to do fancy things with them are still valuable.
One is on small-footprint or embedded Unixes that don't carry an X server.
Another is for tools like OS installers and kernel configurators that may
have to run before X is available.
[end quote]

It seems to me that simulating Leo on a character-cell display is going to
be all work and no gain. I have no interest in designing a dumbed-down
model for Leo that would work in paleolithic environments. Besides, Emacs
probably has a curses mode. Why would I want to compete with that?

So I am not going to do a text-only mode for Leo unless somebody comes up
with a pretty good reason why Leo would be an appropriate tool in
non-windowing environments. Starting up Leo in script mode is still on the
to-do list.

Edward

 

[Aahz]

It seems to me that simulating Leo on a character-cell display is
going to be all work and no gain. I have no interest in designing a
dumbed-down model for Leo that would work in paleolithic environments.
Besides, Emacs probably has a curses mode. Why would I want to compete
with that?

That's fine, although for me you're competing with vi[m]. There are two
reasons why I stick with my requirement for text-only systems:

* As I said earlier, I do much of my work from text consoles, and much of
that is also remote. I think trying to run GUIs over Net connections is
poor use of bandwidth. Not to mention the fact that until a month ago,
my only Net access was still direct-dial shell (yes, vt100 emulator, no
PPP). Even now that I do have DSL, I still do much of my GUI browsing
(when forced to use JavaScript) with images turned off.

* Requiring a text-only system is a reasonable proxy for requiring a
keyboard-centric system, because text-only systems by definition have to
work with a keyboard. Too many GUI-based systems have at least one
oddball corner that just doesn't work well with the keyboard, and it's
invariably one of those corners that I need to be productive.

This is an old argument, of course. I've been using the .sig I'm
including below for more than five years.

 

[Jegenye 2001 Bt]

One thing I mustn't forget to mention: Rodrigo B. is working on "Leo on the
Net" (LeoN for short). This will allow collaborative editing over the net
of a single .leo file. This is based on recent computer science research
and is leading-edge stuff. This is real. See the screen shots at:
http://ryalias.freezope.org/souvenirs/leon/et2003 I would like Rodrigo to
present his work at PyCon 2004. We shall see For more details search on
LeoN on Leo's SourceForge forums.

Much of the reason for the 4.x code base is to support such things.

Edward

Thanks, Edward, I'll check out this. But before that a little ranting..
Speaking of a multi-user version, security comes to mind.
Like you described, the scripts are omnipotent within the .leo files and
itself Leo.. and this is no problem if I use my own outlines.
However when it comes to collaborating and sharing outlines... well, you
know. (You just cannot check each and every line in that way cool outline
someone wrote.)
Then we have the Microsoft (macro virus) symptom.
So I wonder how this is going to be addressed with LeoN... This is not
easy, say, the Bastion/rexec thing was removed from Python...
Zope has its own access right management and that could be perhaps used
somehow and most importantly in a transparent way..

Best,
Miklós

 

[Kent Tenney]

I would welcome text mode Leo because I'm
interested recycling unwanted computers.

It was great to read; (http://lwn.net/Articles/53019/)
"""
William Lee Irwin recently tried the 2.6.0-test kernel on a system limited to 16MB of memory.
....
Andrew Morton says, "we should try to not suck in this situation." William's results indicate that
some work is still required for 2.6 to perform adequately on low-end hardware.
"""

Text mode Leo would not suck in this situation

Thanks,
Kent

It seems to me that simulating Leo on a character-cell display is
going to be all work and no gain. I have no interest in designing a
dumbed-down model for Leo that would work in paleolithic environments.
Besides, Emacs probably has a curses mode. Why would I want to compete
with that?

That's fine, although for me you're competing with vi[m]. There are two
reasons why I stick with my requirement for text-only systems:

* As I said earlier, I do much of my work from text consoles, and much of
that is also remote. I think trying to run GUIs over Net connections is
poor use of bandwidth. Not to mention the fact that until a month ago,
my only Net access was still direct-dial shell (yes, vt100 emulator, no
PPP). Even now that I do have DSL, I still do much of my GUI browsing
(when forced to use JavaScript) with images turned off.

* Requiring a text-only system is a reasonable proxy for requiring a
keyboard-centric system, because text-only systems by definition have to
work with a keyboard. Too many GUI-based systems have at least one
oddball corner that just doesn't work well with the keyboard, and it's
invariably one of those corners that I need to be productive.

This is an old argument, of course. I've been using the .sig I'm
including below for more than five years.

 

[Terry Reedy]

Edward K. Ream email: (e-mail address removed)

In response to this series, I sent you a friendly note saying a) I am
considering Leo as a base for a future project and b) I think there is
a problem with a couple of lines in the code you posted. In response,
you sent me the

----------
You recently sent a message to me at the email address
(e-mail address removed). To help cope with the ever increasing volume of
junk e-mail, I am using ChoiceMail, a permission-based e-mail
filtering tool. Your original e-mail is being held by ChoiceMail until
you complete the following simple one-time process.
Please click on the link
[Click here to request approval]
When your browser opens, fill in your name and a short reason for
wanting to send e-mail to me. If your reason is acceptable, your first
email and all subsequent e-mails from you will be delivered to me
normally.
------------

I will not beg you to read my bug report. Such a request is *NOT* a
legitimate 'anti-junk-mail' measure.

Being unable to communicate directly with you makes Leo less inviting
to me. If I were to adopt Leo for a project anyway, I would warn
users to not email the above address.

Terry J. Reedy

 

[Edward K. Ream]

Speaking of a multi-user version, security comes to mind.

Thanks for mentioning this. Security has been discussed at length on Leo's
SF forums. It is a _really_ serious problem for an environment as powerful
as Leo and Python. The general principle is: you (or your scripts) must
NEVER execute a script blindly. As you say, that is not so easy to do.
Certainly rexec would not help at all: a Leo script could create a time-bomb
and still be legal as far as rexec is concerned. The only solution is a)
know what scripts you are executing and b) know who you are working with.
Using Leo + Python is like using a chain saw: you had best keep it under
control, and you had best know what you are doing.

Edward

 

[Edward K. Ream]

Such a request is *NOT* a legitimate 'anti-junk-mail' measure.

Yes, it is. If you aren't willing to take 20 sec. to talk to me, then I'm
not willing to talk to you. Bye.

Edward

 

[Edward K. Ream]

Several responses mentioned gui issues, which have little to do with
scripting. However, gui issues are near and dear to my heart, so I thought
I would summarize how things stand with Leo and guis:

1. Leo is in the midst of a global reorg that will separate gui-dependent
code from a gui-independent "core". This is going well, and will be
completed in about a week. This reorg is the basis for the 4.1 code base.
4.1 final is due out in December; there will be a lengthy period of testing
(including many new unit tests) while the code settles down.

A prototype plugin for Leo already exists that replaces Leo's default
Tk/tkinter gui with a wxPython gui. The original version of the prototype
had to duplicate a large amount of Leo's core to be even partly functional.
With the new code base the wxPython gui only needs to know how to do generic
wxPython kinds of stuff. All knowledge of Leo's core is handled by base
classes that are in Leo's core.

In another reply I misspoke: I said that anygui probably wasn't up to the
job. In fact, with the new organization it would be dead easy to create a
plugin to support anygui instead of, say, wxPython. Experimenting with
anygui would be virtually risk free: it would take an hour or so of time.
N.B. Gui plugins are _completely_ responsible for handling all gui-related
events. In particular, the interface between Leo's core and gui plugins
ignores event handling completely. This greatly simplifies the interface
and ensures that no assumptions at all about event handling "infect" Leo's
core. I was originally dubious about matching anygui's model of events.
Now I see the fear was unfounded.

2. One of my hopes in writing the original series of postings was that
there would be some discussion of integrating Leo with Emacs using pymacs.
This may have started to happen. I would really like help with this
project, and I'll support anyone working on such a project in any way I can.
I think running some kind of Leo window on Emacs would be totally cool.

3. Several people have mentioned that it would be good to start up Leo in
scripting mode. There are several easy ways to do this. It will happen
eventually: don't know when. Right now it seems more like a cute feature
than something vital...In any event, there will eventually be a way of
running Leo without any gui at all: a null-gui plugin probably could do this
with about 10 lines of code.

Edward

 

[Edward K. Ream]

I forgot to mention that Leo already has plugins that allow fairly easy
integration with vim, xemacs and word. Another plugin allows full
customization of Leo's Open With menu. These plugins aren't perfect, and
they are a real good start.

These plugins were easy to write. For example, the guts of the xemacs
plugin is:

def open_in_emacs (tag,keywords):
if top():
top().frame.OnOpenWith(("os.spawnl",path,None),)

Tighter integration with emacs/xemacs would take more work, and it would be
well worth it. I hope that it will happen soon. The Aha for me came when I
saw tetris in an xemacs window. xemacs probably can host Leo and make Leo
look good. I see hints about "foreign" windows in the emacs docs, and I
haven't followed up on that yet...

Edward

 

[JanC]

Yes, it is. If you aren't willing to take 20 sec. to talk to me, then
I'm not willing to talk to you. Bye.

One question: why not let people do this "confirmation" through email
instead of requiring them to start a separate program? Something like a
mailing list confirmation mail?

 

[Terry Reedy]

One question: why not let people do this "confirmation" through email
instead of requiring them to start a separate program? Something like a
mailing list confirmation mail?

That is exactly what legitimate whitelist programs do. Click reply
and send. I have gotten such and answered such.

Terry J. Reedy

 

[Terry Reedy]

Yes, it is.

Your request is unnecessary to the purported purpose and, to at least
three people, obnoxious. Furthermore, I believe universal
implementation of that system would be terrible, both socially and
technically.

If you aren't willing to take 20 sec. to talk to me,

I spent much more than 20 seconds writing you a polite note for your
benefit. You are the one refusing to take even 5 seconds to look it.
Or to use a decent, unobstrusive or miminal imposition filtering
system.

then I'm not willing to talk to you. Bye.

That is your right, and mine, and probably our mutual loss is some
respects.
....

Edward K. Ream email: (e-mail address removed)

By posting your email address over and over, you are still inviting
responses that you will not read. Not nice. At least warn people
that they will have to fire up a browser a few days later and beg you
to read their original message. Or simply give the URL of your
'correspondant registration' service and suggest that people register
(or not) *before* writing.

Terry J. Reedy

 

[Douglas Alan]

Your request is unnecessary to the purported purpose and, to at least
three people, obnoxious. Furthermore, I believe universal
implementation of that system would be terrible, both socially and
technically.

Although I agree with you that requiring people to fire up a browser
to get yourself on their whitelist is a bit annoying, I also think you
make much ado about nothing. Even if you had to spend 30 seconds
authorizing yourself to everyone you ever needed to contact, it could
hardly take all that much time.

For now, the technique of just asking people to reply to a message is
sufficient as an anti-spam filter, but you have to realize that such a
convenience won't last forever. The spammers will eventually get
smart enough to auto-reply to such messages.

In the long run, the sorts of techniques that Yahoo, and other web
sites, where you have to look at a distorted image and identify it (a
task that cannot easily be automated), in order to put yourself on the
list, will be required, so you should probably get used to it now.

|>oug

 

[Nick Vargish]

Your request is unnecessary to the purported purpose and, to at least
three people, obnoxious.

At least four people.

Furthermore, I believe universal implementation of that system would
be terrible, both socially and technically.

Exactly, this is the root of why this spam-prevention tactic is
awful. It turns people who want to communicate with each other into
supplicants for for the other's time.

There are several ways to communicate as the leader of an open-source
project. One is of condescending arrogance, to wit, "My time is so
important that you must jump through hoops to speak to me." The other
is one of helpful cooperation. "Thanks, I'm really busy right now,
I'll address the issue you raised as soon as I can."

It's left as an excercise to the reader which category I see Edward's
tactic falling into. :^)

Nick

 

[Alexander Schmolck]

2. One of my hopes in writing the original series of postings was that
there would be some discussion of integrating Leo with Emacs using pymacs.
This may have started to happen. I would really like help with this project,
and I'll support anyone working on such a project in any way I can. I think
running some kind of Leo window on Emacs would be totally cool.

Have you come across allout.el and speedbar.el (a seperate tree browser
window)? Emacs has several outlining facilities amongst which allout is the
best IMO (I already use it for some of my python code) -- integrated with
speedbar this would supply quite a bit of the core functionality that leo
offers (plus, of course plenty additional functionality, most importantly good
editing and search facilities).

'as

 

[Stephen Boulet]

I see that allout.el is supplied with xemacs for windows. How does it
get activated?

Stephen

 

[Brad Clements]

_

"Terry Reedy" <[email protected]> writes:

There are several ways to communicate as the leader of an open-source
project.

Including posting in the source forge forum that was setup for that purpose,
rather than emailing the author directly, or posting in c.l.p

One is of condescending arrogance, to wit, "My time is so
important that you must jump through hoops to speak to me." The other
is one of helpful cooperation. "Thanks, I'm really busy right now,
I'll address the issue you raised as soon as I can."

Nick,

While I don't know you, I've read your posts in this forum and found them
all to be reasonable. I'm surprised that you expressed your opinion in such
harsh terms this time.

It's left as an excercise to the reader which category I see Edward's
tactic falling into. :^)

Some time ago I sent email to Edward and I received the "you must confirm
you're a human" thing. True, I thought it was annoying and in fact did not
immediately do anything about it. However we all have Spam issues and while
I wouldn't have taken Edwards approach to solving my spam problems, I
eventually expended 30 seconds to "suplicate myself to his spam software"
and get my message through.

I don't think it's supplication to the author, just his anti-spam guard dog.
I don't see Edwards statement (paraphrased) "if you want me to take time to
solve your problem, take a moment to confirm your a human" as arrogance,
just a fact of life.

Rather than writing to the author, you could just post in the Leo specific
forum, where your post could be answered by Edward or other users, and the
ensuing thread would be archived for other folks to later benefit from.

 

[Nick Vargish]

While I don't know you, I've read your posts in this forum and found them
all to be reasonable.

I'm pretty sure that's the nicest thing anyone's said to me all day. :^)

I'm surprised that you expressed your opinion in such harsh terms
this time.

Me too, actually. I should have just typed my message and then
killed the buffer without sending it, which is how I usually handle
irritations like this.

Rather than writing to the author, you could just post in the Leo specific
forum, where your post could be answered by Edward or other users, and the
ensuing thread would be archived for other folks to later benefit from.

I completely agree. It's too bad that Edward's automated reply did not
refer the supplicant to the appropriate sourceforge forum, which would
have been both non-arrogant, and of benefit to the community at large.

Only rarely have I been in situations where the appropriate face-to-
face response to a question was, "Talk to the hand until you've proved
your question is worth my time."

Anyway, I'm sorry I chimed in on this thread. It's diverged pretty far
from the topic I really care about, which is becoming a better Python
programmer (with a little advocacy thrown in for spice).

Nick