Leo + Python: the ultimate scripting tool

E

Edward K. Ream

I would like to say a few (actually more than a few) words here about some
recent discoveries I have made concerning the interaction of Leo and Python.
If you don't want to hear an inventor enthuse about his work, please feel
free not to read further :)

There are at least three, no four, no five, no six, no seven reasons why Leo
and Python work so well together. Most are new (in my mind) with the 4.1
release of Leo. The first several reasons may not seem like much. As you
will see, this appearance is completely misleading: the combination of
features is incredibly powerful.

To keep the individual posts shorter I shall list these 7 reasons and a
conclusion in "replies" to this posting.

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

1. You can execute Python scripts directly from anywhere in a Leo outline.

The body text of any node in Leo outline may contain any Python script. The
Execute Script commands executes this script in a pristine environment.
However, it is trivial to get access, _from within any script_, to all the
data in the present outline. For example: the following script prints all
the headlines in the outline in which the script is embedded.

from leoGlobals import c # Get access to all data in the present outline.
c = top() # Get the "commander" associated with the topmost outline.
v = c.rootVnode() # Get the root node of the outline.
while v:
...print v.headString()
...v.threadNext() # Move to the next node in "threading" order.

For that matter, scripts can access any data anywhere on your hard drive, or
all the data available on the web, for that matter. You knew that, of
course, but this "innocent" fact will be important later.

As a result, it becomes very easy to experiment with new scripts from within
Leo.

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

2. Leo's outlines are the perfect tool for organizing scripts.

This is an immediate result of the organizational power of Leo's outline.
However, it continues to surprise me how powerful this really is. For
example, at present, Leo has a "Scripts" menu that also organizes scripts.
It may well be that Leo's outlines alone, combined with features that I
shall shortly discuss, will make the scripts menu redundant.

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

3. Scripts may use the structure of Leo's outlines.

This is a first "big" reason why the combination of Python & Leo is
spectacular. As shown in the example above, scripts may get access to the
headline or body text of any node in a Leo outline. Therefore, scripts may
use the _structure_ of the outline to gain access to a) other scripts, b)
fragments of code for exec or eval or c) other data.

It gets better...

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

4. Leo's nodes naturally separate data from meta-data.

The body text of a node is the perfect place data itself (including other
scripts). The headline of the node is the perfect place to _describe_ the
data. That is, headlines are a natural place for meta-data. Scripts can
use this meta-data in many creative ways.

I first saw the power of points 2, 3 and 4 when devising regression tests
for Leo. This leads us to...

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

5. Leo is the ultimate platform on which to run regression tests.

The body text of a node is the perfect place for putting regression test
data. Headlines describe the data, which is exactly what is needed for
organizing tests. Indeed, I use a script to create regression tests based
on the structure of the outline of the children of the node containing the
script (!)

For example, I run the following script to create and run unit tests of
Leo's colorizer.

import unittest,leoTest
suite = leoTest.makeColorSuite("colorizerTests","tempNode")
runner = unittest.TextTestRunner()
runner.run(suite)

This works as follows: the arguments to makeColorSuite tell it to look up
the tree for of an ancestor of the present node (the node containing this
script) for a node whose headline is "colorizerTests". makeColorSuite then
looks for children of _that_ node for nodes containing test data. The node
whose headline is "tempNode" is used by the colorizer during the tests.
Here is the actual code of makeColorSuite:

def makeColorSuite(testParentHeadline,tempHeadline):
..."""Create a colorizer test for every descendant of testParentHeadline.."""
...u = testUtils() ; c = top() ; v = c.currentVnode()
...root = u.findRootNode(v)
...temp_v = u.findNodeInTree(root,tempHeadline)
...vList = u.findSubnodesOf(testParentHeadline)
...# Create the suite and add all test cases.
...suite = unittest.makeSuite(unittest.TestCase)
...for v in vList:
.....test = colorTestCase(c,v,temp_v)
.....suite.addTest(test)
...return suite

The power and simplicity of this code is breathtaking. All that is required
are two utility routines, findNodeInTree and findSubnodesOf that find
various nodes in an outline based on the outline structure and the headlines
of the nodes:

- findNodeInTree(root,tempHeadline) returns the node in the entire tree
contain the current node whose headline is tempHeadline.

- u.findSubnodesOf(testParentHeadline) returns a list of all children of the
node whose name is testParentHeadline.

In short, it is absolutely trivial for a script embedded in a Leo outline to
generate unit tests based on the data in the outline in which the script is
embedded!!

Notice, please, what does _not_ have to be done:

A. No unit tests are created statically: everything is driven by data (the
outline itself)
B. No Python test data is created _at all_. Before I saw this approach, I
was putting text for the syntax-coloring text in Python strings and passing
those strings to the regression tests. Now, the _unchanged_ data in the
body text of nodes becomes the regression test data.
C. There is no need to mark meta-data as separate from test data: headlines
are naturally distinct from body text.

We see now that Reasons 2, 3 and 4 are a lot more potent than they might
appear at first. Indeed, Leo plus Python is the ultimate merging of code,
structure and data. Scripts can use data organized in outlines. Outlines
organize scripts. Outlines (particularly their structure) are _data_ for
scripts. Outlines contains metadata for scripts. Outlines naturally keep
data and meta-data separate.

Please notice, this discussion merely hints at all the games that scripts
could play with outline structure. Just for example, scripts can easily
insert, delete and reorganize nodes. And the uses to which outline
structure may be put are practically unlimited, as we shall see later...

Let's turn now to a completely separate topic...

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

6. Leo is the ultimate platform for finding and changing text.

A few days ago I have added a script-based option to Leo's Find/Change
panel. For more than 30 years I have been disgusted with tools like regular
expressions. Calling re powerful is an utter joke; it's way too complex and
way too wimpy. Script-based find-change is infinitely more powerful.
Instead of containing find text or change _text_, Leo's Find/Change panel
may contains find or change _scripts. This becomes possible because scripts
have complete access to both the structure of the outline and all of Leo's
code.

It all works so smoothly: The find script is responsible for traversing
the tree and for highlighting the found text or otherwise indicating to the
change script what the found text was. Leo dedicates a Python dict called
app.searchDict for communication between the search script and the change
script. The change script is responsible for:

- changing the text, typically by using v.setBodyStringOrPane(newText), and

- implementing undo, typically by calling
c.frame.onBodyChanged(v,"Change",oldText=oldText).

Some details:

- When executing the search and change scripts when the user selects the
Find All or Change All commands, Leo executes the find or change scripts
repeatedly only if app.searchDict.get("continue") evaluates to True.
Otherwise these scripts are only executed once. Otherwise, scripts can use
app.searchDict as they please.

- For simplicity, most find and change scripts will ignore settings in the
Find Panel like "whole word", "pattern match", and "reverse": the scripts
know what to do! However, these settings are available to the scripts via
ivars such as c.whole_word_flag, etc. if desired.

- Either the find and change scripts may use Python's re module. For
example, the find script could set app.searchDict["m"] to the match object
returned by re's match method. The change script would then compute the
result, change the text and set the undo info as usual.

In effect, Leo's Find/Change panel becomes a new platform for running
scripts interactively. Leo now has all the find/change capability of
pattern matching languages like Snobol and Icon, using the capabilities of
the plain Python language. Moreover, Leo can deliver these capabilities
interactively or not depending which buttons you push in Leo's Find/Change
panel.

One last point. As I mentioned much earlier, there is no reason to confine
the find and change scripts to operate on data only within a Leo outline.
These scripts could as easily traverse your file system as the Leo outline.
But notice: scripts could pull data from the file system into the outline so
that you can see the effects of changes as the scripts operate. Again, this
can all happen interactively if you like.

No, one more last point. It's convenient to defined an initScriptFind
utility very similar to the makeColorSuite utility, used like this:

# Initialize Leo's find panel using the named children of this node.
from leoGlobals import *
initScriptFind("Find script","Change script")
# Start searching at the top.
top().selectVnode(c.rootVnode())

Put the search script in a child node called "Find script" (no quotes), put
the change script in a child node called "Change script", execute this
script and the following happens:

- The body text of "Find script" node gets put in the find text of Leo's
Find/Change dialog.

- The body text of "Change script" node gets put in the find text of Leo's
Find/Change dialog.

- The Script Find radio buttons gets selected, as does the Script Change
checkbox.

- The root of the outline becomes the selected node.

Presto! Leo is ready for a script search.

Edward
 
E

Edward K. Ream

Reasons why Leo and Python work so well together:

7. Leo outlines are the ultimate filing cabinet.

One more feature new in Leo 4.0 greatly expands the generality and power of
Leo's outlines combined with Python. Plugins may now attach arbitrary data
to any node of a Leo outline. This is done merely by "injecting" an ivar
called unknownAttributes ivar into Leo's fundamental vnode or tnode objects
(or adding items to the unknownAttributes dict if it already exists). The
unknownAttributes ivar should be a Python dictionary whose keys are xml
attribute names and whose values are the values of those attributes.

When writing a file, Leo will write "foreign" attributes in the <v> or <t>
xml elements of Leo's .leo file if the corresponding vnode or tnode contains
this unknownAttributes ivar. Similarly, Leo will create an
unknownAttributes ivar for a vnode or tnode if the corresponding <v> or <t>
element contains an attribute not normally used by Leo. Leo performs the
standard xml escapes when writing unknown attributes, so plugins can put
_anything_ in the unknownAttributes dictionary. In particular, plugins may
put xml in v.unknownAttributes or t.unknownAttributes.

This means, for example, that plugins may add anything that can be
represented by xml (and that is _everything_) in any node of a Leo outline.
In effect, Leo's file format is now completely extensible. BTW, plugins can
override any aspect of Leo's behavior, so not only can Leo passively accept
the data in "extended" .leo files, but plugins can extend Leo to _use_ this
data!

Edward
 
E

Edward K. Ream

Conclusion

All the power of Leo derives directly from the power, flexibility and
dynamism of Python. But making structure explicit as Leo outlines do adds
an important new dimension, literally and figuratively, to Python.
Moreover, Leo's Find/Change dialog creates a new environment from which to
run Python scripts. At long last the full power of the unification of code
and data is becoming apparent.

Edward
 
A

Aahz

All the power of Leo derives directly from the power, flexibility and
dynamism of Python. But making structure explicit as Leo outlines do
adds an important new dimension, literally and figuratively, to Python.
Moreover, Leo's Find/Change dialog creates a new environment from which
to run Python scripts. At long last the full power of the unification
of code and data is becoming apparent.

Sounds good. However, I won't try it until there's a shell-based mode;
I do too much of my work remotely.
 
E

Edward K. Ream

Sounds good. However, I won't try it until there's a shell-based mode;
I do too much of my work remotely.

Could you be more-specific about what you mean by "shell-based" mode? What
would have to be different in Leo?

Are you talking about a version of Leo that doesn't put up a gui? If so, a
null-gui plugin might suffice. If Leo can't do what you want already, I
suspect that adding this feature would be easy.

Edward

BTW, a null-gui plugin might only have to override the oops() method in
various gui base classes.

EKR
 
A

Aahz

That's what TightVNC is for.

Ugh. While I love what my new DSL can do, I don't think that running GUI
applications remotely is one of the things it *should* do, especially not
when text works so much better.
 
V

val

Ed,
Thanx so much for your thought-provoking,
stimulating and clarifying ideas/comments. It certainly
takes time to digest all you said. But the very first
comment is about significance of putting a *programming
activity* in the *structured* context. Until now,
a command line/shell and a self-centric (or process-centric)
program model was all your assets when you need to interact
and flexibly control the highly structured, multi-tier
(exploration) activities.
Say, in a model building environment, first you dream about
some fuzzy things, then begin chaotic actions and hopefully build
or generate a *trial* model (doc, design, etc); you then need
to run the model and evaluate its results against a certain reference
data/criteria, and based on that to modify the model (or switch
to a new one), and keep this cycle on and on in your multi-tier
automated, partially interactive exploration. It is like
a unit-testing activity (plus optional automatic mismatch-driven
debugging) with all necessary automation support at all tiers/domains.
For that one needs a sort of *structured* and flexible Explorer
with building, running, evaluating (including pattern recognition),
and powerful reconfiguration capabilities.
I guess you are moving in great direction offering
the unique tool/framework for exploration of complex systems.
Biology and bioinformatics (among many other domains) might gain
a lot using the Leo/Python great combo, a powerful expansion
of Python interpreting capabilities into exploration domain which
makes Python a dynamic exploration tool with structured and flexible
control/command capabilities.
explorative-ly y'rs,
val
 
E

Edward K. Ream

I mistakenly first replied to Aahz privately, so some of our conversation
has been off-line. I'd like to repeat it here publicly.

EKR> Could you be more-specific about what you mean by "shell-based" mode?
What would have to be different in Leo?

Aahz> It'd have to support a curses interface or something like it.

EKR> Are you talking about a version of Leo that doesn't put up a gui? If
so, a null-gui plugin might suffice. If Leo can't do what you want already,
I suspect that adding this feature would be easy.

Aahz> Well, I do want it to be screen-oriented. I just want it to be
text/console-based and keyboard-driven. You might be able to use anygui
to do that; haven't played with it.

EKR> Leo is being reorganized to support more than one gui. The work is
well along. I don't believe anygui is ready for prime time, and anyway the
new organization will suffice.

EKR> BTW, a null-gui plugin might only have to override the oops() method in
various gui base classes.

Aahz> Could be, but I don't want it enough to muck around with it. ;-)

EKR> Oh, I wasn't suggesting that you do that. I was mostly thinking out
loud, and maybe bragging that something that might be difficult in other
contexts has already been handled mostly in Leo's code base.

EKR> Thanks for this request. It is something I've never considered, and in
the big picture it will probably be easy enough to do. I'll put it on the
list and keep it in the back of my mind.

EKR> P.S. It might be good also to have an option that fires up Leo in
"script mode", something like:
python leo.py -script script.py <options>

which would open Leo, execute script.py and exit. Should be easy to do.

Edward
 
J

Jegenye 2001 Bt

Edward K. Ream said:
Could you be more-specific about what you mean by "shell-based" mode? What
would have to be different in Leo?

Are you talking about a version of Leo that doesn't put up a gui? If so, a
null-gui plugin might suffice. If Leo can't do what you want already, I
suspect that adding this feature would be easy.

Edward

No, I think he means using Leo with a text terminal a la browsing the web
with lynx.
Btw, a web based interface would be absolutely cool.. Do you see chance to
integrate Leo with, say, Zope?

Miklós

--
PRISZNYÁK Miklós
---
Jegenye 2001 Bt. ( (e-mail address removed) ... to send mail increment
the number and reverse the domainname.. )
Egyedi szoftverkészítés, tanácsadás
Custom software development, consulting
http://jegenye2001.parkhosting.com
 
E

Edward K. Ream

Thanx so much for your thought-provoking, stimulating and clarifying
ideas/comments.

You are welcome. This is an exciting time for me.
Say, in a model building environment, first you dream about some fuzzy
things...

I'm glad you mentioned this. It's so important to be able to explore
_easily_ while still being confused about what it is one is trying to do.
On Leo's SF Forum I described how I ended up creating the script-find and
script-change commands. I'll repeat it here, because it illustrates
something crucial about executing Python scripts in Leo.

As late as yesterday I wasn't sure how this would work, or if these new
options were even necessary. Here is a summary of what I did yesterday.

1. As usual, just being able to begin this project was very important. It
turned out that the ability to execute test scripts rapidly without leaving
Leo was crucial. I was able to run through many ideas quickly so that no
intellectual momentum got wasted.

2. I was quite confused about what was needed. After all, there are already
a lot of find/change scripts in scripts/leoFindScript.py. But these scripts
are not interactive. I wanted to see if I could write an interactive find
script without using Leo's Find panel. If I could, I might simplify the
entire process: no changes required to Leo's find/change code.

I soon discovered, however, exactly why Leo's Find panel is necessary. The
problem is that interactive scripts need a _nonmodal_ dialog always present
so that the user can choose "yes", "no" or "cancel". However, interacting
with that non-modal dialog would be quite a problem for the script. Rather
than reinventing the wheel, it is _so_ much easier just to use the Find
panel that already exists. And of course, it makes sense to do this.

3. So ok, we do, in fact, need Script Search and Script Change options in
Leo's Find panel. The next big big question was: how to do this? Always
present in my awareness was the fact the present find/change code is
complex. A few experiments with hacking into this code made me leery of
messing with this code. For one thing, the code is all Tk based.

4. I messed around with some more test scripts. Don't remember how it
happened, but finally a huge Aha happened: the Find/Change scripts in the
Find panel don't need any help at all from Leo! They can essentially do
everything on their own! Here is how I entered this Aha in my diary:

- Leo doesn't clear app.searchDict. However, a _script_ can a) init itself
and b) bring up the find window (Eureka!)

- The find script is responsible for traversing the tree.

- The find script is responsible for highlighting the found text or
otherwise indicating to the change script what the found text was. For
example, the re find script can set app.searchDict["m"] to the match object.

- The find script must communicate with the change script. Leo does
nothing.

- The change script must change the text, usually by using
v.setBodyStringOrPane.

BTW, the first Eureka seems almost irrelevant now, but somehow it was really
important at the time. Anyway, you can see that these notes clearly
indicate that Leo need not get intimately involved in "managing" the
search/change scripts. This basic principle is the real Aha, and it made
the rest of the work routine.

5. With this clear vision in mind, it became routine to add support for
find/change scripts. What I did:

a) Added support for new c.pattern_match_flag and c.pattern_search_flag
ivars. There was a digression until a fixed a config bug that was blocking
initing of these ivars.

b) Changed the top-level find/change code so that it simply calls code to
execute the scripts in the find/change text area if the corresponding ivars
were set.

That's all!!
[end quote]

The point is this: I did _not_ have a clear picture of how script-find and
script-change would work when I started, and moreover what picture I had was
confused and wrong. Yet somehow it was easy to noodle around until the
creative unconscious created an initial Aha. After that, it was smooth
sailing.
Biology and bioinformatics (among many other domains) might gain a lot
using the Leo/Python great combo.

I've thought so too. One thing I didn't mention in my "seven reasons" is
that Leo's outlines are isomorphic to dags (directed acyclic graphs). Dags
are much _more_ powerful than completely general graphs. Indeed, the
transitive closure of a general graph is just the entire graph again, so
Leo's clones would not be possible (would not be well-defined) in a general
graph.

Edward
 
E

Edward K. Ream

No, I think he means using Leo with a text terminal a la browsing the web
with lynx.

Whatever Aahz means, this idea has merit.
Btw, a web based interface would be absolutely cool..
Do you see chance to integrate Leo with, say, Zope?

There is some great work being done on Leo. I've been very interested in a
Leo + Zope combination, though I still don't understand what it means. Leo
as a "web app" would indeed be cool. There have been discussion of using
Flash plugins to host Leo as well...

One thing I mustn't forget to mention: Rodrigo B. is working on "Leo on the
Net" (LeoN for short). This will allow collaborative editing over the net
of a single .leo file. This is based on recent computer science research
and is leading-edge stuff. This is real. See the screen shots at:
http://ryalias.freezope.org/souvenirs/leon/et2003 I would like Rodrigo to
present his work at PyCon 2004. We shall see :) For more details search on
LeoN on Leo's SourceForge forums.

Much of the reason for the 4.x code base is to support such things.

Edward
 
D

Douglas Alan

Ugh. While I love what my new DSL can do, I don't think that running GUI
applications remotely is one of the things it *should* do, especially not
when text works so much better.

I do so for hours every day with no problem.

That's not to say that an ascii-only version of Leo might not be a
good thing, but rather that in this day and age, not having one should
hardly be a show-stopper.

|>oug
 
M

Mark Hahn

This may be a really off-the-wall question, but how different would Leo be
if the outline were just directories and text files? Would it be a set of
file utilities and a kind of super-editor?

It seems to me that a lot of Leo features can be found in standard
directories and files.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top