Lisp development with macros faster than Python development?..

?

=?iso-8859-1?Q?Fran=E7ois?= Pinard

[Raymond Hettinger]
With Lisp or Forth, a master programmer has unlimited power and
expressiveness. With Python, even a regular guy can reach for the
stars.

A few years ago, I much hesitated between Scheme and Python as my next
day-to-day programming language.

My feeling at the time was that Scheme is a very fast language to write
into, and in which one can implement new concepts cleanly and compactly.
Maybe Python is a bit slower to write, but this is compensated by the
fact Python is more legible when it comes to later maintenance, or when
many people have to share work on a big set of sources.

There is some heaviness and complexity in Python internals, Scheme are
purer and simpler by comparison. On its bright side, Python has a nice
and comprehensive library, and an interesting community of users. These
probably make most of the difference.
 
D

Dan Sommers

On Wed, 06 Jul 2005 08:27:55 -0500,

[ reusable, stable, debugged, and documented libraries are a Good
Thing ]

Absolutely.

Two related stories from my days working as a software engineer for a
large telecomm company. Both stories begin with the annual ritual of
management telling us that code reuse was the silver bullet of software
development.

1. We replied that we needed about 5 calendar years to design, develop,
debug, and document sufficient libraries to pay for the effort. Then
management would tell us that 5 years is too long, and the project(s)
would be scrapped. I worked there for 12 years, and was still watching
new hires argue over linked list code and build tools when I left.

2. Right after that (and often in the same speech), management would
also tell us that they would be measuring our productivity by (a) LOC
written rather than LOC reused, and (b) the number of new ideas and code
we submitted for patent. There is no better example of a Mixed Message.

(Not to mention that we did all or most of our coding in C and C++, and
every department and every project had its own method(s) of determining
what a Line Of Code looked like, but that's even farther off-topic....)

Regards,
Dan
 
J

jayessay

I've been reading the beloved Paul Graham's "Hackers and Painters".
He claims he developed a web app at light speed using Lisp and lots
of macros.

That was the original "yahoo store".

It got me curious if Lisp is inherently faster to develop complex
apps in.

For complex applications IMO/E there is no question that it is. But
that is due to a lot of reasons, not only macros.

It would seem if you could create your own language in Lisp using
macros that that would be quite an advantage....

Here's the story. It has long been known that "domain specific
languages" (DSL) are a significantly more potent means of producing
applications (or chunks of them) in their domains than any so called
"general purpose" language. This is true even if the DSL is not a
particularly great piece of work. There are several examples of this
that most everyone knows about (though they may not realize it), with
"regular expressions" and SQL being among the most obvious.

You can take this further to "application specific languages" (ASL)
which provide even more leverage (though even narrower focus).
Obvious examples of this sort of thing would include AutoCAD and the
"grammar languages" of LALR parser generators.

The reason these things are so much better (for what they do) than
hacking away in your general purpose language is that the abstractions
they provide are much closer to what you want to say. They are much
more _declarative_ than procedural. You say _what_ you want done,
instead of specifying a lot of how that will produce the what [1].

Lisp (by which I mean here Common Lisp) macros enable you to build
DSLs and ASLs directly into the base language. This has several
advantages beyond what the DSLs/ASLs themselves bring; some obvious
ones being:

* You don't have to write lexers and parsers, that part is provided to
you for "free" by the Lisp reader[2]. In particularly sophisticated
cases you may need to alter the reader behavior (via the readtables)
and/or build a code walker to analyze things. But that would be
several sigmas out of the norm.

* Better yet, you don't have to write backend code generators. The
expansion code of a macro just gets compiled by the Lisp compiler
(typically an optimizing native code generator).

* The DSL/ASL code can be seamlessly used with the base language (and
libraries) just like when you use proprietary class libraries you've
built from within the base language. Only here, you get a much
higher level of abstraction and expressive power. So, you can
freely use a DSL/ASL where it applies (and get all its advantages),
but can effortlessly move to the base language when stepping outside
its focus.

* Better yet, in highly complex application scenarios you can have a
number of DSLs/ASLs each bringing their own high expressive power
and move freely among them and the base language. This should all
work naturally together in concert.

* Most idioms can be abstracted away and the bugs associated with them
disappear as well. You don't have to remember sequences of "right,
in these cases I first have to do this, then I have to call this
that and the next thing in this order, and finally make sure I end
it all with this bit over here."



Their are a few standard red herings trotted out against all this.
For example, "you mean you have a different programming language for
each application! That's awful." or some variant.

If you think about it for a moment, this is complete nonsense. Why?
Because you need to do this anyway - building and learning a set of
class libraries for an applicaion or domain (along with all its idioms
as well) is actually much more work, more painful, and (typically)
buys you less. The point is, you need to learn the "vocabulary" of
application or domain code whether or not you use DSLs/ASLs. Worse,
without the DSL/ASL approach you also need to learn a number of idioms
of how to _procedurally_ make use of the resources to achieve _what_
you want.

Another red hering goes something like: "it is much harder to learn a
DSL/ASL than to simply use your base general purpose language". Most
programmer types espousing this are really weird, because in the next
breath they will be extolling the virtues of regular expressions, or
list comprehension or some other DSL that just happens to already be
embedded in their favorite language. Also, it makes no sense anyway -
DSLs and ASLs are exactly the kind of thing provided to lay people at
the user end of many application scenarios. Are they really somehow
that much superior to learning these things than programmers?

Another one is "fragmentation". This is really a kind of variant on
the first, but it is used so often that it deserves its own mention.
Basically the argument is that if you build DSLs and ASLs eventually
your base language "fragments" and you have a few dozen languages
instead. A few seconds thought and it is easy to see that this is no
different than having a number of class libraries and in both cases
the idea of "framenting" the base language is nonsense.


OTOH, there _are_ issues associated with the DSL/ASL approach and Lisp
macro programming generally. IMO, the _key_ thing is to realize that
you don't want to have joe/jane-avg-hack, building these things,
because they will likely *$*%! it up. You don't have to be a genius
or something, but you have to be a _good_ designer and someone who
knows how to keep ergonomics and elegance front and center in the
process. The other thing is (again IMO, others will definitely
disagree with this) that you should focus on getting people who
know something about language design and compilation/translation to do
this work. You certainly need a clear understanding of the
differences between compile time, load time and runtime and what is
going on at each of these stages to do the best work here.

The best way to structure this is to have a group solely devoted to
building the DSL/ASL, with input from the domain experts, other high
level developers, and even some from joe/jane-avg-hack. The result is
then used by the application developers proper, including
joe/jane-avg-hack. This can and probably should be a process of
continual refinement. The entire thing can even be collapsed into one
individual (for smaller projects), but you need to be keenly aware of
what hat you have on at any given point.

Even so, it should be fairly clear that this isn't that much different
from what you would want to do in building your applications core
class system. It just has different areas of emphasis and expertise.

I realize that Python has operator overloading and OOP so I'm not sure.

This is simply orthogonal to the whole issue. Lisp has the most
potent object system available in any extent programming language.
While "operator overloading" (aka "static dispatch" or "compile time
dispatch") is controversial in some circles, it is quite easy to
provide with some macrology as well.

Any *evidence* one way or another?

There's plenty of evidence, but as with any software productivity
issue, it's mostly anecdotal. Hard scientific evidence for such
things is basically non existent and the economics of such things
pretty much guarantee it is going to stay that way.

-----------------------------------

1. Someone recently remarked that good Lisp macros are basically
executable pseudo code. I think that is pretty much exactly right
and is a pretty good "sound bite" distillation of what it is all
about.

2. Or, coming at from another angle, you don't have to figure out how
to cobble together a bunch of extraneous applications (lexers,
parers generators, code generators, etc.) into the one you are
building.


/Jon
 
S

Scott David Daniels

François Pinard said:
My feeling at the time was that Scheme is a very fast language to write
into, and in which one can implement new concepts cleanly and compactly.
Maybe Python is a bit slower to write, but this is compensated by the
fact Python is more legible when it comes to later maintenance, or when
many people have to share work on a big set of sources.

There is some heaviness and complexity in Python internals, Scheme are
purer and simpler by comparison. On its bright side, Python has a nice
and comprehensive library, and an interesting community of users. These
probably make most of the difference.

Well said. Writing speed is not everything; if it is, APL and Scheme
win (and the evil Perl for string processing).

--Scott David Daniels
(e-mail address removed)
 
R

Rahul

Hi.
Instead of listing the difference ..here are some things that are
COMMON to both common lisp and python :

1.Dynamic typing.
2.garbage collection
3.powerful data structures
4.good object systems. (here people from lisp usually claim that clos
is the most powerful object system...but i think python and lisp are
comparable with none of them being better than the other. they are
different than each other. and i consider clos without mop to be
inferior. mop stands for meta object protocol...a thing which hasnt
been specified in the ansi lisp standard but is provided by most
implementations)
5.functions and types as first class objects
6.interactive development.

The differences:

1.Macros : Macros are extremely powerful and a double edges sword. Dont
believe anyone (whether they praise them or abhor them). Go and learn
them and decide for yourself.

2.Readability : Python is generally believed to be far more readable
than ANY other language. (Lisp isnt particularly unreadable).

The only way to really find out which is better is to learn both and
decide yourself.

i personally like python , lisp and c. now c evokes derision from both
python and lisp folks sometimes and thats incorrect too.
rahul
 
G

Gregory Bond

Rocco said:
Actually, Google's answer to that question is something called "ILOG
CPLEX",

We use this. It's a library / command line tool (not a language) for
doing optimization - linear programming, quadratic programming,
mixed-integer programming etc. Very cool and very, very scarey.
 
A

Antoon Pardon

Op 2005-07-07 said:
????

Almost sounds like a racist comment - sorry if I misunderstood

I'll clarify. A lot of the time I hear arguments against
features that boils down to.

1) I don't need it.

2) Having the feature will make my job more difficult.

3) I don't understand the merrits of the feature or I
have difficulty understanding what it does when I
encounter it.

IMO these are arguments if followed sufficiently will lead to
a language that is only usefull for mediocre programmers.
 
T

Terry Reedy

jayessay said:
1. Someone recently remarked that good Lisp macros are basically
executable pseudo code. I think that is pretty much exactly right
and is a pretty good "sound bite" distillation of what it is all
about.

Several years ago I remarked that Python reads like executable pseudocode.
I still think that that is pretty much right.

Googling, I discovered that the creators of some thing I had never heard of
said the same thing about *their* language a couple of years ago. I wish
web pages, like newgroup posts, were dated so one could better trace the
history of such usages.

Terry J. Reedy
 
T

Terry Reedy

Antoon Pardon said:
I'll clarify. A lot of the time I hear arguments against
features that boils down to.

1) I don't need it.

2) Having the feature will make my job more difficult.

3) I don't understand the merrits of the feature or I
have difficulty understanding what it does when I
encounter it.

IMO these are arguments if followed sufficiently will lead to
a language that is only usefull for mediocre programmers.

True. But these arguments did not stop expert-level features like
metaclasses, decorators, and (in 2.5) context-management blocks
(with-blocks).

TJR
 
R

Robert Kern

Terry said:
Several years ago I remarked that Python reads like executable pseudocode.
I still think that that is pretty much right.

Googling, I discovered that the creators of some thing I had never heard of
said the same thing about *their* language a couple of years ago. I wish
web pages, like newgroup posts, were dated so one could better trace the
history of such usages.

Trawling through http://web.archive.org might help.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
 
J

jayessay

Terry Reedy said:
Several years ago I remarked that Python reads like executable pseudocode.
I still think that that is pretty much right.

Googling, I discovered that the creators of some thing I had never heard of
said the same thing about *their* language a couple of years ago. I wish
web pages, like newgroup posts, were dated so one could better trace the
history of such usages.

In the context discussed, the idea was the pseudo code was a _direct_
match to the description of the task in the _domain_. If your domain
is "algorithms" or some such, then I would agree Python would work as
a reasonably decent pseudo language, otherwise no. It's too low
level. Same with base CL. It's too low level.

/Jon
 
M

Mike Meyer

Antoon Pardon said:
I'll clarify. A lot of the time I hear arguments against
features that boils down to.

It seems that you've lost some of the intent during the boiling.
1) I don't need it.

Is that what you get out of the oft-used "What's your use case"?
That's an attempt to find out what problem the proposer thinks the
feature would help solve, so said solution can be evaluated in
comparison to possible existing solutions.
2) Having the feature will make my job more difficult.

Well, if the job is "writing programs" or "debugging programs", then
that's a *good* reason for not adding the feature. Those jobs are hard
enough as it is. Any feature that makes them noticably harder needs to
provide some serious advantages in order to be justified.
3) I don't understand the merrits of the feature or I
have difficulty understanding what it does when I
encounter it.

I don't think I've seen the first one used as an argument against a
feature. I may well have missed them, as I don't follow all such
discussion completely. The second one is a variation on #2, above.

Adding features just because they are "cool" is not the right way to
grow a language. New features need to enhance the language in a
clearly positive manner, while having as little negative impact as
possible. Point 1 and the first part of 3 are cases of people who
don't see a real enhancement to the language. Point 2 and the second
part of 3 are cases where people see a serious negative impact from a
feature.

When people *quit* pointing out those kinds of problems with
proposals, then I'll be worried. Python will be on it's way to
becoming Perl 6 or Common LISP, rather than the powerful, elegant
language I've come to love.

Bringing up such objections also gives the proposer a chance to
correct the problem, which is to everyones advantage. Witness the
recent (short) discussion of adding dynamically scoped variables. As
originally proposed, they can produce very obscure and hard-to-find
bugs. The fix for this made them no worse than global variables.
IMO these are arguments if followed sufficiently will lead to
a language that is only usefull for mediocre programmers.

Unless you're arguing that Python is already a language that is only
usefull for mediocre programmers, then this is sort of moot. Those are
arguments being advanced against *adding* features, not for removing
them (at least, I don't think I saw any of them in the ongoing
discusion re map/filter/reduce/lambda). Or are you arguing that Python
will somehow become less useful as new features are proposed and
rejected?

<mike
 
K

Kay Schluehr

I've been reading the beloved Paul Graham's "Hackers and Painters".
He claims he developed a web app at light speed using Lisp and lots
of macros.

Yes, Paul is a postmodern hero who reininvents himself and his language
every day and with every program:

"Experienced Lisp programmers divide up their programs differently. As
well as top-down design, they follow a principle which could be called
bottom-up design-- changing the language to suit the problem. In Lisp,
you don't just write your program down toward the language, you also
build the language up toward your program. As you're writing a program
you may think "I wish Lisp had such-and-such an operator." So you go
and write it. Afterward you realize that using the new operator would
simplify the design of another part of the program, and so on. Language
and program evolve together."

http://www.paulgraham.com/progbot.html

Remark: The same may be claimed about Forth.

This might be a great self experience for some "great hackers" but just
annoying for others who used to work with modular standard librarys and
think that the border of the language and an application should be
somehow fixed to enable those.

Kay
 
K

Kirk Job Sluder

Kay Schluehr said:
This might be a great self experience for some "great hackers" but just
annoying for others who used to work with modular standard librarys and
think that the border of the language and an application should be
somehow fixed to enable those.

In what way do lisp macros prevent the creation of modular libraries?
Common Lisp does does have mechanisms for library namespaces, and in
practice a macro contained within a library is not that much different
from a function contained in a library or a class contained in a
library.

Macros just provide another mechanism for creating useful
domain-specific abstractions. The primary advantage to macros is that
you can create abstractions with functionality that is not easily
described as either a function or a class definition.
 
K

Kay Schluehr

Kirk said:
In what way do lisp macros prevent the creation of modular libraries?
Common Lisp does does have mechanisms for library namespaces, and in
practice a macro contained within a library is not that much different
from a function contained in a library or a class contained in a
library. Macros just provide another mechanism for creating useful
domain-specific abstractions.

To be honest I don't understand what a "domain-specific abstraction"
could be? What is the benefit of abstractions if they are not
abstracting from particular domain specific stuff?
The primary advantage to macros is that
you can create abstractions with functionality that is not easily
described as either a function or a class definition.

As long as macros are used to create new language features such as an
object system like CLOS this technique may be perfectly justified for
language developers ( ! ) but I still consider it as a bad idea to
muddle the language development and the application development, that
seems to be the favourite programming style of Paul Graham. On the
other hand thinking about language development as a certain application
domain I find nothing wrong with the idea that it once reaches somehow
a state of a mature product that should not be altered in arbitrary
manner for the sake of a large user community.

Kay
 
K

Kirk Job Sluder

Kay Schluehr said:
To be honest I don't understand what a "domain-specific abstraction"
could be? What is the benefit of abstractions if they are not
abstracting from particular domain specific stuff?

The usual trend in higher level languages is to abstract away from the
algorithmic details into domain-specific applications. So for example,
rather than writing a block of code for handling the regular expression
'[a-zA-Z]+, then a different block of code for the case, 'a|b|c', we
have a regular expression library that packages up the algorithm and the
implementation details into an interface. The python standard library
is basically a collection of such abstractions. In python you usually
work with strings as an object, rather than as an array of byte values
interpreted to be linguistic characters located at a specific memory
address as you would in c.

Object oriented programming is all about creating domain-specific
abstractions for data. This enables us to talk about GUIs as widgits
and frames in addition to filling in pixels on a screen. Or to talk
about an email Message as a collection of data that will be stored in a
certain format, without having to do sed-like text processing.
As long as macros are used to create new language features such as an
object system like CLOS this technique may be perfectly justified for
language developers ( ! ) but I still consider it as a bad idea to
muddle the language development and the application development, that
seems to be the favourite programming style of Paul Graham. On the
other hand thinking about language development as a certain application
domain I find nothing wrong with the idea that it once reaches somehow
a state of a mature product that should not be altered in arbitrary
manner for the sake of a large user community.

Well, at this point, Common Lisp has been formally standardized, so
changing the core standard would be very difficult. There is in fact,
strong resistance to reopening the standards process at this time, based
on the impression that most of what needs to be done, can be
accomplished by developing libraries. So I think that CL as a mature
product is not altered in an arbitrary manner.

However, from my view, quite a bit of development in python involves
adding new language constructs in the form of classes, functions, and
instance methods, as well as interfaces to C and C++ libraries. I would
argue this is one of the core strengths of python as a language, the
fact that we are only limited to the builtin functions and standard
library if we choose to be. As an example, whenever I work with a new
data source, I usually end up creating a class to describe the kinds of
records I get from that data source. And some functions for things that
I find myself repeating multiple times within a program.

Macros are just another way to write something once, and use it over and
over again.
 
T

Thomas Bartkus

David

That is what I thought too. It makes sense but I wasn't sure. Still
ain't.
The problem is that questions like 'What lang is fastest to develop
in?'
are hard to answer definitively.

No it's not.

The answer is always whatever language you enjoy the most and know the best.

That's a somewhat redundant statement because if you enjoy a language, you
are highly motivated to use it often and learn it well.

An intimate knowledge of any particular language is *far* more important
than the syntactic and semantic arcana that people usually argue over.

So - what the question really boils down to is which language(s) have the
best balance of approachability (easy to learn!) and capability. Bear in
mind though, that if the language sacrifices capability in favor of being
easy, then the fun runs out of it too soon :)

Thomas Bartkus
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top