[OT] code is data

[Anton Vredegoor]

With the inclusion of ElementTree (an XML-parser) in Python25 and recent
developments concerning JSON (a very Pythonesque but somewhat limited
XML notation scheme, let's call it statically typed XML) Python seems to
have reached a stage where it now seems to be possible to completely
swallow lesser languages code, modify it, and spit out new source code
targeting the original language the code was written in, or even make a
translation to other languages.

The idea is that we now have a fast parser (ElementTree) with a
reasonable 'API' and a data type (XML or JSON) that can be used as an
intermediate form to store parsing trees. Especially statically typed
little languages seem to be very swallow-able. Maybe I will be able to
reimplement GFABasic (my first love computer language, although not my
first relationship) someday, just for fun.

Then there are things like cTypes (calling functions from native DLL's)
and PyPy (implementing Python in Python).

All this taken together, to me it starts looking like we're now entering
a territory that traditionally was exclusively in the Lisp domain.

Yes, Python had eval and exec for a long time already, and metatypes and
generators are having some strange unexplored possibilities too, but the
day will come soon (and at last when PyPy is reaching execution speeds
close to cPython) where Python will be able to swallow smaller
languages, and finally it will be able to swallow its own tail, like
Lisp but then more powerful (because of the widely used standard data
types and the code exchange between languages that that makes possible).

Your thoughts please.

Anton

 

[John Roth]

With the inclusion of ElementTree (an XML-parser) in Python25 and recent
developments concerning JSON (a very Pythonesque but somewhat limited
XML notation scheme, let's call it statically typed XML) Python seems to
have reached a stage where it now seems to be possible to completely
swallow lesser languages code, modify it, and spit out new source code
targeting the original language the code was written in, or even make a
translation to other languages.

When I heard of the new AST based compiler, I thought it
would finally be possible to extend things cleanly. Then
I learned that they weren't going to allow modification of
the AST.

There are a lot of things you can't do in source if the
language doesn't allow it. Python is pretty good compared
to other languages, but it's still not possible to create
new control structures with short circuit semantics.
And that's one example.

I saw the "make" statement as a breath of fresh air.
Then it got shot down for what were, to me, totally
trivial reasons. That's a second one.

Sigh.

John Roth

 

[Paddy]

With the inclusion of ElementTree (an XML-parser) in Python25 and recent
developments concerning JSON (a very Pythonesque but somewhat limited
XML notation scheme, let's call it statically typed XML) <SNIP

Your thoughts please.

Anton

Hi Anton.
If you mean this JSON: http://www.json.org/example.html
then I'd just point out that JSON isn't XML-like at all. In fact the
examples look like valid Python nested dictionaries.

- Pad.

P.S. This is good too: http://en.wikipedia.org/wiki/JSON

 

[Ravi Teja]

Hi Anton.
If you mean this JSON: http://www.json.org/example.html
then I'd just point out that JSON isn't XML-like at all. In fact the
examples look like valid Python nested dictionaries.

It is the same JSON. JSON is typically seen as a human friendly
replacement for some of the functions that XML is otherwise used for,
where the full blown XML spec is an overkill and JSON does not need
complicated parsers in some common languages because it can express
hierarchical data just like XML.

 

[Ravi Teja]

With the inclusion of ElementTree (an XML-parser) in Python25 and recent
developments concerning JSON (a very Pythonesque but somewhat limited
XML notation scheme, let's call it statically typed XML) Python seems to
have reached a stage where it now seems to be possible to completely
swallow lesser languages code, modify it, and spit out new source code
targeting the original language the code was written in, or even make a
translation to other languages.

The idea is that we now have a fast parser (ElementTree) with a
reasonable 'API' and a data type (XML or JSON) that can be used as an
intermediate form to store parsing trees. Especially statically typed
little languages seem to be very swallow-able. Maybe I will be able to
reimplement GFABasic (my first love computer language, although not my
first relationship) someday, just for fun.

Then there are things like cTypes (calling functions from native DLL's)
and PyPy (implementing Python in Python).

All this taken together, to me it starts looking like we're now entering
a territory that traditionally was exclusively in the Lisp domain.

Yes, Python had eval and exec for a long time already, and metatypes and
generators are having some strange unexplored possibilities too, but the
day will come soon (and at last when PyPy is reaching execution speeds
close to cPython) where Python will be able to swallow smaller
languages, and finally it will be able to swallow its own tail, like
Lisp but then more powerful (because of the widely used standard data
types and the code exchange between languages that that makes possible).

Your thoughts please.

I don't share your optimism at all. Most of the things you mentioned
have existed for long. Just because some of them are now included in
the standard library isn't going to change things drastically.
Installing them earlier was never hard at all.

People like to call everything with the lightest semblence, a DSL. That
gives the feel that the language is more powerful. Ruby people do it
all the time. Python cannot be called a DSL language until, creating
them is a natural language feature (like Lisp). And that does not seem
to be happening anytime soon. Boo for example allows you to write new
constructs with it's AST library. It still cannot be called a DSL
"language".

People have however written various language interpreters (Scheme,
Forth and yes, even Basic) in Python, just for kicks. Still does not
make it a DSL language anymore than it makes C a DSL language.

At present, the closest thing to writing a DSL in Python is Logix
http://livelogix.net/logix/
Too bad though, the project is defunct and there has never been enough
interest in it.

Personally, I would like to see macros in Python (actually Logix
succeeding is good enough). But I am no language designer and the
community has no interest in it. When I absolutely need macros, I will
go elsewhere.

 

[=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=]

Personally, I would like to see macros in Python (actually Logix

succeeding is good enough). But I am no language designer and the
community has no interest in it. When I absolutely need macros, I will
go elsewhere.

One must wonder, when is that? When do you absolutely need macros?

 

[Ravi Teja]

One must wonder, when is that? When do you absolutely need macros?

Whenever there is significant boiler plate code that functions and
classes cannot eliminate alone.
Whenever there is a more elegant way to express your code.

Python 2.5 introduced conditional expressions and with statement. With
macros, one would not have to wait for the language team to implement
them. More so for features which only a small part of the community has
an interest in.

I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
have done it myself for *my* code.

I would like special behaviour code blocks in my programs, for say DBC
(I am aware of the work arounds).

 

[Paddy]

Whenever there is significant boiler plate code that functions and
classes cannot eliminate alone.
Whenever there is a more elegant way to express your code.

Me, I am torn. I should now better. I have listened to the arguments
against Macros in Python and the ones that struck home were the
argument for maintainability:
Without macros, Python is Python. Statements do what you expect.

And the argument against DSLs altogether:
Make Python your DSL! If you design your own DSL before long you start
to embellish it with more statements or datatypes and before long it
becomes complex. If you used Python from the beginning then you would
have a community for support.

I know the arguments, but every once in a while I think if only I could
craft my own ??? statement or ....

Don't go their Paddy.

;-)

 

[Ravi Teja]

Me, I am torn. I should now better. I have listened to the arguments
against Macros in Python and the ones that struck home were the
argument for maintainability:
Without macros, Python is Python. Statements do what you expect.

Yes! I heard those arguments too. And I am not convinced.

Static language programmer: Lack of static typing removes the necessary
safeguards. The code is more error prone. Objects have behavior that is
not obvious.
Dynamic language programmer: Really? I don't seem to have any more bugs
than in my statically typed code. And my code is compact and reads
better. I don't want to go back.

No to macros proponent: Macros introduce a lot of potential for abuse.
Code will be worse to read than Perl.
Macros proponent: Really? We have been doing macros for decades. We all
think our code is better for macros, not worse. We are not going back.

I just don't get it. Don't we often invoke the "We are all adults here"
argument.

Writing a macro is not as simple as writing a function. Sort of like
metaclasses. Many will stay off them. Those that really need them will
walk that extra mile. Don't we all believe that "Simple should be
possible. Complex should be doable"

And the argument against DSLs altogether:
Make Python your DSL! If you design your own DSL before long you start
to embellish it with more statements or data types and before long it
becomes complex. If you used Python from the beginning then you would
have a community for support.

Python has a low cognitive overhead. But it not a DSL by definition. No
language can be. The idea is that when the domain changes, a DSL should
be driven by the new domain as warranted. In other words, driven "by
the problem, not the tool".

I don't want "a DSL". I want a language that allows me to make "my
DSL" based on it. That means I don't loose the community connection. I
can still use all the rich libraries in my DSL.

I like Python for its indentation syntax, sensible semantics and
readability. I invested a lot of time in Python. After much language
hopping, I settled with Python. I like the community and the code base
available for it. The libraries just seem to be designed at the right
level of abstraction for me (as opposed to say, Java). When I need to
do something, I know where to go. But all this ties me to the language
tightly that I cannot change.

I know the arguments, but every once in a while I think if only I could
craft my own ??? statement or ....

My thoughts exactly.

Web frameworks, which seem to be the rage now in Python community could
have benefited tremendously from Macro capabilities since they have a
lot of boiler plate.

 

[Fredrik Lundh]

Web frameworks, which seem to be the rage now in Python community could
have benefited tremendously from Macro capabilities since they have a
lot of boiler plate.

they do? methinks you haven't done much web programming lately...

</F>

 

[Ravi Teja]

they do? methinks you haven't done much web programming lately...

</F>

You blogged on Django. Let's use that. Don't you think model creation
in Django can be represented better, given that it is done often
enough?

Let's take an example from the official tutorial
from
http://www.djangoproject.com/documentation/tutorial1/#creating-models

class Poll(models.Model):
question = models.CharField(maxlength=200)
pub_date = models.DateTimeField('date published')

class Choice(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(maxlength=200)
votes = models.IntegerField()

I don't use Django and I made this up quickly, so please don't pick on
subtleties.

@Poll:
question: char length 200
pub_date('date published'): date

@Choice:
poll -> Poll
choice: char length 200
votes: int

The following is my rationale. Annoted variables, symbols and code
layout visually cue more efficiently to the object nature than do
explicit text definitions. Of course, this is only sensible when there
aren't too many of any of those. In that case, the cognitive cost of
notation outweighs the representational cost of text.

Representational minimalism is troublesome in general code (ala Perl),
but not so in a DSL where the context is constrained.

I would also like to symbolize field types since they occur so commonly
in a definition file and only a few of them are commonly used. I admit
though that I find the code below a bit visually jarring and I might
use something else. But it serves to illustrate the point. I chose the
respective symbols based on their colloquial use and association with
the field types.

@Poll:
$question: length 200
%pub_date('date published')

@Choice:
poll -> Poll
$choice: length 200
#votes

Since you are on thread and are a prominent and involved member of the
Python community, I would like it if you (or any such other) can
provide feedback on the rest of my previous post rather than be
dismissive by just a small portion of it. Perhaps, that will give me
some insight how these language design decisions are rationally made (I
am not strictly a programmer by profession, much less a language
designer).

 

[Roberto Bonvallet]

I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
have done it myself for *my* code.

You can write your own preprocessor to handle things like that.

 

[=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=]

community has no interest in it. When I absolutely need macros, I will

I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
have done it myself for *my* code.

I think this example more is a symptom of a childish need to get
things your way than of a deficiency in Python. BTW, range(5) = 0..4
in Ada and Ruby.

You said "when I absolutely need macros" but none of your examples
demonstrate any "absolute need." I can't see your point.

 

[bruno at modulix]

With the inclusion of ElementTree (an XML-parser) in Python25 and recent
developments concerning JSON (a very Pythonesque but somewhat limited
XML notation scheme, let's call it statically typed XML)

JSON stands for JavaScript Object Notation, and has *nothing* to do with
XML - except for the fact that it's more and more used instead of XML
for AJAX stuff.

Python seems to
have reached a stage where it now seems to be possible to completely
swallow lesser languages code, modify it, and spit out new source code
targeting the original language the code was written in, or even make a
translation to other languages.

If you mean "parsing source in a given format and outputting another -
modified or not - representation, in the same or another format", Python
as always been able to do so.

The idea is that we now have a fast parser (ElementTree) with a
reasonable 'API' and a data type (XML or JSON) that can be used as an
intermediate form to store parsing trees. Especially statically typed
little languages seem to be very swallow-able. Maybe I will be able to
reimplement GFABasic (my first love computer language, although not my
first relationship) someday, just for fun.

Then there are things like cTypes (calling functions from native DLL's)
and PyPy (implementing Python in Python).

All this taken together, to me it starts looking like we're now entering
a territory that traditionally was exclusively in the Lisp domain.

Sorry, but I just don't get the point. Parsing, working with trees and
calling native code are in no way "exclusively in the Lisp domain".

Yes, Python had eval and exec for a long time already, and metatypes and
generators are having some strange unexplored possibilities too, but the
day will come soon (and at last when PyPy is reaching execution speeds
close to cPython) where Python will be able to swallow smaller
languages, and finally it will be able to swallow its own tail, like
Lisp but then more powerful

I'm afraid Python is still very far from Lisp - and will never get there
(FWIW, this seems not to be the goal anyway).

(because of the widely used standard data
types and the code exchange between languages that that makes possible).

I still don't get the point.

 

[bruno at modulix]

One must wonder, when is that? When do you absolutely need macros?

One must wonder, when do you absolutely need HOFs, closures, OO,
functions, or even structured programming. All we 'absolutely' need is
tests and gotos... (and love, of course !-).

 

[bruno at modulix]

Whenever there is significant boiler plate code that functions and
classes cannot eliminate alone.
Whenever there is a more elegant way to express your code.

Python 2.5 introduced conditional expressions and with statement. With
macros, one would not have to wait for the language team to implement
them. More so for features which only a small part of the community has
an interest in.

I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
have done it myself for *my* code.

And that's the downside with macros - and with anything that's not
officially part of the language or it's standard lib : everybody
implements it it's own way, and you end up with dozens non-standard ways
of doing the same thing.

Not to say this is absolutely bad, but there's a balance to be found
here. One could do function decorators long before we had official
syntactic sugar for it, but it only started to be a common idiom with
the @decorator syntax. Python 2.5 introduces a 'partial' type, that is
quite easy to implement with 2.4 (and probably with older versions too),
but having it in the builtins or standard lib means it will become the
standard way to do it - no need to deal with half a dozen half-backed
implementations of it no more.

 

[bruno at modulix]

Ravi Teja wrote:

(snip)

Annoted variables, symbols and code
layout visually cue more efficiently to the object nature than do
explicit text definitions. Of course, this is only sensible when there
aren't too many of any of those. In that case, the cognitive cost of
notation outweighs the representational cost of text.

Representational minimalism is troublesome in general code (ala Perl),
but not so in a DSL where the context is constrained.

This still impose the need to learn a new language.

I would also like to symbolize field types since they occur so commonly
in a definition file and only a few of them are commonly used. I admit
though that I find the code below a bit visually jarring and I might
use something else. But it serves to illustrate the point. I chose the
respective symbols based on their colloquial use and association with
the field types.

@Poll:
$question: length 200
%pub_date('date published')

@Choice:
poll -> Poll
$choice: length 200
#votes

<IMHO>Yuck</IMHO>.

 

[Anton Vredegoor]

I still don't get the point.

Well, I've got to be careful here, lest I'd be associated with the
terr.., eh, the childp..., eh the macro-enablers.

The idea is to have a way to transform a Python (.py) module into XML
and then do source code manipulations in XML-space using ElementTree.

But rest assured, there is no such module, nor will we ever need it for
anything.

Anton

"use cases are for the faint-hearted"

 

[Michele Simionato]

I saw the "make" statement as a breath of fresh air.
Then it got shot down for what were, to me, totally
trivial reasons.

Which reasons? I as I recall, Guido cut it out without giving any
reason.
Of course Guido has the right to do so, but it is not respectful of
all the work people like Steven Bethard and others did :-(

Michele Simionato

 

[bruno at modulix]

Well, I've got to be careful here, lest I'd be associated with the
terr.., eh, the childp..., eh the macro-enablers.

The idea is to have a way to transform a Python (.py) module into XML
and then do source code manipulations in XML-space using ElementTree.

My my my... I'm not against the idea of dynamic source code
transformation, but for heaven's sake, *why* would one put XML in the
mix ???????