[OT] code is data

[Ravi Teja]

Or... maybe to be more specific, the hard work later on goes into

*code*. If you are enhancing your model, you do so with methods on the
model classes, and those methods don't effect the DSL, they are just
"code". You create some raw XML in the beginning, but quickly it's
just a matter of gluing those pieces together, using functions instead
of DSLs, and that's just "code".

That doesn't look that much better. How do you create it
programmatically? I know how to pass a variable to
CharField(maxlength=200); can I pass a variable to "char length 200"
just as easily? Can I use **kw? Can I read it from a CSV file and
construct the class that way? Maybe, but only by recreating all the
native patterns that I can infer easily looking at the Django class.

I am looking at it from the cognitive perspective. You are looking at
it from the compiler perspective.

I think you are talking about full blown DSLs like SQL which try to be
self contained for a given domain. The ones I am referring are only
thin layers on Python.

Words are great. Python is light on symbols, and that is good.

Agreed. When I came to Python from Perl, I loved the clean syntax.
Scalars, arrays, hashes occur too frequently in Perl code that using
symbols to denote them causes more noise than cognitive assistance. On
the other hand, using symbols to denote an occational special construct
is helpful (as in decorators).

Even the Lisps stick to an incredibly homogenous syntax (far more
homogeneous than Python) to make macros feel familiar.

Yes! The parser friendly, "everything is a list" syntax does help. I
did consider that to be an essential feature to enable dynamic macros.
However I changed my mind when I saw Logix macro syntax.

Constrained context is a step backward! How do you add methods? How
do you do looping? How do you write *code*? If you aren't going to
allow those things, then just make a parser and build the structure
from the file, and make it a DSL implemented entirely external to
Python. That's completely okay, though in my experience it's not very
satisfying for something like a model definition (see MiddleKit for an
example of an ORM that doesn't use Python code).

I agree that constrained context is a step back in terms flexibility.
But it is a strategic step backwards, in this case to trade for
representational benefits. The extent of constraints is a judgement
call. And proof of utility can only be emperical.

However I think that you are seeing my sample differently than I meant
it. I did not mean to create a special syntax file that would be parsed
as a text file such that it would loose all the benefits of Python. It
is just a thin layer over Python code for specific representational
benefits. Kay Schluehr does a good job of identifying it as such in his
reply.

 

[Ian Bicking]

I am looking at it from the cognitive perspective. You are looking at
it from the compiler perspective.

I don't think that distinction is very meaningful. As a programmer I
have to understand both. I have to be able to synthesize correct
expressions. I need to understand the compiler. I understand the
Python compiler well, and it gives me reasonably good feedback when I
get things wrong, and it has a lot of flexibility along several
orthogonal lines.

We're talking about programming languages, so it's useless to consider
a cognitive perspective without considering the compiler perspective.

I think you are talking about full blown DSLs like SQL which try to be
self contained for a given domain. The ones I am referring are only
thin layers on Python.

I understand you, but it's very unclear to me how you can make a thin
layer that's not horribly leaky or stunted. In my experience one or
both are likely in DSLs, and the result is horrible and only useful as
a toy.

If you are really serious about the implementation, sure. But DSL
almost screams out that it lacks seriousness, and a long-term
commitment to debuggability, generality, and documentation.

However I think that you are seeing my sample differently than I meant
it. I did not mean to create a special syntax file that would be parsed
as a text file such that it would loose all the benefits of Python. It
is just a thin layer over Python code for specific representational
benefits. Kay Schluehr does a good job of identifying it as such in his
reply.

I understood the distinction you were making. But you were also
speaking generally about generally programmable syntax, and I don't
think that's a good idea, and it's very unclear how that mixes with
Python. You can always do:

model = make_model("""
my funny syntax
""")

And you actually get something that can be a close peer to normal
Python code. But if you are talking about Python and some novel syntax
interleaved, then the details seem to matter a lot, because the
implementation is substantial and potentially invasive.

There are also very *specific* things that can be discussed that are
more conventional, and driven as much by the ease of implementation as
expressiveness. I think that is much more productive, because I think
the general case is a bad idea. The 'make' syntax is an example of
this.

Ian

 

[Anton Vredegoor]

You mean like 'converting' javascript to python or python to ruby (or
converting any home-grown DSL to Python, etc) ?

Yes, but also what some other posters mentioned, making Pythons internal
parsing tree available to other programs (and to Python itself) by using
a widely used standard like XML as its datatype.

count me in then

Sorry about that.

No way you will escape from your responsabilities so easily !-)

Ok, count me back in then too Of course I will be available for
further discussion. If more than ten people demand a PEP and no better
champion is available (very unlikely) I'll even write a proposal.

Anton

 

[Paul Boddie]

Yes, but also what some other posters mentioned, making Pythons internal
parsing tree available to other programs (and to Python itself) by using
a widely used standard like XML as its datatype.

http://pysch.sourceforge.net/ast.html

I was going to write a long reply to one of your previous messages, but
the above link references a project which may intersect with some of
your expectations. Meanwhile, it should be noted that the availability
of Python AST processing tools is not a recent thing: the compiler
module has been around for a long time, and it is possible to modify
the AST and to generate bytecode from it; my own experiments have
centred on producing other representations from the AST, and other more
successful projects (eg. ShedSkin) produce other languages (eg. C++)
from the AST.

Paul

 

[Ravi Teja]

I don't think that distinction is very meaningful. As a programmer I

have to understand both.
I understand the Python compiler well, and it gives me reasonably good feedback when I
get things wrong, and it has a lot of flexibility along several
orthogonal lines.
We're talking about programming languages, so it's useless to consider
a cognitive perspective without considering the compiler perspective.

As some one who has followed your excellent articles, posts and tools
over the last many years, I don't doubt your expertise at all.

I admit I am not be entirely looking at this as a programmer. I am
working in informatics for the last few years. At least in this domain,
it is useful to seperate the perspectives. Sort of like designing a
functional language based primarily on mathematical abstractions rather
than imperative realities. Yes! Once the implementation is underway,
the practical realities are unavoidable but they interfere in thinking,
otherwise.

I understand you, but it's very unclear to me how you can make a thin
layer that's not horribly leaky or stunted. In my experience one or
both are likely in DSLs, and the result is horrible and only useful as
a toy.

I am not making this up out of thin air. There is already an
implementation, Logix, which allows you make thin layer DSLs on top of
itself (which in turn sits on top of Python). It did not fly but I
could not technically fault it (except maybe that it is not very
optimized at the moment). Maybe with your better understanding of
language implementations, you can.

I understood the distinction you were making. But you were also
speaking generally about generally programmable syntax, and I don't
think that's a good idea, and it's very unclear how that mixes with
Python. You can always do:

model = make_model("""
my funny syntax
""")

And you actually get something that can be a close peer to normal
Python code. But if you are talking about Python and some novel syntax
interleaved, then the details seem to matter a lot, because the
implementation is substantial and potentially invasive.

NO! I have a feeling that you are interpreting me as a language
radical. I do not think that Python syntax should have too many
novelties. Currently Python feels a cohesive whole. A well designed
entity rather than something that evolved by accretion of features like
Perl. With active attempts to prune inconsistencies for that end
(Python 3000). And I value all that.

But Logix does not add ANY new syntactic constructs to Python. There is
a special import function that imports a macro supporting Python
language variant module where one can create special syntactic
constructs. I am quite satisfied with the macro capabilities of Logix.
But I am just grumbling that the Python community does not find macros
interesting. But without enough users, it simply dies away.

 

[Fredrik Lundh]

You blogged on Django. Let's use that. Don't you think model creation
in Django can be represented better, given that it is done often
enough?

nope, because 1) it's not done very often, and 2) the existing syntax is
already very minimal, and defined in terms of a language that I
already understand.

there might be cognitive theories that argue that the length of the
symbols used to describe something is more important than the symbols
you use, and how they can be "chunked" by the brain, but sturgeon's law
applies to cognitive scientists too ;-)

Since you are on thread and are a prominent and involved member of the
Python community, I would like it if you (or any such other) can
provide feedback on the rest of my previous post rather than be
dismissive by just a small portion of it. Perhaps, that will give me
some insight how these language design decisions are rationally made (I
am not strictly a programmer by profession, much less a language
designer).

see Ian's posts for some excellent discussion.

</F>

 

[Anton Vredegoor]

Anton Vredegoor wrote:

http://pysch.sourceforge.net/ast.html

Very interesting, it led me to some sxml describing pages and it also
tricked me into reading some Python documentation that I had always
considered to be hiding some arcane deep Python magic. I guess now that
Python is officially entering tree territory (as opposed to providing
third party functionality) it seems unavoidable that Python's officially
endorsed tree datatype will also be used for some of its internal
structures, thereby making it more accessible to programmers like me and
to outside programmers.

I was going to write a long reply to one of your previous messages, but
the above link references a project which may intersect with some of
your expectations. Meanwhile, it should be noted that the availability

Somehow I get the impression of getting away with my posts luckily,
while you now are duping other interested readers into not reading your
innermost feelings about this subject. Let's get it in the open, don't
spare me

of Python AST processing tools is not a recent thing: the compiler
module has been around for a long time, and it is possible to modify
the AST and to generate bytecode from it; my own experiments have
centred on producing other representations from the AST, and other more
successful projects (eg. ShedSkin) produce other languages (eg. C++)
from the AST.

Well maybe this trick of vehemently denying the existence of something
on Usenet worked again, by materializing the thing as a reaction.

However, I knew of the existence of such languages but I am mostly
interested in standardized code interchange, like for example with JSONP
which fetches some external javascriptcode from another server using
JSON and places the translated javascript into a webpage at the request
of the clients browser or so it seems. Maybe a Python webserver could
also emit pieces of javascript code by getting them from a *Python* code
library after translating Python code on the fly?

That would open up the web to Python programmers without browsers
needing to understand Python. Like Jython, but now as separately
distributed functions from different servers.

Anton

 

[Kay Schluehr]

Very interesting, it led me to some sxml describing pages and it also
tricked me into reading some Python documentation that I had always
considered to be hiding some arcane deep Python magic. I guess now that
Python is officially entering tree territory (as opposed to providing
third party functionality) it seems unavoidable that Python's officially
endorsed tree datatype will also be used for some of its internal
structures, thereby making it more accessible to programmers like me and
to outside programmers.

I had the same initial impression with the "deep" aspects of Python
parser API but I learned soon that I just stumbled about an awkward
kind parse-tree representation. The structures are as simple as they
could be - due to the nice LL(1) Python grammar which are reflected.

I think node manipulation is just a first basic step for defining
source transformers i.e. they provide a sound basis. For usability
purposes a template language is a far better solution.

Here is my own attempt. Defining a repeat statement ( for demonstration
purposes ) using EasyExtend requires following translation:

repeat:
<suite>
until:
<test>

==>

while 1:
<suite>
if <test>:
break

The placeholders <suite> and <test> correspond to nodes in the
parse-tree ( and constants in symbol.py ). The repeat-stmt is defined
by an added grammar rule. The corresponding node in the parse-tree is
dispatched against Transformer.handle_repeat_stmt() during parse-tree
traversal. Matching against <suite> and <test> in the repeat_stmt node
is done by applying:

suite_node = find_node(repeat_node, symbol.suite)
test_node = find_node(repeat_node, symbol.test, level=1)

What becomes tedious is the creation of the corresponding while_stmt
node which is the expected result of the transformation. I actually
want to use the declaration as it is written above and insert the
suite_node and the test_node where the placeholders <suite> and <test>
are defined. Using EasyExtend one can define an extension language
where the placeholders ( and some enhanced versions of those ) become
language elements! Given this one can put everything together:

import macro # module that corresonds to the macro extension
language

class FastTransformer(Transformer):
def handle_repeat_stmt(self, repeat_node):
suite_node = find_node(repeat_node, symbol.suite)
test_node = find_node(repeat_node, symbol.test, level=1)
target_stmt = """
while 1:
<suite>
if <test>:
break
"""
while_node = macro.expand(target_stmt,
{'suite': suite_node,
'test': test_node},
goal =
symbol.while_stmt)
return any_stmt(while_node)

This is pretty safe and works recursively: if the <suite> node contains
another <repeat_stmt> node it will be replaced as well.

 

[Max Erickson]

However, I knew of the existence of such languages but I am
mostly interested in standardized code interchange, like for
example with JSONP which fetches some external javascriptcode
from another server using JSON and places the translated
javascript into a webpage at the request of the clients browser
or so it seems. Maybe a Python webserver could also emit pieces
of javascript code by getting them from a *Python* code library
after translating Python code on the fly?

Anton

I have no idea how close it is to what you are talking about, but pypy
does have some sort of python->javascript support:

http://codespeak.net/pypy/dist/pypy/doc/getting-
started.html#translating-the-flow-graph-to-javascript-code

About two thirds down, if the above link is broken:

http://codespeak.net/pypy/dist/pypy/doc/getting-started.html


max

 

[Bruno Desthuilliers]

Anton Vredegoor wrote:
(snip)

However, I knew of the existence of such languages but I am mostly
interested in standardized code interchange, like for example with JSONP
which fetches some external javascriptcode from another server using
JSON and places the translated javascript into a webpage at the request
of the clients browser or so it seems.

This is AJAX with JSON instead of XML (should we call this AJAJ ?-).
It's quite handy, since it saves both the extra bits to be transfered
(XML is way much verbose than JSON) and the XML to javascript parsing.

Maybe a Python webserver could
also emit pieces of javascript code by getting them from a *Python* code
library after translating Python code on the fly?

If you restrict this on data, it's already done (Turbogears uses this
for AJAX).

 

[Paul Boddie]

Somehow I get the impression of getting away with my posts luckily,
while you now are duping other interested readers into not reading your
innermost feelings about this subject. Let's get it in the open, don't
spare me

I was just going to say that your Web application could have used XSLT
more or less exclusively if the data arrives in XML (and obviously
leaves in some kind of XML), provided you aren't doing exotic
processing on the data.

[...]

Well maybe this trick of vehemently denying the existence of something
on Usenet worked again, by materializing the thing as a reaction.

My intention was to correct the perception that before a bunch of work
was done on exposing the Python AST (the so-called "AST branch") there
weren't any tools to get nice ASTs which could be used to generate
code. Of course there are such tools, with the compiler module being in
the standard library itself - not to be confused with the parser module
which everyone seems to mention and about whose representation everyone
seems to complain, but that's also obviously in the standard library
too.

I'm not sure about the ready availability of Python-to-Python
transformers, though. I have made HTML "reports" of Python programs -
annotated source code with type information in CSS-based pop-ups - and
a plain text serialiser for the compiler module's AST wouldn't be as
difficult as that. The hardest part of working with the compiler API
would be in the modification of the AST, and perhaps it is in that area
that the "AST branch" shows promise.

[...]

Maybe a Python webserver could also emit pieces of javascript code by
getting them from a *Python* code library after translating Python
code on the fly?

http://subway.python-hosting.com/file/crackajax/trunk/crackajax.py

I personally don't go for this kind of thing, but people have given it
some thought.

Paul

 

[Ravi Teja]

I missed this reply earlier.

there might be cognitive theories that argue that the length of the
symbols used to describe something is more important than the symbols
you use and how they can be "chunked" by the brain

Expert communication is known to work differently. For example, take a
doctor's note. It's minimalist representation and domain specific
notational standardizations are absolute gibberish for a reader without
the knowledge of medicine (or more specifically, the sub-specialty) and
the context of the patient. But it is very efficient for the audience
it is intended for. Now, the hand writing - that's a wholly different
story .

BTW, I am studying clinical information representations. This
discussion was interesting to me because I am trying to see programming
language representation as a better discussed and more structured
microcosm of the relatively fuzzy domain I am studying. Of course,
there are many important differences and I may be drawing the parallels
liberally at this point but it gives me an interesting perspective.