ANN: a mini-language for encapsulating deep-copy operations on Pythondata structures

[Steve Howell]

During the last few days I have written code in support of a small DDL
language that encapsulates a concise representation of the
manipulations needed to make a deep subcopy of a Python-like data
structure. It is inspired by syntax from mainstream modern languages,
including, of course, Python. The DDL can be converted to an AST. That
AST can be walked to either generate Python code that expresses the
mapping or to generate Python objects that can execute the mapping.
Either of the prior outputs can then subsequently be applied to real
world inputs to create new Python data structures from old ones, using
the mechanisms specified in the original DDL for attribute access,
dictionary lookup, iteration, method invocation, etc.

Here is an example of the DDL (and I hate the terminology "DDL," just
cannot think of anything better):


{
'show_table_of_contents',
'author' {
.person 'name',
.person 'location' as city,
.favorite_books()[
.title,
.cost() as expense
] as books}
}


There are more details here:

http://showellonprogramming.blogspot.com/2009/11/more-on-python-deep-copy-schema.html

Apart from shamelessly plugging my blog, I am hoping to generate
ideas, constructive criticism, etc.

Feel free to comment here or on the blog. So far there are three
entries on the blog, all pertaining to Python.

The implementation of the idea is quite new, but it is a topic that I
have been pondering for a long time--no matter how expressive your
programming language of choice might be, there are certain programming
tasks that just seem needlessly tedious. The mini-language above
proposes to solve one problem but solve it well.

I am particularly interested to find out whether somebody has tried
something like this before.

 

[Steve Howell]

[...]
Here is an example of the DDL (and I hate the terminology "DDL," just
cannot think of anything better):

            {
                'show_table_of_contents',
                'author' {
                    .person 'name',
                    .person 'location' as city,
                    .favorite_books()[
                        .title,
                        .cost() as expense
                        ] as books}
            }

There are more details here:

http://showellonprogramming.blogspot.com/2009/11/more-on-python-deep-...

Thanks for your feedback, Marc-Andre! Comments inline...

Personally, I find the explicit approach more intuitive, since
you immediately see what you are going to get:

show_table_of_contents = context['show_table_of_contents']
author = context['author']
d = {
    'show_table_of_contents': show_table_of_contents,
    'author': {
        'name': author.person['name'],
        'city': author.person['location'],
        'books': [{
            'title': item.title,
            'expense': item.cost(),
            }
            for item in author.favorite_books()],
        },
    }

I understand what you mean about the explicit approach. When you go
for brevity, you sacrifice some explicitness. One of the things I'm
actually looking to do next is to be able to generate code like what
you've written above. You could make the generated code even more
procedural in terms of how it walks the structure--for example,
replace the list comprehension with a for loop. This could be useful
for inserting tracing code, for example.

I particularly find this part non-intuitive:

                    .favorite_books()[
                        .title,
                        .cost() as expense
                        ] as books}

Yep, I think what I really want is this:

.favorite_books()[{
.title,
.cost() as expense
}] as books}

I am not sure the curlies make the transformation completely
intuitive, but they are more suggestive that you are generating a list
of dictionaries.

and would leave in the square brackets for __getitem__
lookups on these:

Yep, I agree. The square brackets were not a deliberate omission.
They were just slightly more tricky to implement because of '[' having
double syntactical meaning.

For additional inspiration, you might want to look at XSLT
which provides similar transformations on XML data structures.

There are also a number of other transformation languages:

http://en.wikipedia.org/wiki/Model_...p://en.wikipedia.org/wiki/Data_transformation

Thanks for the links!

Regarding the term "DDL": that's normally used for "data definition
language" and doesn't really have all that much to do with
transforming data. You normally define data structures using
DDL - without actually putting data into those structures.

Why not "PyDTL".... Python data transformation language ?!

Yep, I like that suggestion. The language does indeed describe a
transformation. In some ways I was wanting to think of it more
broadly, in terms that the transformation also acts as a schema for
any input objects. For example, you could use '{.title, .publisher,
{.name} }' as a schema to validate that an object behaves like a
book. I suppose that's still a transformation in an abstract sense,
where the output is just True, False, or maybe even the slice of the
transformation that can/cannot be executed.