Code generator and visitor pattern

K

Karsten Wutzke

Hello,

this is obviously a Python OO question:

Since Python isn't stringly typed, single-dispatch isn't available per
se. So is the "double-dispatch" Visitor pattern, which is usually used
in OO systems to implement code generators. So, what is the de facto
method in Python to handle source code generation?

Karsten
 
T

Thomas Jollans

Hello,

this is obviously a Python OO question:

Since Python isn't stringly typed,

I expect this is an innocent typo, and you mean strictly.
single-dispatch isn't available per se. So is the "double-dispatch" Visitor pattern,

Wait, what?
First of all, python is strictly typed in that every object has exactly
one type, which is different from other types. So you can't do "1"+2, as
you can in some other languages.

Anyway, this is interesting: Tell me more about how Python's dynamic
nature makes it impossible to do whatever you're trying to do. I'm
baffled. What are you trying to do, anyway?
which is usually used
in OO systems to implement code generators. So, what is the de facto
method in Python to handle source code generation?

WHOA! Now if that isn't a Gedankensprung. Also, I'm still very far from
your train of thought, apparently: Now, the thing that code generators
probably share is that they write code to files. It depends on what I'm
trying to do of course, but I expect there's a good chance that if I
wrote a code generator in Python, it wouldn't be particularly
object-oriented at all.
 
K

Karsten Wutzke

I expect this is an innocent typo, and you mean strictly.

Yes, typo, I meant strictly.
Wait, what?
First of all, python is strictly typed in that every object has exactly
one type, which is different from other types. So you can't do "1"+2, as
you can in some other languages.

Anyway, this is interesting: Tell me more about how Python's dynamic
nature makes it impossible to do whatever you're trying to do. I'm
baffled. What are you trying to do, anyway?


WHOA! Now if that isn't a Gedankensprung. Also, I'm still very far from
your train of thought, apparently: Now, the thing that code generators
probably share is that they write code to files. It depends on what I'm
trying to do of course, but I expect there's a good chance that if I
wrote a code generator in Python, it wouldn't be particularly
object-oriented at all.

Well, I'm most experienced in OO, so writing OO in Python seems like
the way to start with Python. The visitor pattern uses single-
dispatch, that is, it determines which method to call be the type of
object passed in. I did some reading and it turned out that Python
can't do it without some tricks (function decorators and 3rd party
code). For what I'm doing, I can't, or rather don't want to rely on
3rd party code (that has reasons). Thus, the visitor OO pattern must
be replaced by some other way.

As I expected, you already hinted a non-OO solution. Which is now that
*I* am wondering what that would look like...

Note, that I have an hierarchical object structure which I want to
iterate over, so using OO looked natural to me. If there's a better
approach, I'm all ears.

Karsten
 
T

Thomas Jollans

Yes, typo, I meant strictly.


Well, I'm most experienced in OO, so writing OO in Python seems like
the way to start with Python. The visitor pattern uses single-
dispatch, that is, it determines which method to call be the type of
object passed in. I did some reading and it turned out that Python
can't do it without some tricks (function decorators and 3rd party
code).

Well, yes: the name of a function or method refers to a single callable
object and piece of code. if you care about the type of the argument,
you must say so explicitly:

class A:
def dothing(self, obj):
if isinstance(obj, str):
self.dostringthing(obj)
elif isinstance(obj, (int,float)):
self.donumberthing(obj)
else:
self.dogenericthing(obj)

# ...

while python doesn't have C++-style function overloading, its
alternative also doesn't have the limitations that come with it.
 
S

Stefan Behnel

Karsten Wutzke, 15.07.2010 20:45:
Well, I'm most experienced in OO, so writing OO in Python seems like
the way to start with Python. The visitor pattern uses single-
dispatch, that is, it determines which method to call be the type of
object passed in.

Well, then do that. Put the types into a dict and map them to the functions
to call, then call the function through the dict. That's a pretty common
way to do dispatching.

Note, that I have an hierarchical object structure which I want to
iterate over, so using OO looked natural to me. If there's a better
approach, I'm all ears.

You speak in riddles, but my guess is that your problem is that you don't
want to dispatch mechanism to match only exact types but also subtypes. No
problem, just build your dict incrementally and add new types as they come
in. See this file for an example:

http://hg.cython.org/cython-devel/file/tip/Cython/Compiler/Visitor.py

Stefan
 
M

MRAB

Stefan said:
Karsten Wutzke, 15.07.2010 20:45:

Well, then do that. Put the types into a dict and map them to the
functions to call, then call the function through the dict. That's a
pretty common way to do dispatching.



You speak in riddles, but my guess is that your problem is that you
don't want to dispatch mechanism to match only exact types but also
subtypes. No problem, just build your dict incrementally and add new
types as they come in. See this file for an example:

http://hg.cython.org/cython-devel/file/tip/Cython/Compiler/Visitor.py
Another variation: the dispatch table starts with entries for certain
types then it adds subtypes on demand:

def visit(self, obj):
try:
handler = self.dispatch_table[type(obj)]
except KeyError:
for disp_type, disp_func in self.dispatch_table.items():
if isinstance(obj, disp_type):
self.dispatch_table[type(obj)] = disp_func
handler = disp_func
else:
raise RuntimeError("Visitor does not accept object: %s"
% obj)
return handler(obj)
 
M

Matt McCredie

Karsten Wutzke said:
So, what is the de facto method in Python to handle source code generation?


Take a look at the NodeVisitor class in the ast module in python 2.6+.
The visitor pattern is implemented in the python standard library.

Matt
 
S

Stefan Behnel

MRAB, 15.07.2010 21:33:
Stefan said:
Karsten Wutzke, 15.07.2010 20:45:

Well, then do that. Put the types into a dict and map them to the
functions to call, then call the function through the dict. That's a
pretty common way to do dispatching.



You speak in riddles, but my guess is that your problem is that you
don't want to dispatch mechanism to match only exact types but also
subtypes. No problem, just build your dict incrementally and add new
types as they come in. See this file for an example:

http://hg.cython.org/cython-devel/file/tip/Cython/Compiler/Visitor.py
Another variation: the dispatch table starts with entries for certain
types then it adds subtypes on demand:

def visit(self, obj):
try:
handler = self.dispatch_table[type(obj)]
except KeyError:
for disp_type, disp_func in self.dispatch_table.items():
if isinstance(obj, disp_type):
self.dispatch_table[type(obj)] = disp_func
handler = disp_func
else:
raise RuntimeError("Visitor does not accept object: %s" % obj)
return handler(obj)

Well, yes, that's basically what the code behind the above link does,
except that it follows the type hierarchy correctly to find the closest match.

Stefan
 
I

Ian Hobson

Hello,

this is obviously a Python OO question:

Since Python isn't stringly typed, single-dispatch isn't available per
se. So is the "double-dispatch" Visitor pattern, which is usually used
in OO systems to implement code generators. So, what is the de facto
method in Python to handle source code generation?

Karsten
I'm baffled. Not by what you mean by stringly, but....

What feature of Python stops you writing the three parts of the visitor
pattern:

IIRC you need:

A tree walker that creates the visitor and walks the tree calling
node.visitFrom(visitor)
on each one in the required order.

The visitfrom(aVisitor) routines in each node type that calls
aVisitor.visitedMyNodeType(self)
where MyNodeType is, naturally different for each node type!

All the
def visitedNodeType(aNode): routines in visitor to generate the code.

Simples! No? :)

Ian
 
I

Ian Hobson

Hello,

this is obviously a Python OO question:

Since Python isn't stringly typed, single-dispatch isn't available per
se. So is the "double-dispatch" Visitor pattern, which is usually used
in OO systems to implement code generators. So, what is the de facto
method in Python to handle source code generation?

Karsten
I'm baffled. Not by what you mean by stringly, but....

What feature of Python stops you writing the three parts of the visitor
pattern:

IIRC you need:

A tree walker that creates the visitor and walks the tree calling
node.visitFrom(visitor)
on each one in the required order.

The visitfrom(aVisitor) routines in each node type that calls
aVisitor.visitedMyNodeType(self)
where MyNodeType is, naturally different for each node type!

All the
def visitedNodeType(aNode): routines in visitor to generate the code.

Simples! No? :)

Ian
 
I

Ian Hobson

Hello,

this is obviously a Python OO question:

Since Python isn't stringly typed, single-dispatch isn't available per
se. So is the "double-dispatch" Visitor pattern, which is usually used
in OO systems to implement code generators. So, what is the de facto
method in Python to handle source code generation?

Karsten
I'm baffled. Not by what you mean by stringly, but....

What feature of Python stops you writing the three parts of the visitor
pattern:

IIRC you need:

A tree walker that creates the visitor and walks the tree calling
node.visitFrom(visitor)
on each one in the required order.

The visitfrom(aVisitor) routines in each node type that calls
aVisitor.visitedMyNodeType(self)
where MyNodeType is, naturally different for each node type!

All the
def visitedNodeType(aNode): routines in visitor to generate the code.

Simples! No? :)

Ian
 
C

Carl Banks

Yes, typo, I meant strictly.








Well, I'm most experienced in OO, so writing OO in Python seems like
the way to start with Python. The visitor pattern uses single-
dispatch, that is, it determines which method to call be the type of
object passed in. I did some reading and it turned out that Python
can't do it without some tricks (function decorators and 3rd party
code). For what I'm doing, I can't, or rather don't want to rely on
3rd party code (that has reasons). Thus, the visitor OO pattern must
be replaced by some other way.

Oh brother.

Around these parts, we consider the main use of most Design Patterns
to be to work around limitations of other languages. Visitor Pattern
is probably the worst example of it.

In Python it's completely unnecessary (at least in its boilerplate-
heavy incarnation as used in C++), and the fact that Python isn't
strongly typed, as you put it, is exactly the reason why.

Say you have a bunch of unrelated types that define a calculate
method, you have a variable x that could be any of these types.
Here's how you would do that in Python:

x.calculate()

Bam, that's it. Visitor Pattern in Python. You don't have to create
a bunch of homemade dispatching boilerplate like you do in C++.


Carl Banks
 
S

Stefan Behnel

Carl Banks, 16.07.2010 01:14:
Around these parts, we consider the main use of most Design Patterns
to be to work around limitations of other languages. Visitor Pattern
is probably the worst example of it.

In Python it's completely unnecessary (at least in its boilerplate-
heavy incarnation as used in C++), and the fact that Python isn't
strongly typed, as you put it, is exactly the reason why.

Say you have a bunch of unrelated types that define a calculate
method, you have a variable x that could be any of these types.
Here's how you would do that in Python:

x.calculate()

Bam, that's it. Visitor Pattern in Python. You don't have to create
a bunch of homemade dispatching boilerplate like you do in C++.

Well, you can do that in every OO language. It's not what the visitor
pattern is there for, though.

The code I referenced is from the Cython compiler, and we use it to "do
stuff" with the AST. The visitor pattern is actually a pretty common way to
bind code in a single place that does a certain thing to different parts of
a data structure. Without it, if you kept that code *inside* of the data
structure, you'd have to spill the type specific parts all over your code.

Stefan
 
C

Carl Banks

Carl Banks, 16.07.2010 01:14:










Well, you can do that in every OO language. It's not what the visitor
pattern is there for, though.

The code I referenced is from the Cython compiler, and we use it to "do
stuff" with the AST. The visitor pattern is actually a pretty common way to
bind code in a single place that does a certain thing to different parts of
a data structure. Without it, if you kept that code *inside* of the data
structure, you'd have to spill the type specific parts all over your code..

Ahh, so this aspect oriented programming is it.

I see your point, Visitor Pattern isn't necessary to work around the
"can't easily polymorph unrelated types" limitation, but could still
be necessary to work around "can't easily dispatch on methods not part
of the class" limitation. So, ok, Visitor Pattern maybe isn't the
worst one. My bad


Carl Banks
 
M

Michele Simionato

Hello,

this is obviously a Python OO question:

Since Python isn't stringly typed, single-dispatch isn't available per
se. So is the "double-dispatch" Visitor pattern, which is usually used
in OO systems to implement code generators. So, what is the de facto
method in Python to handle source code generation?

Karsten

You ask about code generation and you already had answers in that
area, but let me talk a bit about a simpler topic, traversing a
hierarchical file system. I think this is relevant (even if not
answering your question) if you want to get familiar with the Python
way. In the old days, the way to traverse a file system was through
the os.path.walk function. Here is what the docs say (from
http://docs.python.org/library/os.path.html):

"""
os.path.walk(path, visit, arg)

Calls the function visit with arguments (arg, dirname, names) for each
directory in the directory tree rooted at path (including path itself,
if it is a directory). The argument dirname specifies the visited
directory, the argument names lists the files in the directory (gotten
from os.listdir(dirname)). The visit function may modify names to
influence the set of directories visited below dirname, e.g. to avoid
visiting certain parts of the tree. (The object referred to by names
must be modified in place, using del or slice assignment.)
"""

As you see the documentation make explicit reference to the visitor
pattern.
However a note below says:

"""
This function is deprecated and has been removed in 3.0 in favor of
os.walk().
"""

In other word, the visitor pattern is *not* the Pythonic way to solve
this
problem. The Pythonic way is to use os.walk, which converts the nested
structure in a flat structure. From the docs
(http://docs.python.org/library/os.html):

"""
This example displays the number of bytes taken by non-directory files
in each directory under the starting directory, except that it doesn’t
look under any CVS subdirectory:

import os
from os.path import join, getsize
for root, dirs, files in os.walk('python/Lib/email'):
print root, "consumes",
print sum(getsize(join(root, name)) for name in files),
print "bytes in", len(files), "non-directory files"
if 'CVS' in dirs:
dirs.remove('CVS') # don't visit CVS directories
"""

There is a big conceptual difference between os.path.walk and os.walk.
The first works like a framework: you pass a function to it and
os.path.walk
is in charging of calling it when needed. The second works like a
library:
os.walk flattens the hierarchical structure and then you are in charge
of
doing everything you wish with it.

os.walk is the Pythonic way, and you suggested to follow that
approach;
for instance elementTree and lxml (libraries for parsing XML data)
work
exactly that way. Actually one of the motivating examples for the
introduction of generators in Python was their use in flattening
data structure, i.e. exactly the pattern used by os.walk.

The message is stop thinking like in Java and start using idiomatic
Python. We are here to help.

Michele Simionato
 
S

Stefan Behnel

Carl Banks, 16.07.2010 07:50:
Ahh, so this aspect oriented programming is it.

Never thought about it that way, but you have a point there. It's pretty
much the same idea as AOP, but without any of those huge and feature
drooling code injection frameworks.

Stefan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,777
Messages
2,569,604
Members
45,234
Latest member
SkyeWeems

Latest Threads

Top