any macro-like construct/technique/trick?

[Mac]

Is there a way to mimic the behaviour of C/C++'s preprocessor for
macros? The problem: a lot of code like this:

def foo():
# .... do some stuff
if debug:
emit_dbg_obj(DbgObjFoo(a,b,c))

# .... do more stuff
if debug:
emit_dbg_obj(DbgObjBar(d,e))

# ... and so on ...

Notes:

* the two-lines of debug conditional tend to really break up the flow
of the surrounding code

* in C you could wrap them with a macro so you could do
DEBUG_EMIT(DbgObjFoo(a,b,c)), etc, with the macro only instantiating
the object and processing it if the debug flag was set. The one-liner
is MUCH less disruptive visually when reading code

* using
def debug_emit(obj):
if debug:
emit_dbg_obj(obj)
is a poor solution, because it *always* instantiates DbgObj*, even when
not needed; I want to avoid such unnecessary waste

 

[Jordan Rastrick]

Is having to read two lines really that much worse than one? And the
only thing you find objectionable about the most obvious solution?
If so, then what's wrong with:

def foo():
# .... do some stuff
if debug: emit_dbg_obj(DbgObjFoo(a,b,c))
# .... do more stuff

To my mind, this is no less readable and no more clutter than
DEBUG_EMIT(DbgObjFoo(a,b,c)) Then again, I don't mind the two line
version - personally, I prefer my debug code be more conspicuous rather
than less, so that its clearly seperate, visually and hence mentally,
from functional part of the code.

Preprocessor Macros and their ilk have always seemed like an awful
kludge to me, nessecary and useful in a language like C I'll grant, but
not very Pythonic at all.

If you really feel you can't stand the (presumably marginal) expense of
creating the debug objects when unnessecary, and want to get rid of
certain lines of code conditionally - using textual replacement, which
is essentially, by my limited understanding, all a pre-processor does -
then why not just do it yourself? Have
emit_dbg_obj(DbgObjFoobar(a,b,c)) all you want in your in development
code, and use a simple regex search and replace to comment it out of
your source when the code ships.

Most critically, though, if you're worried about efficiency issues this
fine-grained, trying to scrounge every last byte of memory, I'd suggest
Python is probably the wrong language for your problem. Object creation
overhead and the like is just part of the regular cost of having a high
level, dynamic, interpreted language. If you really, really need
preprocessor macros for efficiency reasons, then your probably need C.

I'm sorry, my reply hasn't been all that helpful I guess It seems a
bit of a trend on this mailing list - ask for some feature, or a way to
emulate it, and instead get a bunch of arguments as to why that feature
doesn't fit the design and philosophy of the language, and some
suggestions on how to do things differently.

Probably some much more knowledgable Pythonista can cook up a way to
achieve your desired behaviour, using exec or eval or some other such
unsightly hack... but I can only hope such practices don't become
widespread.

 

[Mac]

After I wrote my post I realized I didn't provide enough context of
what I'm doing, hence it is not quite clear to any readers why the heck
I would want this. The gist is this. I have a number of algorithms
that perform the same task. I'm trying to assess which ones work better
under which circumstances, which means I'm timing them (interested in
*relative* speed, so Python is acceptable). I also have a second app
which allows me to "observe" the operation of said algorithms by
consuming such emitted debug objects and graphically displaying the
information therein. I'm "observing" with a very fine granularity,
which means a debug object is emitted between almost every single line.
Naturally, if every second line is debug code (or worse, as above, 1
line of real code, 2 lines of debug code), the readability of the
algorithm is horrendous, making further development a pain. Since,
when I'm NOT observing the algorithms then I'm timing them, I want the
debug/observe code to have minimal impact on the runtimes;
instantiating some of these debug objects is rather costly.

There, hopefully this gives a better picture of my motivation. If
anyone can suggest alternate way to structure such code, I'm all ears
(... well, eyes, really...

Your proposal for one-lining the code as "if debug: emit_....."
occurred to me, but for readability reasons I want as visually *short*
a line as possible, so it doesn't detract from the other code. The C
preprocessor was particularly helpful in such "shortening". (e.g.,
print "foo"; DBG(DbgObj(..)); print "bar"

Hmm, with this setup you can't get any shorter than the instantiation
of the DebugObject... which gives me an idea... the "if debug:" could
be placed within DebugObjectBase, and all the DebugObjects inherit from
it... no... wait... no good, the problem is the following case:
...
# real line of code
DbgObjFoo(a,b,costly_function(c))
# real line of code

On a debugging/observing run, the debugging objects occasionally have
their constructor values computed externally, as above, and the
proposed solution would not save us from that costly computation.

 

[Andrew Dalke]

Is there a way to mimic the behaviour of C/C++'s preprocessor for
macros?

There are no standard or commonly accepted ways of doing that.

You could do as Jordan Rastrick suggested and write your own sort
of preprocessor, or use an existing one. With the new import
hooks you can probably make the conversion happen automatically,
though I hesitate suggestions that as you might actually do that.
It's typically a bad idea because you're in essence creating a
new language that is similar to but not Python, making it harder
for people to understand what's going on.

The problem: a lot of code like this:

def foo():
# .... do some stuff
if debug:
emit_dbg_obj(DbgObjFoo(a,b,c)) ...
* the two-lines of debug conditional tend to really break up the flow
of the surrounding code

If flow is your only concern you can have a do-nothing
function and at the top have

if debug:
emit = emit_dbg_obj
else:
def emit(*args, **kwargs): pass

then write all your code as

emit(DbgObjFoo(a,b,c))

* using
def debug_emit(obj):
if debug:
emit_dbg_obj(obj)
is a poor solution, because it *always* instantiates DbgObj*, even when
not needed; I want to avoid such unnecessary waste

That would work as well of course.

How bad is the waste? Is it really a problem?

Is all your code of the form

emit(Class(constructor, args))

? If so, your debug_emit could be made to look like

debug_emit(klass, *args, **kwargs):
if debug:
emit_dbg_obj(klass(*args, **kwargs))

and used

debug_emit(DbgObjFoo, a, b, c)
debug_emit(DbgObjBar, d, e)

though I would use the do-nothing function version I sketched
earlier because, *ahem*, it avoids an unnecessary waste of
the extra "if debug:" check.

Andrew
(e-mail address removed)

 

[Andrew Dalke]

After I wrote my post I realized I didn't provide enough context of
what I'm doing, [explanation followed]

I have a similar case in mind. Some graph algorithms work with a
handler, which is notified about various possible events: "entered
new node", "doing a backtrack", "about to leave a node". A general
algorithm may implement many of these. But if the handler doesn't
care about some of the events there's still the cost of either
doing a no-op call or checking if the callback function doesn't exist.

I've considered the idea of limited macro/template support for that
case, which either removes a callback or perhaps in-lines some
user-supplied code for the given circumstance.

it... no... wait... no good, the problem is the following case:
# real line of code
DbgObjFoo(a,b,costly_function(c))
# real line of code

In another branch I suggested

debug_emit(DbgObjFoo, a, b, costly_function(c))

which obviously wouldn't work for this case. The following lightly
tested code would work (assuming appropriate debugging)

debug_emit(DbgObjFoo, a, b,
Call(costly_function, c, Call(expensive_function, d)))

def debug_emit(klass, *args):
if debug:
emit_dbg_code(Call(klass, *args)())

class Call:
def __init__(self, f, *args):
self.f = f
self.args = args
def __call__(self):
args = []
for arg in self.args:
if isinstance(arg, Call):
args.append(arg())
else:
args.append(arg)
return self.f(*args)

There's still the overhead of making the Call objects, but it
shouldn't be that large. You can save a smidgeon by doing

if debug:
class Call:
... as defined earlier
else:
def Call(f, *args): pass



Andrew
(e-mail address removed)

 

[alex23]

* using
def debug_emit(obj):
if debug:
emit_dbg_obj(obj)
is a poor solution, because it *always* instantiates DbgObj*, even when
not needed; I want to avoid such unnecessary waste

Rather than passing in an instantiated object and only operating on it
if the debug flag is set, how about passing in the class &
initialisation parameters and only creating the object if necessary?
.... if debug: emit_dbg_obj(cls(*args))

Then your calls will just be a single line:

Does that help at all?

-alex23

 

[Neal Norwitz]

Andrew's approach is good, but you could so something a little
simpler/more flexible. Untested of course.

Every callable object is followed by the args to pass it. So this:
debug_emit(DbgObjFoo(a, b, costly_function(c)))

becomes:
debug_emit(DbgObjFoo, (a, b, costly_function, (c,)))

def debug_emit(*args):
if not debug:
return

# assume last arg is not a callable and skip it
i = len(args) - 2

while i > 0:
if callable(args):
# call it! assume the next arg is a tuple of params
new_value = args(*args[i+1])
args = args[:i] + (new_value,) + args[i+2:]

emit_dbg_code(*args)

cheers,
n

 

[Paddy]

If you still must have something like the c preprocessor then unix has
m4 (and it seems there is a windows version
http://gnuwin32.sourceforge.net/packages/m4.htm).

The start of the doc reads:


GNU M4
******

GNU `m4' is an implementation of the traditional UNIX macro processor.
It is mostly SVR4 compatible, although it has some extensions (for
example, handling more than 9 positional parameters to macros). `m4'
also has builtin functions for including files, running shell commands,
doing arithmetic, etc. Autoconf needs GNU `m4' for generating
`configure' scripts, but not for running them.


- Paddy.

 

[Georges JEGO]

Mac a écrit :

# .... do some stuff
if debug:
emit_dbg_obj(DbgObjFoo(a,b,c))

Assuming your debug functions always return true, you could use:

assert emit_dbg_obj(DbgObjFoo(a,b,c))

and have this code executed -or not- depending on the use of "-O"

-- Georges

 

[=?iso-8859-1?Q?Fran=E7ois?= Pinard]

[Georges JEGO]

Mac a écrit :

Assuming your debug functions always return true, you could use:

assert emit_dbg_obj(DbgObjFoo(a,b,c))

and have this code executed -or not- depending on the use of "-O"

Dirty, dirty trick .

`assert' is quite useful in its real and genuine purpose. There is
something heretic in allowing side effects in `assert' statements. A
few centuries ago, Python programmers doing this were burnt in public,
and this is how Python got forgotten for so long. (Guido rediscovered
it a dozen years ago, the wise guy attributed all the merit to himself!)

 

[=?iso-8859-1?Q?Fran=E7ois?= Pinard]

[Paddy]

If you still must have something like the c preprocessor
then unix has m4 (and it seems there is a windows version
http://gnuwin32.sourceforge.net/packages/m4.htm).

The difficulty of `m4' for Python source is that macro expansions should
be accompanied with proper adjustment of indentation, for adapting to
the context where macros are getting expanded.

René Seindal's `m4' is surprisingly powerful, so maybe that with enough
trickery (I really mean: a _lot_ of trickery), adjustment of indentation
could be possible. And maybe that a simpler `m4' approach to Python
macro-expansion could be made available through the later pluggin
features which René added to `m4'. (Yet I do not know if the pluggin
features have been integrated in the `m4' distribution mainstream.)

It would be much easier using `m4' for Python, if there was a way
to invoke it after Python lexer and before further parsing, because
<indent> and <dedent> tokens would already been identified. If this was
possible, `m4' would be a breeze to use as a pre-processor for Python.

Still in this dreaming mode, there would also be one necessary detail
missing for full comfort: that is, the recognition of `#line' like
directives as generated by `m4' so Python tracebacks, for example, would
correctly refer to the Python source lines before macro-expansion.

 

[Mike Meyer]

There are no standard or commonly accepted ways of doing that.

You could do as Jordan Rastrick suggested and write your own sort
of preprocessor, or use an existing one.

I've never tried it with python, but the C preprocessor is available
as 'cpp' on most Unix systesm. Using it on languages other than C has
been worthwhile on a few occasions. It would certainly seem to
directly meet the OP's needs.

<mike

 

[Andrew Dalke]

I've never tried it with python, but the C preprocessor is available
as 'cpp' on most Unix systesm. Using it on languages other than C has
been worthwhile on a few occasions. It would certainly seem to
directly meet the OP's needs.

Wouldn't that prohibit using #comments in the macro-Python code?
I suppose they could be made with strings, as in


"here is a comment"
do_something()

but it's ... strange.

Andrew
(e-mail address removed)

 

[Roy Smith]

Is there a way to mimic the behaviour of C/C++'s preprocessor for
macros?
[...]

* using
def debug_emit(obj):
if debug:
emit_dbg_obj(obj)
is a poor solution, because it *always* instantiates DbgObj*, even when
not needed; I want to avoid such unnecessary waste

How about using assert statements? When you run with optimization turned
on, they don't even get compiled. Try something like this:

class Debug:
def __init__ (self, message):
print "debug (%s)" % message

print "foo"
assert (Debug ("bar"))

which produces:

Roy-Smiths-Computerlay$ python x.py
foo
debug (bar)
Roy-Smiths-Computerlay$ python -O x.py
foo

Notice that the Debug() object doesn't even get created when optimization
is turned on. It's a little funky, but at least anybody who knows python
will understand your code. I agree with Andrew Dalke when he says of
preprocessors:

 

[Kay Schluehr]

Is there a way to mimic the behaviour of C/C++'s preprocessor for
macros? The problem: a lot of code like this:

def foo():
# .... do some stuff
if debug:
emit_dbg_obj(DbgObjFoo(a,b,c))

# .... do more stuff
if debug:
emit_dbg_obj(DbgObjBar(d,e))

# ... and so on ...

Notes:

* the two-lines of debug conditional tend to really break up the flow
of the surrounding code

* in C you could wrap them with a macro so you could do
DEBUG_EMIT(DbgObjFoo(a,b,c)), etc, with the macro only instantiating
the object and processing it if the debug flag was set. The one-liner
is MUCH less disruptive visually when reading code

Make emit_dbg_obj() a class to create one-liners:

class Emit_dbg_obj:
debug = True
def __init__(self, algo):
self.algo = algo

def call_if_debug(self):
if self.debug:
return self.algo

def foo2():
# .... do some stuff
Emit_dbg_obj(DbgObjFoo(a,b,c)).call_if_debug()

# .... do more stuff
Emit_dbg_obj(DbgObjFoo(c,d)).call_if_debug()


Ciao,
Kay

 

[Kay Schluehr]

Make emit_dbg_obj() a class to create one-liners:

class Emit_dbg_obj:
debug = True
def __init__(self, algo):
self.algo = algo

def call_if_debug(self):
if self.debug:
return self.algo

def foo2():
# .... do some stuff
Emit_dbg_obj(DbgObjFoo(a,b,c)).call_if_debug()

# .... do more stuff
Emit_dbg_obj(DbgObjFoo(c,d)).call_if_debug()


Ciao,
Kay

Hmmm... this won't work as expected.

class Emit_dbg_obj:
debug = True
def __init__(self, algo, args):
self.algo = algo
self.args = args


def call_if_debug(self):
if self.debug:
self.algo(*self.args)

def foo2():
# .... do some stuff
Emit_dbg_obj(DbgObjFoo,(a,b,c)).call_if_debug()

# .... do more stuff
Emit_dbg_obj(DbgObjFoo,(c,d)).call_if_debug()


Regards,
Kay