Custom behavior defined in the imported module

[Paul McGuire]

I have a new enhancement to pyparsing that doubles the parse speed (using a
technique called "packrat parsing"), but which is not suitable for all
parsers, specifically those that have complex parse actions. I don't want
to just enable this feature by default - I think there is too much risk of
it breaking existing code. Actually, I think the chance is <10%, but I
think I'm better off leaving the feature dormant unless explicitly enabled,
so that it doesn't take someone by surprise when they install the new
version.

The alternatives I've come up with for the user to enable this packrat parse
mode are:

1. Add a staticmethod enablePackrat() to the pyparsing ParserElement class,
to modify the ParserElement defintion of the internal (non-packrat) parse()
method. This method essentially runs code looking like:

ParserElement.parse,ParserElement.originalParse = \
ParserElement.packratParse,ParserElement.parse

where the packratParse method is a memoizing wrapper around calls to
originalParse. This technique requires the caller to invoke
ParserElement.enablePackrat() after having imported pyparsing.

This renaming trick actually breaks when using psyco, if psyco has compiled
the pyparsing methods before calling enablePackrat(). You have to import
pyparsing, call enablePackrat(), then call psyco.full(), or whatever psyco
commands to cache compiled versions of the pyparsing function code.

2. Implement some sort of "onImport" hook to allow the calling routine to
use a specialized import statement, something like "from pyparsing import
packratParsing" that would do the same as the above enablePackrat() call,
but perhaps in a more Pythonic-looking manner. (I was inspired along these
lines from Jim Hugunin's IronPython talk, in which he used import hooks to
modify the behavior of the base Python string class.) I've looked a bit at
imp, imputil, and ihooks, and these all seem to be modifiers to the import
mechanism as called from the importing module. What import hooks are
accessible from within the module being imported? Is there any way to see
which form of import was used, and to access the arguments of the import
command as invoked from the calling module?

I was hoping that by hooking into the import command, I could make this
mode-setting command more atomic with the actual import of pyparsing's class
and function definitions, to avoid problems like the psyco issue.


Plus I'm open to other suggestions as well.

Thanks,
-- Paul

 

[Kent Johnson]

The alternatives I've come up with for the user to enable this packrat parse
mode are:

1. Add a staticmethod enablePackrat() to the pyparsing ParserElement class,
to modify the ParserElement defintion of the internal (non-packrat) parse()
method. This method essentially runs code looking like:

ParserElement.parse,ParserElement.originalParse = \
ParserElement.packratParse,ParserElement.parse

Could you just define a module pyparsingpackrat like this:
from pyparsing import *
ParserElement.parse,ParserElement.originalParse = \
ParserElement.packratParse,ParserElement.parse

Then users would just import pyparsingpackrat instead of pyparsing.

Kent

 

[Paul McGuire]

Could you just define a module pyparsingpackrat like this:
from pyparsing import *
ParserElement.parse,ParserElement.originalParse = \
ParserElement.packratParse,ParserElement.parse

Then users would just import pyparsingpackrat instead of pyparsing.

Kent

I *really* like keeping pyparsing's footprint down

 

[Paul McGuire]

Could you just define a module pyparsingpackrat like this:
from pyparsing import *
ParserElement.parse,ParserElement.originalParse = \
ParserElement.packratParse,ParserElement.parse

Then users would just import pyparsingpackrat instead of pyparsing.

Kent

(damned touchy touchpad!)

.... anyway...

I *really* like keeping pyparsing's footprint down to just one Python
module. It would be nice if I could pass an argument along with the import
statement:

import pyparsing(packrat=True)

or some such.

-- Paul

 

[Paul McGuire]

Could you just define a module pyparsingpackrat like this:
from pyparsing import *
ParserElement.parse,ParserElement.originalParse = \
ParserElement.packratParse,ParserElement.parse

Then users would just import pyparsingpackrat instead of pyparsing.

Kent

I'm also not overkeen on elevating the word "packrat" into the module name
itself!

-- Paul