Step-by-step exec

[gregory.lielens]

Hi,

I am using a small python file as an input file (defining constants,
parameters, input data, ...) for a python application.
The input file is simply read by an exec statement in a specific
dictionary, and then the application retrieve all the data it need
from the dictionary...
Everything is working nicely, but I'd like to have something a little
bit more robust regarding input file errors: now
any error in the python input script raise an exception and stop the
execution.
What I am trying to do is to execute it "step-by-step", so that I can
capture the exception if one line (or multi-line statement) fails,
print a warning about the failure, and continue the execution fo the
following lines/statements. Of course, an error on one line can
trigger errors in the following lines, but it does not matter in the
application I have in mind, the goal is to parse as much of the input
script as possible, warn about the errors, and check what's inside the
dictionary after the exec.
One way to do it is to read the input script line per line, and exec
each line in turn. However, this is not convenient as it does not
allow multi-line statements, or basic control flow like if - else
statements or loops.

Is there a better way for a step-by-step exec? Syntax errors in the
input script are not really a problem (as it is generated elsewhere,
it is not directly edited by users), although it would be nice to
catch. The biggest problem are runtime errors (attribute error, value
error, ...). Maybe compiling the file into a code object, and
executing this code object step-by-step in a way similar to debug? pdb
module should do something similar....

Best regards,

Greg.

 

[saju.pillai]

Hi,

I am using a small python file as an input file (defining constants,
parameters, input data, ...) for a python application.
The input file is simply read by an exec statement in a specific
dictionary, and then the application retrieve all the data it need
from the dictionary...
Everything is working nicely, but I'd like to have something a little
bit more robust regarding input file errors: now
any error in the python input script raise an exception and stop the
execution.
What I am trying to do is to execute it "step-by-step", so that I can
capture the exception if one line (or multi-line statement) fails,
print a warning about the failure, and continue the execution fo the
following lines/statements. Of course, an error on one line can
trigger errors in the following lines, but it does not matter in the
application I have in mind, the goal is to parse as much of the input
script as possible, warn about the errors, and check what's inside the
dictionary after the exec.
One way to do it is to read the input script line per line, and exec
each line in turn. However, this is not convenient as it does not
allow multi-line statements, or basic control flow like if - else
statements or loops.

Do you have control over the input file generation ? If the input file
can be easily divided into self sufficient blocks of code, you could
read each block in one at a time and do a compile() and exec(). Your
input file need not be a full python script too, you could just have
token delimited blocks of python code which are read in 1 block at a
time and then exec().

-srp

 

[gregory.lielens]

Do you have control over the input file generation ? If the input file
can be easily divided into self sufficient blocks of code, you could
read each block in one at a time and do a compile() and exec(). Your
input file need not be a full python script too, you could just have
token delimited blocks of python code which are read in 1 block at a
time and then exec().

-srp

Thanks for your input!
I had a similar solution in mind (with continuation comments to be
able to read multi-line statements, instead of statement delimiters,
but
I think your delimiter idea would be easier to implement).
The only problem is that files generated by current and older version
of the input generator (which do not have any kind of statement
delimiter)
will not be readable, or, if we just consider old file as one big
statement, will not offer the step-by-step execution. It would work
nice with new files,
but the best solution would be something that work also with older
auto-generated files...

In fact, the error in the input script are mainly caused by version
mismatch, and the step-by-step approach is
mainly for improving backward compatibility.
Having better backward compatibility starting from now on would be
nice, but having backward compatibility with previous versions would
be even better ;-)
contains multi-line statements

 

[Aaron Brady]

Thanks for your input!
I had a similar solution in mind (with continuation comments to be
able to read multi-line statements, instead of statement delimiters,
but
I think your delimiter idea would be easier to implement).
The only problem is that files generated by current and older version
of the input generator (which do not have any kind of statement
delimiter)
will not be readable, or, if we just consider old file as one big
statement, will not offer the step-by-step execution. It would work
nice with new files,
but the best solution would be something that work also with older
auto-generated files...

In fact, the error in the input script are mainly caused by version
mismatch, and the step-by-step approach is
mainly for improving backward compatibility.
Having better backward compatibility starting from now on would be
nice, but having backward compatibility with previous versions would
be even better ;-)
contains multi-line statements

Check out the InteractiveConsole and InteractiveInterpreter classes.
Derive a subclass and override the 'push' method. It's not documented
so you'll have to examine the source to find out exactly when and what
to override.

 

[Steven D'Aprano]

Hi,

I am using a small python file as an input file (defining constants,
parameters, input data, ...) for a python application. The input file is
simply read by an exec statement in a specific dictionary, and then the
application retrieve all the data it need from the dictionary...

Surely a better, more Pythonic, and *faster* way to accomplish the same
thing is with import?

Everything is working nicely, but I'd like to have something a little
bit more robust regarding input file errors: now any error in the python
input script raise an exception and stop the execution.

Which is the right thing to do.

What I am trying to do is to execute it "step-by-step", so that I can
capture the exception if one line (or multi-line statement) fails, print
a warning about the failure, and continue the execution fo the following
lines/statements. Of course, an error on one line can trigger errors in
the following lines, but it does not matter in the application I have in
mind,

I'm curious what sort of application you have where it doesn't matter
that programmatic statements are invalid.

the goal is to parse as much of the input script as possible, warn
about the errors, and check what's inside the dictionary after the exec.
One way to do it is to read the input script line per line, and exec
each line in turn. However, this is not convenient as it does not allow
multi-line statements, or basic control flow like if - else statements
or loops.

So basically you want to create a Python interpreter you can stop and
start which runs inside Python. Is that right?

 

[gregory.lielens]

Check out the InteractiveConsole and InteractiveInterpreter classes.
Derive a subclass and override the 'push' method.  It's not documented
so you'll have to examine the source to find out exactly when and what
to override.

Thanks, this is the thing I was looking for: it works very nicely

 

[gregory.lielens]

I'm curious what sort of application you have where it doesn't matter
that programmatic statements are invalid.

Well, it is not that it does not matter, it is that I'd like to get as
much as possible of the input file "executed".
The application is roughly to save a graph, and retrieve it later.
The graph is made of a set of python classes, derived from a GraphNode
instance but each having particular sets of new attributes.
The storage format should be human readable and editable (this is why
a simple pickle or equivalent is not OK), store only part of the
database, and automatically reflect the changes of the classes (new
attributes in old classes, new classes, ...). The format I used was
"simply" instructions that re-create the graph...

It works, but could be robust accross changes (basically, an old
database read with new version of the application should
recreate as much nodes as possible, with new attributes intialized at
default value). The node classes allows that, but during the parsing
when attributes names or classes names have changed, some errors can
happen, which will not only affect those nodes but stop execution. In
many cases, even with those error, executing the rest of the commands
will re-create an usefull graph, of course lacking some informations
but much more usefull that the partial graph obtained when stopping at
the first error...

So basically you want to create a Python interpreter you can stop and
start which runs inside Python. Is that right?

Yep, and the InteractiveConsole works very nicely for doing that

 

[M.-A. Lemburg]

Well, it is not that it does not matter, it is that I'd like to get as
much as possible of the input file "executed".
The application is roughly to save a graph, and retrieve it later.
The graph is made of a set of python classes, derived from a GraphNode
instance but each having particular sets of new attributes.
The storage format should be human readable and editable (this is why
a simple pickle or equivalent is not OK), store only part of the
database, and automatically reflect the changes of the classes (new
attributes in old classes, new classes, ...). The format I used was
"simply" instructions that re-create the graph...

It works, but could be robust accross changes (basically, an old
database read with new version of the application should
recreate as much nodes as possible, with new attributes intialized at
default value). The node classes allows that, but during the parsing
when attributes names or classes names have changed, some errors can
happen, which will not only affect those nodes but stop execution. In
many cases, even with those error, executing the rest of the commands
will re-create an usefull graph, of course lacking some informations
but much more usefull that the partial graph obtained when stopping at
the first error...


Yep, and the InteractiveConsole works very nicely for doing that

Instead of creating a Python interpreter in Python (using e.g.
PyPy ... http://www.pypy.org/), you might want to look at
using a trace function using sys.settrace():

http://www.python.org/doc/2.5.2/lib/module-sys.html

These functions are called for every line the Python interpreter
executes and have full access to the stack, variables, etc.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 07 2008)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611