Getting better traceback info on exec and execfile - introspection?

Discussion in 'Python' started by R. Bernstein, Jan 14, 2006.

  1. R. Bernstein

    R. Bernstein Guest

    In doing the extension to the python debugger which I have here:
    http://sourceforge.net/project/showfiles.php?group_id=61395&package_id=175827
    I came across one little thing that it would be nice to get done better.

    I notice on stack traces and tracebacks, an exec or execfile command
    appears as a stack entry -- probably as it should since it creates a
    new environment.

    However the frame information for exec or execfile looks like this:
    File "<string>", line 1, in ?

    Okay, the "line 1" probably changes, but most of the time it's going
    to be one. And in code that's in CVS (cvsview:
    http://cvs.sourceforge.net/viewcvs.py/bashdb/pydb/) you'll see that
    instead of this I now put some thing like:

    ## 1 in exec '<string>' at line 1

    which is perhaps is a little more honest since one is not really in a
    file called <string>. However the way the debugger gets this *is*
    still a little hoaky in that it looks for something in the frame's
    f_code.co_filename *called* <string>. And from that it *assumes* this
    is an exec, so it can't really tell the difference between an exec
    command an execfile command, or a file called <string>. But I suppose
    a little more hoakiness *could* be added to look at the outer frame
    and parse the source line for "exec" or "execfile".

    And suppose instead of '<string>' I'd like to give the value or the
    leading prefix of the value instead of the unhelpful word '<string>'?
    How would one do that? Again, one way is to go into the outer frame
    get the source line (if that exists), parse that and interpolate
    argument to exec(file). Is there are better way?

    Another and perhaps more direct way to distinguish exec/execfile might
    be to some how look at the byte code of the frame entry and decode
    that. It also could be used for example in another place in the
    debugger where one wants to skip over executing "def" statements (not
    the function invocation, but the statement which adds the
    function/method so it can get called). But depending on how things are
    done unless there are nice version-independent and
    implementation-independent symbolic definitions for exec_opcode,
    execfile_opcode and def_opcode, although straight-forward and reliable
    for a given version/implementation it might not be a good thing to do.

    Any help, thoughts or pointers?
    R. Bernstein, Jan 14, 2006
    #1
    1. Advertising

  2. R. Bernstein

    R. Bernstein Guest

    I suggested:
    > And suppose instead of '<string>' I'd like to give the value or the
    > leading prefix of the value instead of the unhelpful word '<string>'?
    > How would one do that? Again, one way is to go into the outer frame
    > get the source line (if that exists), parse that and interpolate
    > argument to exec(file). Is there are better way?


    I've hacked this approach into the CVS for pydb, but it is not all
    that accurate (although in practice I'm pleased with the result).

    My current regular expression is (^|\s+)exec\s+(.*) which does match
    simple things, but it would fail on say by *not* matching:
    x=1;exec "y=2";exec "z=3"
    and fail by *wrongly* matching on:
    x=" exec meeting in 5 mins"

    Even if I knew how to call the python parser on the line, that still
    might not be good enough in the presence of continuation lines.

    This probably should be addressed at a lower level in gathering
    traceback data inside Python.

    > Another and perhaps more direct way to distinguish exec/execfile might

    ^^^^^^^^
    execfile is not a problem, just exec.
    R. Bernstein, Jan 15, 2006
    #2
    1. Advertising

  3. Re: Getting better traceback info on exec and execfile -introspection?

    R. Bernstein wrote:

    > In doing the extension to the python debugger which I have here:
    >

    http://sourceforge.net/project/showfiles.php?group_id=61395&package_id=175827
    > I came across one little thing that it would be nice to get done better.
    >
    > I notice on stack traces and tracebacks, an exec or execfile command
    > appears as a stack entry -- probably as it should since it creates a
    > new environment.
    >
    > However the frame information for exec or execfile looks like this:
    > File "<string>", line 1, in ?


    That comes from how the code object was compiled:

    >>> code = compile('error','Anything you want goes here','single')
    >>> exec code

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "Anything you want goes here", line 1, in ?
    NameError: name 'error' is not defined


    For example, in ipython:

    In [1]: error
    ---------------------------------------------------------------------------
    exceptions.NameError Traceback (most recent
    call last)

    /home/fperez/<ipython console>

    NameError: name 'error' is not defined


    So any regexp-matching based approach here is likely to be fairly brittle,
    unless you restrict your tool to the standard python interpreter, and you
    get some guarantee that it will always tag interactive code with
    '<string>'.

    Regards,

    f
    Fernando Perez, Jan 15, 2006
    #3
  4. R. Bernstein

    R. Bernstein Guest

    Fernando Perez <> writes:
    > R. Bernstein wrote:

    ....
    > > However the frame information for exec or execfile looks like this:
    > > File "<string>", line 1, in ?

    >
    > That comes from how the code object was compiled:

    ....
    > So any regexp-matching based approach here is likely to be fairly brittle,
    > unless you restrict your tool to the standard python interpreter, and you
    > get some guarantee that it will always tag interactive code with
    > '<string>'.


    Thanks for the information! Alas, that's bad news. The question then
    remains as to how to accurately determine if a frame is running an
    exec operation and accurately get the string associated with the exec.

    Given all the introspection about python's introspection, it should be
    possible.

    I also tried pydb.py using IPython 0.7.0 (using an automatically
    updated Fedora Core 5 RPM) and although I was expecting pydb.py breakage
    from what you report, I didn't find any difference.


    % ipython
    Python 2.4.2 (#1, Dec 17 2005, 13:02:22)
    Type "copyright", "credits" or "license" for more information.

    IPython 0.7.0 -- An enhanced Interactive Python.
    ? -> Introduction to IPython's features.
    %magic -> Information about IPython's 'magic' % functions.
    help -> Python's own help system.
    object? -> Details about 'object'. ?object also works, ?? prints more.

    In [1]: %run /usr/lib/python2.4/site-packages/pydb.py test1.py
    > /home/rocky/python/test1.py(2)?()

    -> import sys
    (Pydb) where
    -> 0 in file '/home/rocky/python/test1.py' at line 2
    ## 1 in exec cmd in globals, locals at line 1
    ## 2 run() called from file '/usr/lib/python2.4/bdb.py' at line 366
    (Pydb)

    Note entry ##1 is *not*
    File "<string>", line 1, in ?

    So somehow I guess ipython is not intercepting or changing how compile
    was run here.
    R. Bernstein, Jan 15, 2006
    #4
  5. Re: Getting better traceback info on exec and execfile -introspection?

    R. Bernstein wrote:

    > Fernando Perez <> writes:
    >> R. Bernstein wrote:

    > ...
    >> > However the frame information for exec or execfile looks like this:
    >> > File "<string>", line 1, in ?

    >>
    >> That comes from how the code object was compiled:

    > ...
    >> So any regexp-matching based approach here is likely to be fairly
    >> brittle, unless you restrict your tool to the standard python
    >> interpreter, and you get some guarantee that it will always tag
    >> interactive code with '<string>'.

    >
    > Thanks for the information! Alas, that's bad news. The question then
    > remains as to how to accurately determine if a frame is running an
    > exec operation and accurately get the string associated with the exec.
    >
    > Given all the introspection about python's introspection, it should be
    > possible.


    I thought a little about this. One possibility would be to check whether
    the argument given in the string exists as a file; if not, assume it's a
    string. I can't think of another way, since the string given is completely
    arbitrary. But someone who knows more about the internal structure of code
    objects may give you better tips.

    > I also tried pydb.py using IPython 0.7.0 (using an automatically
    > updated Fedora Core 5 RPM) and although I was expecting pydb.py breakage
    > from what you report, I didn't find any difference.
    >
    >
    > % ipython
    > Python 2.4.2 (#1, Dec 17 2005, 13:02:22)
    > Type "copyright", "credits" or "license" for more information.
    >
    > IPython 0.7.0 -- An enhanced Interactive Python.
    > ? -> Introduction to IPython's features.
    > %magic -> Information about IPython's 'magic' % functions.
    > help -> Python's own help system.
    > object? -> Details about 'object'. ?object also works, ?? prints more.
    >
    > In [1]: %run /usr/lib/python2.4/site-packages/pydb.py test1.py
    >> /home/rocky/python/test1.py(2)?()

    > -> import sys
    > (Pydb) where
    > -> 0 in file '/home/rocky/python/test1.py' at line 2
    > ## 1 in exec cmd in globals, locals at line 1
    > ## 2 run() called from file '/usr/lib/python2.4/bdb.py' at line 366
    > (Pydb)
    >
    > Note entry ##1 is *not*
    > File "<string>", line 1, in ?
    >
    > So somehow I guess ipython is not intercepting or changing how compile
    > was run here.


    Oh, that's because you're using %run, so your code is in complete control.
    What I meant about a restriction was thiking about using your pydb as the
    debugger called by ipython when exceptions occur.

    Try

    %pdb

    and then %run any script with an error in it, to be dropped in post-mortem
    mode inside the stack. That uses an ipython-enhanced pdb, but not your
    pydb. As your code matures and improves, it would be nice to, for example,
    make it possible to plug it into ipython as a 'better debugger'. But
    there, you'll see '<ipython console>' as the filename markers.

    Anyway, good luck with this. If you are interested in ipython integration,
    I suggest the ipython-dev list as a better place for discussion: I only
    monitor c.l.py on occasion, so I may well miss things here.

    Regards,

    f
    Fernando Perez, Jan 15, 2006
    #5
  6. R. Bernstein wrote:
    ..
    ..
    ..
    > which is perhaps is a little more honest since one is not really in a
    > file called <string>. However the way the debugger gets this *is*
    > still a little hoaky in that it looks for something in the frame's
    > f_code.co_filename *called* <string>. And from that it *assumes* this
    > is an exec, so it can't really tell the difference between an exec
    > command an execfile command, or a file called <string>. But I suppose
    > a little more hoakiness *could* be added to look at the outer frame
    > and parse the source line for "exec" or "execfile".


    You should check the getFrameInfo function in zope.interface package:
    http://svn.zope.org/Zope3/trunk/src/zope/interface/advice.py?rev=25177&view=markup

    > And suppose instead of '<string>' I'd like to give the value or the
    > leading prefix of the value instead of the unhelpful word '<string>'?
    > How would one do that? Again, one way is to go into the outer frame
    > get the source line (if that exists), parse that and interpolate
    > argument to exec(file). Is there are better way?


    Py library (http://codespeak.net/py/current/doc/index.html) has some
    similar functionality in the code subpackage.
    Ziga Seilnacht, Jan 15, 2006
    #6
  7. R. Bernstein

    R. Bernstein Guest

    "Ziga Seilnacht" <> writes:
    > You should check the getFrameInfo function in zope.interface package:
    > http://svn.zope.org/Zope3/trunk/src/zope/interface/advice.py?rev=25177&view=markup


    Thanks! Just looked at that. The logic in the relevant part (if I've extracted
    this correctly):

    f_globals = frame.f_globals
    hasName = '__name__' in f_globals
    module = hasName and sys.modules.get(f_globals['__name__']) or None
    namespaceIsModule = module and module.__dict__ is f_globals
    if not namespaceIsModule:
    # some kind of funky exec
    kind = "exec"

    is definitely not obvious to me. exec's don't have module namespace
    (or something or other)? Okay. And that the way to determine the
    module-namespace thingy is whatever that logic is? Are the assumptions
    here be likely to be valid in the future?

    Another problem I have with that code is that it uses the Zope Public
    License. But the code is adapted from the Python Enterprise
    Application Kit (PEAK) which doesn't seem to use the Zope Public
    License. I'm not sure it's right to adopt a Zope Public License just
    for this.

    So instead, I followed the other avenue I suggested which is
    dissembling the statement around the frame. Talk about the kettle
    calling the pot black! Yes, I don't know if the assumptions in this
    method are likely to be valid in the future either.

    But I can use op-code examination to also help me determine if we are
    stopped at a def statement which I want to skip over. So here's the
    code I've currently settled on:

    from opcode import opname
    def op_at_frame(frame):
    code = frame.f_code.co_code
    pos = frame.f_lasti
    op = ord(code[pos])
    return opname[op]

    def is_exec_stmt(frame):
    """Return True if we are looking at an exec statement"""
    return frame.f_back is not None and op_at_frame(frame.f_back)=='EXEC_STMT'

    re_def = re.compile(r'^\s*def\s+')
    def is_def_stmt(line, frame):
    """Return True if we are looking at a def statement"""
    # Should really also check that the operand is a code object
    return re_def.match(line) and op_at_frame(frame)=='LOAD_CONST'

    But it may just be because I wrote it that I find it easier to
    understand and more straightforward to fathom.

    > > And suppose instead of '<string>' I'd like to give the value or the
    > > leading prefix of the value instead of the unhelpful word '<string>'?
    > > How would one do that? Again, one way is to go into the outer frame
    > > get the source line (if that exists), parse that and interpolate
    > > argument to exec(file). Is there are better way?

    >
    > Py library (http://codespeak.net/py/current/doc/index.html) has some
    > similar functionality in the code subpackage.


    Again, many thanks. I'll have to study this further. It may be that
    exec and so on are wrapped so that it's possible to squirrel away the
    string before calling exec. Again, dunno. But thanks for the pointer.
    R. Bernstein, Jan 16, 2006
    #7
  8. R. Bernstein

    R. Bernstein Guest

    Fernando Perez <> writes:
    > I thought a little about this. One possibility ...


    Thanks. A sibling thread has the code I'm currently using.

    > Oh, that's because you're using %run, so your code is in complete control.
    > What I meant about a restriction ...

    Okay.

    > If you are interested in ipython integration,


    Yes I am.

    > I suggest the ipython-dev list as a better place for discussion: I only
    > monitor c.l.py on occasion, so I may well miss things here.


    Okay now subscribed. But interestingly I looked in the IPython PDF and
    didn't see mention of it when looking for contact info. I do however
    see it (now) listed on http://ipython.scipy.org/. Thanks again.
    R. Bernstein, Jan 16, 2006
    #8
  9. R. Bernstein

    R. Bernstein Guest

    Fernando Perez <> writes:
    > So any regexp-matching based approach here is likely to be fairly brittle,
    > unless you restrict your tool to the standard python interpreter, and you
    > get some guarantee that it will always tag interactive code with
    > '<string>'.


    Meant to mention for what it's worth. Looks like I'm not the first to use the
    filename == '<string>' test. I note this in the stock pdb.py:

    # To be overridden in derived debuggers
    def defaultFile(self):
    """Produce a reasonable default."""
    filename = self.curframe.f_code.co_filename
    if filename == '<string>' and self.mainpyfile:
    filename = self.mainpyfile
    return filename

    I'm not sure under what conditions this equality test normally occurs
    though.
    R. Bernstein, Jan 16, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jonathan
    Replies:
    2
    Views:
    516
    Steven Taschuk
    Aug 10, 2003
  2. Jan Decaluwe
    Replies:
    0
    Views:
    294
    Jan Decaluwe
    Nov 7, 2003
  3. Chris Rebert
    Replies:
    2
    Views:
    328
    Peter Otten
    Apr 22, 2009
  4. Jack Bates
    Replies:
    0
    Views:
    257
    Jack Bates
    May 2, 2011
  5. Przemek Wrzos

    Ruby 1.9 Binding#exec and introspection

    Przemek Wrzos, Jul 29, 2008, in forum: Ruby
    Replies:
    2
    Views:
    116
    Przemek Wrzos
    Aug 3, 2008
Loading...

Share This Page