Tracing the execution of scripts?

Discussion in 'Python' started by Michael B. Trausch, Oct 27, 2006.

  1. Alright, I seem to be at a loss for what I am looking for, and I am not
    even really all that sure if it is possible or not. I found the 'pdb'
    debugger, but I was wondering if there was something that would trace or
    log the order of line execution for a multi-module Python program. I am
    having a little bit of a problem tracking down a problem that I
    mentioned earlier
    (http://groups.google.com/group/comp.lang.python/msg/9c759fc888b365be),
    and I am hoping that maybe seeing how the statements are executed will
    shed some light on the entire thing for me.

    The debugger isn't working, though, because I have a timer set up to
    fire off every 20ms to check for incoming network traffic from the
    server, and that timer firing off makes stepping through with the
    debugger seemingly impossible to get any useful data.

    Basically, is there something that will log every line of Python code
    executed, in its order of execution, to a text file so that I can see
    what is (or isn't) happening that I am expecting? I know that the
    server is sending and receiving properly, because I can connect to it
    with a regular telnet client and it works -- but at the same time,
    everything I can think of to check my program's functionality checks out
    just fine (e.g., it reports that it is successfully sending what I am
    typing to the server, and it says that the server is not sending
    anything back to be read from the socket).

    If it makes any difference, I am using wxWidgets' Python bindings for
    the UI code (and the timer), and using Python's socket library for the
    network stuff.

    -- Mike
    Michael B. Trausch, Oct 27, 2006
    #1
    1. Advertising

  2. Michael B. Trausch

    Ben Finney Guest

    Jean-Paul Calderone <> writes:

    > 1) Write unit tests for your code. Keep writing unit tests until
    > you have some that _don't pass_. Then fix your code so that they
    > do. When you do further development, write the tests first, then
    > implement the code that makes them pass.


    Hear hear.

    Be advised, though, that attempting to apply unit tests to code that
    wasn't designed with testing in mind may very quickly reveal a poor
    design. [0] If you can't easily test pieces of the code independently,
    you probably haven't written those pieces to be loosely coupled and
    well-defined.

    The moral? Writing unit tests *along with* the functional code will
    result in a design that is loosely coupled, and probably
    better-defined. Also, hopefully, easy to test :)


    [0] I have no idea whether this is the case for the OP. It's a very
    common symptom that arises from people who are first advised to
    introduce tests to their code, though.

    --
    \ "If you ever catch on fire, try to avoid seeing yourself in the |
    `\ mirror, because I bet that's what REALLY throws you into a |
    _o__) panic." -- Jack Handey |
    Ben Finney
    Ben Finney, Oct 27, 2006
    #2
    1. Advertising

  3. "Michael B. Trausch" <"mike$#at^&nospam!%trauschus"> wrote:

    > Basically, is there something that will log every line of Python code
    > executed, in its order of execution, to a text file so that I can see
    > what is (or isn't) happening that I am expecting?


    Python itself can do this for you. A __VERY__ simple approach:

    |def myCallTrace (frame, event, arg):
    | tracefile.write('localTrace\n frame: '+str(frame)+'\n event:
    |'+str(event)+'\n arg: '+str(arg)+'\n')
    | tracefile.write(' '+str(inspect.getframeinfo(frame))+'\n')
    |
    |def myTrace (frame, event, arg):
    | tracefile.write('globalTrace\n frame: '+str(frame)+'\n event:
    |'+str(event)+'\n arg: '+str(arg)+'\n')
    | if event == 'call':
    | return myCallTrace
    |
    |import inspect
    |import sys
    |
    |sys.settrace(myTrace)
    |tracefile = file('trace.txt', 'w')

    Insert this in you program and you get a trace of every line of Python-Code
    executed in the file trace.txt. You must read the documentation of the
    module inspect and of sys.settrace() to understand, what happens and what
    it means. Additionally, if it should work with threads, you must take care
    that every thread gets its own output file. But as a first step to get a
    trace of execution, this should do it.

    HTH, Regards
    Stephan
    Stephan Kuhagen, Oct 27, 2006
    #3
  4. Michael B. Trausch

    R. Bernstein Guest

    pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines
    as they are executed as well as an --output option to have this sent
    to a file rather than stdout. If your program has threads it would be
    good to use the --threading option. (The best threading support is if
    your program uses the threading module for threads rather than the
    lower-level thread module.)

    A demo of the debugger including showing line tracing and output to a file is
    here:
    http://showmedo.com/videos/video?name=pythonBernsteinPydbIntro&fromSeriesID=28


    "Michael B. Trausch" <"mike$#at^&nospam!%trauschus"> writes:

    > Alright, I seem to be at a loss for what I am looking for, and I am not
    > even really all that sure if it is possible or not. I found the 'pdb'
    > debugger, but I was wondering if there was something that would trace or
    > log the order of line execution for a multi-module Python program. I am
    > having a little bit of a problem tracking down a problem that I
    > mentioned earlier
    > (http://groups.google.com/group/comp.lang.python/msg/9c759fc888b365be),
    > and I am hoping that maybe seeing how the statements are executed will
    > shed some light on the entire thing for me.
    >
    > The debugger isn't working, though, because I have a timer set up to
    > fire off every 20ms to check for incoming network traffic from the
    > server, and that timer firing off makes stepping through with the
    > debugger seemingly impossible to get any useful data.
    >
    > Basically, is there something that will log every line of Python code
    > executed, in its order of execution, to a text file so that I can see
    > what is (or isn't) happening that I am expecting? I know that the
    > server is sending and receiving properly, because I can connect to it
    > with a regular telnet client and it works -- but at the same time,
    > everything I can think of to check my program's functionality checks out
    > just fine (e.g., it reports that it is successfully sending what I am
    > typing to the server, and it says that the server is not sending
    > anything back to be read from the socket).
    >
    > If it makes any difference, I am using wxWidgets' Python bindings for
    > the UI code (and the timer), and using Python's socket library for the
    > network stuff.
    >
    > -- Mike
    R. Bernstein, Oct 27, 2006
    #4
  5. Ben Finney wrote:
    > Jean-Paul Calderone <> writes:
    >
    >> 1) Write unit tests for your code. Keep writing unit tests until
    >> you have some that _don't pass_. Then fix your code so that they
    >> do. When you do further development, write the tests first, then
    >> implement the code that makes them pass.

    >
    > Hear hear.
    >
    > Be advised, though, that attempting to apply unit tests to code that
    > wasn't designed with testing in mind may very quickly reveal a poor
    > design. [0] If you can't easily test pieces of the code independently,
    > you probably haven't written those pieces to be loosely coupled and
    > well-defined.
    >


    I will whole-heartedly admit that my code is probably poorly designed.
    *shrug* Though, I *do* try to write everything in such a way as to be
    able to easily re-use it later. When I wrote little things just to help
    me sysadmin, I learned that was really key to making my life easier.

    Besides, I am lazy... reuse certainly serves my needs there! :)

    >
    > The moral? Writing unit tests *along with* the functional code will
    > result in a design that is loosely coupled, and probably
    > better-defined. Also, hopefully, easy to test :)
    >


    I think I have more to learn on the concept of unit-testing. I have
    never seen a group push the idea so hard. I have read about unit
    testing before, and even written tests (in other languages) to test
    modules of library code that I put together once, but that was about it.
    I need to, I think, get more "into" the concept, though. It isn't
    something that I am able to just whip out and go "Hey, a widget unit
    test! And it works!" probably because of my own lack of practice.

    >
    > [0] I have no idea whether this is the case for the OP. It's a very
    > common symptom that arises from people who are first advised to
    > introduce tests to their code, though.
    >


    It is very likely that it is the case. I am, by all means, what I could
    consider to be a novice programmer when it comes to anything outside of
    my little world of PHP. I can write web applications in PHP until the
    cows come home with little to no problem, and I can play with SQL the
    same way. But, I seem to have problems when I step out of that little
    box, and so, that tells me that I need to work harder at it. :)

    -- Mike
    Michael B. Trausch, Oct 27, 2006
    #5
  6. Stephan Kuhagen wrote:
    > "Michael B. Trausch" <"mike$#at^&nospam!%trauschus"> wrote:
    >
    >> Basically, is there something that will log every line of Python code
    >> executed, in its order of execution, to a text file so that I can see
    >> what is (or isn't) happening that I am expecting?

    >
    > Python itself can do this for you. A __VERY__ simple approach:
    >

    [snip]
    >
    > Insert this in you program and you get a trace of every line of Python-Code
    > executed in the file trace.txt. You must read the documentation of the
    > module inspect and of sys.settrace() to understand, what happens and what
    > it means. Additionally, if it should work with threads, you must take care
    > that every thread gets its own output file. But as a first step to get a
    > trace of execution, this should do it.
    >


    I will need to keep this around to look at a little later. It looks
    like it would be something particularly useful when line-tracing by
    itself fails. I am constantly finding myself amazed at Python's
    capabilities.

    -- Mike
    Michael B. Trausch, Oct 27, 2006
    #6
  7. Michael B. Trausch

    Fulvio Guest

    ***********************
    Your mail has been scanned by InterScan MSS.
    ***********************


    On Friday 27 October 2006 17:31, R. Bernstein wrote:
    > pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines


    I faced several time that pydb stuck without sign of errors. In the other hand
    Pdb doesn't appear that problem.
    Mostly pydb freeze on long loops.
    It might be some problem on my setup, I'll check it up...

    F
    Fulvio, Oct 27, 2006
    #7
  8. Michael B. Trausch

    Larry Bates Guest

    I just thought I would put my 2 cents in on this issue. Others
    have suggested that unit tests are an excellent way of debugging
    your code and I agree. I also find that writing code from the
    outset using a logging class (there is one in the standard
    library) that allows you to create log files of information as
    you run your application is a good idea. Using increasingly
    detailed logs (I use a debug mode of 1,2,3,4 for more detail)
    that dump where you are and intermediate variable values works
    EXTREMELY well for me. I leave this code in the application
    so that I can have customers (or myself) run the application
    in debug mode should I have a hard to find problem. This is
    especially true for long running or lights-out batch apps
    that have no UI making debugging even more difficult. I find
    that the overhead of testing if I'm in debug mode and logging
    results is almost non-existent to the overall execution speed
    of my scripts, but then I don't have very many really speed
    sensitive scripts so your mileage might vary.

    Hope the information helps.

    -Larry
    Larry Bates, Oct 27, 2006
    #8
  9. Michael B. Trausch

    fumanchu Guest

    Stephan Kuhagen wrote:
    > "Michael B. Trausch" <"mike$#at^&nospam!%trauschus"> wrote:
    >
    > > Basically, is there something that will log every line of Python code
    > > executed, in its order of execution, to a text file so that I can see
    > > what is (or isn't) happening that I am expecting?

    >
    > Python itself can do this for you. A __VERY__ simple approach:
    > ...
    > Additionally, if it should work with threads, you must take care
    > that every thread gets its own output file.


    Or be differentiated *somehow*. My pyconquer module tabs each thread's
    activity into swimlanes, so you can even see when each thread starts,
    obtains/releases the GIL, and stops. See the longer example here:
    http://projects.amor.org/misc/wiki/PyConquer


    Robert Brewer
    System Architect
    Amor Ministries
    fumanchu, Oct 27, 2006
    #9
  10. Michael B. Trausch

    R. Bernstein Guest

    Fulvio <> writes:
    > ***********************
    > Your mail has been scanned by InterScan MSS.
    > ***********************

    Delighted to know that.

    >
    > On Friday 27 October 2006 17:31, R. Bernstein wrote:
    > > pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines

    >
    > I faced several time that pydb stuck without sign of errors. In the other hand
    > Pdb doesn't appear that problem.
    > Mostly pydb freeze on long loops.


    In version 1.19 (released today), I extended signal handling so that
    you can send the program a signal and it will print a stack trace of
    where it is and continue. But even before this release, you could
    arrange to enter the debugger by sending it a signal. It's possible
    something like this might be used help track down the problem in
    either pydb or another Python program that seems to be not
    responding. On the other hand, use at your own risk - I don't
    guarentee this will work for you.

    And.. before you ask for more details, I'll repeat what someone else
    posted in response to another of your questions:

    I guess you've been told to read this here, but just in case it
    wasn't, or you didn't bother to read it:

    http://catb.org/esr/faqs/smart-questions.html

    > It might be some problem on my setup, I'll check it up...


    Given your other posts, that's quite possible. If it's not, submit a
    bug report. (Posting to c.l.r isn't the same as submitting a bug
    report). Thanks.
    R. Bernstein, Oct 27, 2006
    #10
  11. Michael B. Trausch

    Ben Finney Guest

    Test-driven development (was: Tracing the execution of scripts?)

    "Michael B. Trausch" <mike$#at^&nospam!%> writes:

    > Ben Finney wrote:
    > > Jean-Paul Calderone <> writes:
    > >
    > >> 1) Write unit tests for your code. Keep writing unit tests until
    > >> you have some that _don't pass_. Then fix your code so that they
    > >> do. When you do further development, write the tests first, then
    > >> implement the code that makes them pass.

    > >
    > > Hear hear.
    > >
    > > Be advised, though, that attempting to apply unit tests to code
    > > that wasn't designed with testing in mind may very quickly reveal
    > > a poor design. [0] If you can't easily test pieces of the code
    > > independently, you probably haven't written those pieces to be
    > > loosely coupled and well-defined.
    > >

    >
    > I will whole-heartedly admit that my code is probably poorly
    > designed. *shrug* Though, I *do* try to write everything in such a
    > way as to be able to easily re-use it later.


    The trick is to write pieces that are small and loosely-coupled, so
    that small pieces of code (not just entire modules) can be re-used
    without being re-written.

    Test-driven development helps with this, because a small,
    loosely-coupled piece of code is easy to test, so you will gravitate
    toward that style.

    > Besides, I am lazy... reuse certainly serves my needs there! :)


    The programming virtues espoused by Larry Wall continue to hold true :)

    > I think I have more to learn on the concept of unit-testing.


    > I have never seen a group push the idea so hard.


    Writing each test before the functional code is not a new idea, but
    Python makes it trivially easy to do, so the excuses for not doing it
    are much weaker and easier to ridicule :)

    > I have read about unit testing before, and even written tests (in
    > other languages) to test modules of library code that I put together
    > once, but that was about it.


    Here's an article that goes through the concepts, and happens to use
    Python.

    <URL:http://www.onlamp.com/pub/a/python/2004/12/02/tdd_pyunit.html>

    > I need to, I think, get more "into" the concept, though. It isn't
    > something that I am able to just whip out and go "Hey, a widget unit
    > test! And it works!" probably because of my own lack of practice.


    One thing to note is that UI widgets shouldn't be the main thing
    you're testing. You need to write code that can be easily tested
    independent of everything else, because that way you can know that it
    works and move on to other things quickly, building on it as a
    foundation.

    UI widgets should be as dumb as possible, because they don't make a
    whole lot of sense by themselves except in the dumbest sense. Every
    part of your program that actually does something interesting with
    data or resources should be utterly independent of any UI
    widgets. *That* is the code that you need to get right, so that should
    be riddled with tests.

    Your widgets should be plugging into code that dumbly passes the data
    to the well-tested, UI-independent code that does work with that data;
    and the widgest should be getting their data from such UI-independent
    outputs. This is one aspect of "loose coupling": the code talks along
    a very narrow, well-defined, simple interface to other parts of the
    code.

    UI code especially is prone to change entirely independent of the
    desired functionality. for this reason, it's vital that you *can*
    change the functionality and UI code each without needing to change
    the other. The same principle applies along any interface between
    parts of code that can be expected to change independent of each
    other: loosely couple them together, with narrow well-defined
    interfaces.

    You may have heard of the "Model, View, Controller" pattern. This is a
    design pattern that emphasises a loose coupling between the UI (the
    "controller") and the data model, by abstracting each to the other via
    a simple "view" interface module. That way, each of the Controller and
    the Model can be oblivious to the most common changes in the other,
    even when those changes make for complex code, because the View they
    interact with stays simple and doesn't need to change.

    <URL:http://en.wikipedia.org/wiki/Model-view-controller>

    The reason test-driven development is so useful here is that, in order
    to write a test for something *before* it exists, you must think about
    how that piece of code will interact with the rest of the code. You
    must, in other words, design its interface before writing its
    code. This is excellent, because it means you're focussing on exactly
    the thing that will help to keep your overall design loosely coupled.

    Then, after you have a test that interacts with this interface, all
    you need to do is write the simplest code to satisfy that
    interface. The temptation to write clever, complex code is reduced,
    because you're now interested in satisfying the external test, not in
    treating this piece of code as an awesome complex machine ignoring the
    rest of the code.

    Good luck with getting test infected :)

    <URL:http://c2.com/cgi/wiki?TestInfected>

    --
    \ "I bought a self learning record to learn Spanish. I turned it |
    `\ on and went to sleep; the record got stuck. The next day I |
    _o__) could only stutter in Spanish." -- Steven Wright |
    Ben Finney
    Ben Finney, Oct 28, 2006
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Graham Cottle

    Tracing Execution in Class Library

    Graham Cottle, Jan 21, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    369
    Daniel Bass
    Jan 21, 2004
  2. Jan Danielsson
    Replies:
    8
    Views:
    613
    Mike Meyer
    Jul 22, 2005
  3. Jp Calderone
    Replies:
    0
    Views:
    445
    Jp Calderone
    Jul 21, 2005
  4. Nikhil

    Dynamic tracing of C code execution

    Nikhil, May 12, 2006, in forum: C Programming
    Replies:
    5
    Views:
    1,600
    Frank Pittel
    May 16, 2006
  5. Sabiyur

    Code execution Tracing

    Sabiyur, Aug 16, 2006, in forum: C++
    Replies:
    1
    Views:
    330
    Jim Langston
    Aug 17, 2006
Loading...

Share This Page