Strange Hotshot problem...

Discussion in 'Python' started by KefX, Oct 24, 2003.

  1. KefX

    KefX Guest

    Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed
    Python into a plugin for a freeware (closed-source) Windows music program
    called Jeskola Buzz. (Man, I can't believe how many people keep telling me
    "Extend Python instead!" when clearly I can't do that in this case. If somebody
    here tells me this, I will explode in an appropriately gory fashion.) It's a
    great program, and it works mainly on plugins other people write. Well, I had
    been working on a guitar synth plugin in the summer, which started to sound
    good. Then I lost the code (I was trying to rewrite it anyway but I wanted to
    keep it as reference.) So I began to think about ways to try to make it better.

    I eventually hit on the idea of using a scripting language to do this. I
    started designing a language with a simple grammar (I like doing this kind of
    thing in a semi-compulsive manner, heh)...but I tossed that aside, worked on a
    ROM hacking/translating project some...well, my ROM hacking project required me
    to write a small script, so I decided it was finally time to learn Python.

    Geez, I love this language. :)

    Anyway, now it's a couple months later, translation project gets put on hold
    for reasons outside of my control, so I'm trying the guitar synth again. And I
    thought...why don't I use Python? I was told I could never get it to run in
    real time (we're talking about DSP, after all). I was about to give up on
    pursuing the idea when I heard about Psyco, which should help speed up
    everything that isn't rewritten as a C extension. I decided I'll give that a
    shot, no matter how ridiculous the idea seems. My experience with ridiculous
    ideas is that sometimes they actually work. :)

    Well, to make a long story longer, I got it running and it does produce sound
    (not the sound I would like to hear from it, though!) But brokenness of the
    sound suggested that it's running too freakin' slow, and it was: Buzz's CPU
    display said my own plugin was hogging the whole CPU! Well, okay, it's not
    hopeless yet, I can profile it and see what I can do about the bottlenecks.

    Which finally leads me to my problem: I can't get Hotshot to work properly. I
    can't fix a bottleneck if I don't know where it is! ;) Remember, I'm calling
    various Python functions from C++. See, in my module's init code, I'm basically
    I'm creating the hotshot.Profile object with the required path, bound to
    _profiler, then I do _profiler.start() ...at the very end, I do
    _profiler.stop() and _profiler.close(), and then I try to load the stats back
    with hotshot.stats.load() with the same path. Well, it finds the file just
    fine, but it can't read it. It raises an IndexError from trying to pop an empty
    list or some such. I've tested Hotshot with simple test scripts in pure Python
    and it works just fine. I thought Psyco might be the problem, but I commented
    out all use of Psyco in my script and the problem still exists, so I think the
    use of C++ to call Python code is messing it up.

    Is this a bug in the Hotshot package? Is there a workaround? Oh, yes, I also
    uploaded the offending profiler stats snapshot, in case you want to look at it.

    http://members.aol.com/keflimarcvsx/hotshot.prof

    If you open it in the interactive Python interpreter and try
    hotshot.stats.load('hotshot.prof') you should get the same result I do,
    complete with traceback.

    - Kef

    P.S. Sorry for making this post so long :p
    KefX, Oct 24, 2003
    #1
    1. Advertising

  2. KefX

    Neil Hodgson Guest

    KefX:
    > Well, it finds the file just
    > fine, but it can't read it. It raises an IndexError from trying to pop an

    empty
    > list or some such. I've tested Hotshot with simple test scripts in pure

    Python
    > and it works just fine.


    I saw a similar issue that could be 'fixed' by running Python in
    optimized '-O' mode rather than normal mode as -O turns off assertions.

    Neil
    Neil Hodgson, Oct 25, 2003
    #2
    1. Advertising

  3. KefX

    Neil Hodgson Guest

    I wrote:

    > I saw a similar issue that could be 'fixed' by running Python in
    > optimized '-O' mode rather than normal mode as -O turns off assertions.


    Disregard that - it doesn't solve your problem.

    Neil
    Neil Hodgson, Oct 25, 2003
    #3
  4. Nope...doesn't fix anything.

    - Kef

    > I saw a similar issue that could be 'fixed' by running Python in
    >optimized '-O' mode rather than normal mode as -O turns off assertions.
    Kef Li Marcus X, Oct 25, 2003
    #4
  5. KefX

    KefX Guest

    Well, I found a rather unhelpful non-solution (just posting about it so nobody
    else suggests it). I figured that the Work() function (being called the most
    often given the architecture of the plugin, and the only one called from the
    C++ code while outputting sound) would probably have the biggest bottleneck, so
    I renamed my Work function to Work_Impl, and added a new Work function with
    this:

    def Work(num_samples):
    if _profiler is not None:
    return _profiler.runcall(Work_Impl, num_samples)
    else:
    return Work_Impl(num_samples)

    This technically worked (my code finally produced a readable log in the proper
    place). However, it was a failure in that this seemed to circumvent Psyco
    completely (making my code WAY too slow), and the results weren't too helpful
    in any case. An obvious solution is to make sure to call psyco.bind(Work_Impl),
    and while this did speed it up I get no information on the functions that
    Work_Impl called. Is there any other way to profile code that uses Psyco?
    (Other than the older 'profile' module, which is probably even less likely to
    work well...)
    KefX, Oct 25, 2003
    #5
  6. kefx> (Other than the older 'profile' module, which is probably even
    kefx> less likely to work well...)

    The profile module might help, though since it's written in Python, it will
    definitely run slower than the hotshot profiler (what about profile+psyco?).
    If nothing else you should be able to get some call counts. Another option
    is to use the trace module to count executed lines. If you submit a bug
    report to SourceForge (http://sourceforge.net/projects/python and assign it
    to Fred Drake (fdrake I believe) you'll at least be alerting the primary
    author of the hotshot code.

    Skip
    Skip Montanaro, Oct 25, 2003
    #6
  7. KefX

    James Kew Guest

    "KefX" <> wrote in message
    news:...
    > See, in my module's init code, I'm basically
    > I'm creating the hotshot.Profile object with the required path, bound to
    > _profiler, then I do _profiler.start() ...at the very end, I do
    > _profiler.stop() and _profiler.close(), and then I try to load the stats

    back
    > with hotshot.stats.load() with the same path. Well, it finds the file just
    > fine, but it can't read it. It raises an IndexError from trying to pop an

    empty
    > list or some such.


    Not that this is going to help, but: me too. I've seen the exact same
    problem -- loading saved stats throws an exception -- when profiling with
    Hotshot.

    This was in a pure-Python project, and it seemed sensitive to the exact code
    being profiled: it'd work fine on some inputs, but then I'd change one or
    two lines to attempt an optimisation and it'd fail.

    I ended up giving up and going back to profile -- slower, but livable with.

    James
    James Kew, Oct 25, 2003
    #7
  8. KefX

    John J. Lee Guest

    OSPAM (KefX) writes:

    > Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed
    > Python into a plugin for a freeware (closed-source) Windows music program
    > called Jeskola Buzz. (Man, I can't believe how many people keep telling me
    > "Extend Python instead!" when clearly I can't do that in this case. If somebody
    > here tells me this, I will explode in an appropriately gory fashion.) It's a

    [...]

    Why don't you extend Python instead?

    <stands back and watches with interest>


    John
    John J. Lee, Oct 25, 2003
    #8
  9. KefX

    KefX Guest

    >> (Man, I can't believe how many people keep telling me
    >> "Extend Python instead!" when clearly I can't do that in this case. If

    somebody
    >> here tells me this, I will explode in an appropriately gory fashion.)


    >Why don't you extend Python instead?
    >
    ><stands back and watches with interest>


    *** KABOOM ***

    - Kef
    KefX, Oct 25, 2003
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Paul McGuire

    Getting line-level timings using hotshot

    Paul McGuire, Dec 22, 2003, in forum: Python
    Replies:
    0
    Views:
    340
    Paul McGuire
    Dec 22, 2003
  2. Irmen de Jong
    Replies:
    0
    Views:
    439
    Irmen de Jong
    Jun 9, 2004
  3. Rodrigo Daunaravicius

    interpreting hotshot results

    Rodrigo Daunaravicius, Jun 16, 2004, in forum: Python
    Replies:
    0
    Views:
    306
    Rodrigo Daunaravicius
    Jun 16, 2004
  4. Irmen de Jong
    Replies:
    0
    Views:
    253
    Irmen de Jong
    Oct 30, 2004
  5. Irmen de Jong
    Replies:
    2
    Views:
    339
    Irmen de Jong
    Nov 3, 2004
Loading...

Share This Page