Strange Hotshot problem...

[KefX]

Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed
Python into a plugin for a freeware (closed-source) Windows music program
called Jeskola Buzz. (Man, I can't believe how many people keep telling me
"Extend Python instead!" when clearly I can't do that in this case. If somebody
here tells me this, I will explode in an appropriately gory fashion.) It's a
great program, and it works mainly on plugins other people write. Well, I had
been working on a guitar synth plugin in the summer, which started to sound
good. Then I lost the code (I was trying to rewrite it anyway but I wanted to
keep it as reference.) So I began to think about ways to try to make it better.

I eventually hit on the idea of using a scripting language to do this. I
started designing a language with a simple grammar (I like doing this kind of
thing in a semi-compulsive manner, heh)...but I tossed that aside, worked on a
ROM hacking/translating project some...well, my ROM hacking project required me
to write a small script, so I decided it was finally time to learn Python.

Geez, I love this language.

Anyway, now it's a couple months later, translation project gets put on hold
for reasons outside of my control, so I'm trying the guitar synth again. And I
thought...why don't I use Python? I was told I could never get it to run in
real time (we're talking about DSP, after all). I was about to give up on
pursuing the idea when I heard about Psyco, which should help speed up
everything that isn't rewritten as a C extension. I decided I'll give that a
shot, no matter how ridiculous the idea seems. My experience with ridiculous
ideas is that sometimes they actually work.

Well, to make a long story longer, I got it running and it does produce sound
(not the sound I would like to hear from it, though!) But brokenness of the
sound suggested that it's running too freakin' slow, and it was: Buzz's CPU
display said my own plugin was hogging the whole CPU! Well, okay, it's not
hopeless yet, I can profile it and see what I can do about the bottlenecks.

Which finally leads me to my problem: I can't get Hotshot to work properly. I
can't fix a bottleneck if I don't know where it is! Remember, I'm calling
various Python functions from C++. See, in my module's init code, I'm basically
I'm creating the hotshot.Profile object with the required path, bound to
_profiler, then I do _profiler.start() ...at the very end, I do
_profiler.stop() and _profiler.close(), and then I try to load the stats back
with hotshot.stats.load() with the same path. Well, it finds the file just
fine, but it can't read it. It raises an IndexError from trying to pop an empty
list or some such. I've tested Hotshot with simple test scripts in pure Python
and it works just fine. I thought Psyco might be the problem, but I commented
out all use of Psyco in my script and the problem still exists, so I think the
use of C++ to call Python code is messing it up.

Is this a bug in the Hotshot package? Is there a workaround? Oh, yes, I also
uploaded the offending profiler stats snapshot, in case you want to look at it.

http://members.aol.com/keflimarcvsx/hotshot.prof

If you open it in the interactive Python interpreter and try
hotshot.stats.load('hotshot.prof') you should get the same result I do,
complete with traceback.

- Kef

P.S. Sorry for making this post so long

 

[Neil Hodgson]

KefX:

Well, it finds the file just
fine, but it can't read it. It raises an IndexError from trying to pop an empty
list or some such. I've tested Hotshot with simple test scripts in pure Python
and it works just fine.

I saw a similar issue that could be 'fixed' by running Python in
optimized '-O' mode rather than normal mode as -O turns off assertions.

Neil

 

[Neil Hodgson]

I saw a similar issue that could be 'fixed' by running Python in
optimized '-O' mode rather than normal mode as -O turns off assertions.

Disregard that - it doesn't solve your problem.

Neil

 

[Kef Li Marcus X]

Nope...doesn't fix anything.

- Kef

 

[KefX]

Well, I found a rather unhelpful non-solution (just posting about it so nobody
else suggests it). I figured that the Work() function (being called the most
often given the architecture of the plugin, and the only one called from the
C++ code while outputting sound) would probably have the biggest bottleneck, so
I renamed my Work function to Work_Impl, and added a new Work function with
this:

def Work(num_samples):
if _profiler is not None:
return _profiler.runcall(Work_Impl, num_samples)
else:
return Work_Impl(num_samples)

This technically worked (my code finally produced a readable log in the proper
place). However, it was a failure in that this seemed to circumvent Psyco
completely (making my code WAY too slow), and the results weren't too helpful
in any case. An obvious solution is to make sure to call psyco.bind(Work_Impl),
and while this did speed it up I get no information on the functions that
Work_Impl called. Is there any other way to profile code that uses Psyco?
(Other than the older 'profile' module, which is probably even less likely to
work well...)

 

[Skip Montanaro]

kefx> (Other than the older 'profile' module, which is probably even
kefx> less likely to work well...)

The profile module might help, though since it's written in Python, it will
definitely run slower than the hotshot profiler (what about profile+psyco?).
If nothing else you should be able to get some call counts. Another option
is to use the trace module to count executed lines. If you submit a bug
report to SourceForge (http://sourceforge.net/projects/python and assign it
to Fred Drake (fdrake I believe) you'll at least be alerting the primary
author of the hotshot code.

Skip

 

[James Kew]

See, in my module's init code, I'm basically
I'm creating the hotshot.Profile object with the required path, bound to
_profiler, then I do _profiler.start() ...at the very end, I do
_profiler.stop() and _profiler.close(), and then I try to load the stats back
with hotshot.stats.load() with the same path. Well, it finds the file just
fine, but it can't read it. It raises an IndexError from trying to pop an empty
list or some such.

Not that this is going to help, but: me too. I've seen the exact same
problem -- loading saved stats throws an exception -- when profiling with
Hotshot.

This was in a pure-Python project, and it seemed sensitive to the exact code
being profiled: it'd work fine on some inputs, but then I'd change one or
two lines to attempt an optimisation and it'd fail.

I ended up giving up and going back to profile -- slower, but livable with.

James

 

[John J. Lee]

Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed
Python into a plugin for a freeware (closed-source) Windows music program
called Jeskola Buzz. (Man, I can't believe how many people keep telling me
"Extend Python instead!" when clearly I can't do that in this case. If somebody
here tells me this, I will explode in an appropriately gory fashion.) It's a

[...]

Why don't you extend Python instead?

<stands back and watches with interest>


John

 

[KefX]

(Man, I can't believe how many people keep telling me

Why don't you extend Python instead?

<stands back and watches with interest>

*** KABOOM ***

- Kef