Need Psyco profiling help...

[KefX]

Hey...as for what I'm doing with Python, look at my post "Strange Hotshot
problem". To make a long story short, I'm embedding Python in order to write a
plugin to a freeware music program; the plugin interfaces to the program with
C++, but the plugin calls Python routines to do much (but not all) of the work.

Which brings me to the problem. It's SLOW! Well, that's hardly surprising, but
anyway...I'm using Psyco, and it's definitely at least twice as fast with it
than it is without it. However this is not enough: my program produces crackles
and such as a result of hogging the CPU so much that there's latency. So
obviously I need to optimize.

But to optimize, I need to find a bottleneck, using a profiler. An obvious way
to do this with Psyco is to just use psyco.log('somefile.log') and then
psyco.profile()...however most of the time seems to be spent in Psyco's
active_start function! I decided the profiler may well be screwy so I tries
Hotshot. Well, Hotshot and Psyco don't work well together, I found out. But my
buddy CyanPhase suggested I write a do-nothing function in place of my Work
function, bind it with Psyco, and see how much CPU it takes. This function
simply filled an array of floats with zeroes and returned it (because the
caller needed some data to copy to another buffer). This only took 2.5% of the
CPU! Therefore the overhead of using Psyco certainly can't be eating up the
unaccounted-for 97.5% of the CPU, it has to be my code. But Psyco's profiler
says it isn't.

What's going on here? How can I find where the real bottleneck is so I can fix
it?

- Kef

 

[Andrew Bennetts]

But to optimize, I need to find a bottleneck, using a profiler. An obvious way
to do this with Psyco is to just use psyco.log('somefile.log') and then
psyco.profile()...however most of the time seems to be spent in Psyco's
active_start function! I decided the profiler may well be screwy so I tries

As far as I'm aware, there's no way to use the standard python profiling
tools to profile non-python functions, like C extension functions or
psyco'd functions. Your best bet is to probably profile without psyco, and
optimise the bottlenecks you find in pure python. When you're done, try
using psyco again and see if performance is good enough. Very roughly
speaking, two psyco'd functions are likely to have roughly the same relative
performance as they did before psycoing, so hopefully this approach is good
enough to get the improvements you need.

-Andrew.

 

[Brian Kelley]

Can you run any test code *outside* of hotshot? This would certainly
tell you whether it was worth the effort. I.e. if you can't get it fast
outside of hotshot, it won't work inside

 

[KefX]

Can you run any test code *outside* of hotshot? This would certainly

tell you whether it was worth the effort. I.e. if you can't get it fast
outside of hotshot, it won't work inside

Um...what?
- Kef

 

[John J. Lee]

Um...what?

s/hotshot/psyco/ I guess


John

 

[Brian Kelley]

Um...what?
- Kef

Oops, I got the names confused. I was wondering if you were
profiling/running your code inside the plugin or outside. I have
noticed problems using psyco in a plugin because it kept forgetting that
it had compiled the code before so it spent most of its time optimizing
but never really using the optimization.

Brian

 

[Armin Rigo]

Hotshot. Well, Hotshot and Psyco don't work well together, I found out.

You seem not to have thought about running Hotshot only (without Psyco)
to find out where your code spends most of its time. Psyco not only
isn't happy with Hotshot, but whatever results Hotshot could get are
wrong because it doesn't know about machine-code-compiled functions.

Also note that if the Python code has no long loop, e.g. if it is only
functions that are repeatedly called by C/C++ code (with the loop in
C/C++), then Psyco won't help at all, it will probably even slow things
down. It can only accelerate functions that run for some time (I mean,
doing at least 100 or 1000 iterations to fill some array should be ok,
but if C/C++ calls your function 100 or 1000 times to fill the same
array it is not).


A bientot,

Armin.

 

[KefX]

Also note that if the Python code has no long loop, e.g. if it is only

functions that are repeatedly called by C/C++ code (with the loop in
C/C++), then Psyco won't help at all, it will probably even slow things
down. It can only accelerate functions that run for some time (I mean,
doing at least 100 or 1000 iterations to fill some array should be ok,
but if C/C++ calls your function 100 or 1000 times to fill the same
array it is not).

Psyco does speed it up considerably. I don't have any profiling data to back it
up, but the latency I hear in my synthesized instrument is greatly reduced when
I use Psyco. Or to put it in a sillier fashion, if the sound my synth should
make is "PIIIING!", with Psyco it goes "PI-I-I-ING!", and without it, it goes
"PI...I...I...I...N...G..." (in other words, the difference is VERY audible.
)

The reason I don't really want to profile without Psyco is that I'm worried
that the data may not be representative of the conditions with Psyco. However,
it's becoming increasingly apparent that I may have no other choice.

- Kef