Need Psyco profiling help...

Discussion in 'Python' started by KefX, Oct 25, 2003.

  1. KefX

    KefX Guest

    Hey...as for what I'm doing with Python, look at my post "Strange Hotshot
    problem". To make a long story short, I'm embedding Python in order to write a
    plugin to a freeware music program; the plugin interfaces to the program with
    C++, but the plugin calls Python routines to do much (but not all) of the work.

    Which brings me to the problem. It's SLOW! Well, that's hardly surprising, but
    anyway...I'm using Psyco, and it's definitely at least twice as fast with it
    than it is without it. However this is not enough: my program produces crackles
    and such as a result of hogging the CPU so much that there's latency. So
    obviously I need to optimize.

    But to optimize, I need to find a bottleneck, using a profiler. An obvious way
    to do this with Psyco is to just use psyco.log('somefile.log') and then
    psyco.profile()...however most of the time seems to be spent in Psyco's
    active_start function! I decided the profiler may well be screwy so I tries
    Hotshot. Well, Hotshot and Psyco don't work well together, I found out. But my
    buddy CyanPhase suggested I write a do-nothing function in place of my Work
    function, bind it with Psyco, and see how much CPU it takes. This function
    simply filled an array of floats with zeroes and returned it (because the
    caller needed some data to copy to another buffer). This only took 2.5% of the
    CPU! Therefore the overhead of using Psyco certainly can't be eating up the
    unaccounted-for 97.5% of the CPU, it has to be my code. But Psyco's profiler
    says it isn't.

    What's going on here? How can I find where the real bottleneck is so I can fix
    it?

    - Kef
     
    KefX, Oct 25, 2003
    #1
    1. Advertising

  2. On Sat, Oct 25, 2003 at 01:49:40AM +0000, KefX wrote:
    >
    > But to optimize, I need to find a bottleneck, using a profiler. An obvious way
    > to do this with Psyco is to just use psyco.log('somefile.log') and then
    > psyco.profile()...however most of the time seems to be spent in Psyco's
    > active_start function! I decided the profiler may well be screwy so I tries


    As far as I'm aware, there's no way to use the standard python profiling
    tools to profile non-python functions, like C extension functions or
    psyco'd functions. Your best bet is to probably profile without psyco, and
    optimise the bottlenecks you find in pure python. When you're done, try
    using psyco again and see if performance is good enough. Very roughly
    speaking, two psyco'd functions are likely to have roughly the same relative
    performance as they did before psycoing, so hopefully this approach is good
    enough to get the improvements you need.

    -Andrew.
     
    Andrew Bennetts, Oct 25, 2003
    #2
    1. Advertising

  3. KefX

    Brian Kelley Guest

    Can you run any test code *outside* of hotshot? This would certainly
    tell you whether it was worth the effort. I.e. if you can't get it fast
    outside of hotshot, it won't work inside ;)
     
    Brian Kelley, Oct 25, 2003
    #3
  4. KefX

    KefX Guest

    >Can you run any test code *outside* of hotshot? This would certainly
    >tell you whether it was worth the effort. I.e. if you can't get it fast
    >outside of hotshot, it won't work inside ;)


    Um...what?
    - Kef
     
    KefX, Oct 26, 2003
    #4
  5. KefX

    John J. Lee Guest

    OSPAM (KefX) writes:

    > >Can you run any test code *outside* of hotshot? This would certainly
    > >tell you whether it was worth the effort. I.e. if you can't get it fast
    > >outside of hotshot, it won't work inside ;)

    >
    > Um...what?


    s/hotshot/psyco/ I guess


    John
     
    John J. Lee, Oct 26, 2003
    #5
  6. KefX

    Brian Kelley Guest

    KefX wrote:
    >>Can you run any test code *outside* of hotshot? This would certainly
    >>tell you whether it was worth the effort. I.e. if you can't get it fast
    >>outside of hotshot, it won't work inside ;)

    >
    >
    > Um...what?
    > - Kef


    Oops, I got the names confused. I was wondering if you were
    profiling/running your code inside the plugin or outside. I have
    noticed problems using psyco in a plugin because it kept forgetting that
    it had compiled the code before so it spent most of its time optimizing
    but never really using the optimization.

    Brian
     
    Brian Kelley, Oct 26, 2003
    #6
  7. KefX

    Armin Rigo Guest

    KefX wrote:
    > Hotshot. Well, Hotshot and Psyco don't work well together, I found out.


    You seem not to have thought about running Hotshot only (without Psyco)
    to find out where your code spends most of its time. Psyco not only
    isn't happy with Hotshot, but whatever results Hotshot could get are
    wrong because it doesn't know about machine-code-compiled functions.

    Also note that if the Python code has no long loop, e.g. if it is only
    functions that are repeatedly called by C/C++ code (with the loop in
    C/C++), then Psyco won't help at all, it will probably even slow things
    down. It can only accelerate functions that run for some time (I mean,
    doing at least 100 or 1000 iterations to fill some array should be ok,
    but if C/C++ calls your function 100 or 1000 times to fill the same
    array it is not).


    A bientot,

    Armin.
     
    Armin Rigo, Oct 26, 2003
    #7
  8. KefX

    KefX Guest

    >Also note that if the Python code has no long loop, e.g. if it is only
    >functions that are repeatedly called by C/C++ code (with the loop in
    >C/C++), then Psyco won't help at all, it will probably even slow things
    >down. It can only accelerate functions that run for some time (I mean,
    >doing at least 100 or 1000 iterations to fill some array should be ok,
    >but if C/C++ calls your function 100 or 1000 times to fill the same
    >array it is not).


    Psyco does speed it up considerably. I don't have any profiling data to back it
    up, but the latency I hear in my synthesized instrument is greatly reduced when
    I use Psyco. Or to put it in a sillier fashion, if the sound my synth should
    make is "PIIIING!", with Psyco it goes "PI-I-I-ING!", and without it, it goes
    "PI...I...I...I...N...G..." (in other words, the difference is VERY audible.
    :))

    The reason I don't really want to profile without Psyco is that I'm worried
    that the data may not be representative of the conditions with Psyco. However,
    it's becoming increasingly apparent that I may have no other choice.

    - Kef
     
    KefX, Oct 26, 2003
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ingo Blank

    psyco + expat SEGVs

    Ingo Blank, Dec 13, 2003, in forum: Python
    Replies:
    1
    Views:
    318
    Andrew MacIntyre
    Dec 15, 2003
  2. Delaney, Timothy C (Timothy)

    RE: What psyco is goot at [Was: Rookie Speaks]

    Delaney, Timothy C (Timothy), Jan 8, 2004, in forum: Python
    Replies:
    2
    Views:
    316
    Peter Hansen
    Jan 9, 2004
  3. Jeremy Sanders

    Psyco distutils problem

    Jeremy Sanders, Jan 23, 2004, in forum: Python
    Replies:
    0
    Views:
    323
    Jeremy Sanders
    Jan 23, 2004
  4. Han Benjamin

    Psyco on PowerPC?

    Han Benjamin, Jan 23, 2004, in forum: Python
    Replies:
    3
    Views:
    401
    Andrew MacIntyre
    Jan 25, 2004
  5. Martin Jansson
    Replies:
    5
    Views:
    168
    Isaac Gouy
    Sep 12, 2005
Loading...

Share This Page