overhead of starting threads

[Will McGugan]

Hi,

What is the overhead of starting threads in Python? Currently my app
starts around 20 threads per second, but I'm considering creating a pool
of a fixed number of threads and keeping them fed with data. But only if
it would give me a noticeable performance boost.

Has anyone got any any experience in this area that they could share? -
before I start investigating myself..

I'm running on Windows, if thats relevant.


Regards,

Will McGugan

 

[Simon Percivall]

How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.

 

[Will McGugan]

How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.

I wasn't asking about wether to use threads or not, I already am - for
several long running partialy I/O bound tasks.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.

I haven't had much luck with the profile module in the standard library.
It only shows me data from the main thread. Is there a better way of
profile heavily threaded code?


Will McGugan