Maximum number of threads

[Kirby Angell]

We are porting our applications from Python 2.2 on RH9 to Python 2.3 on
RH9. One of our apps used to be able to create threads with wild
abandon, 800 or more if necessary. With Python 2.3 we are hitting a
hard limit of 254 on RH9 and around 400 on FC2.

What is the limiting factor for the number of threads we can create? Is
there a way to increase it?

-- Kirby

 

[Michael Fuhr]

We are porting our applications from Python 2.2 on RH9 to Python 2.3 on
RH9. One of our apps used to be able to create threads with wild
abandon, 800 or more if necessary. With Python 2.3 we are hitting a
hard limit of 254 on RH9 and around 400 on FC2.

What happens when you hit this "hard limit"? If there's an error
message then it might hint at the problem.

What is the limiting factor for the number of threads we can create? Is
there a way to increase it?

I'm not current on Linux threading models -- do these versions of
Linux create a new process for each thread? If so, could resource
limits be preventing you from creating more threads?

 

[Kirby Angell]

What happens when you hit this "hard limit"? If there's an error
message then it might hint at the problem.

If I run this program:

import thread, time

def t(parm1,parm2):
time.sleep( 1000 )

tc = 0
try:
while 1:
thread.start_new_thread( t, (None,None) )
tc += 1
time.sleep( 0.05 )
finally:
print tc


This is the result:
$ python t.py
404
Traceback (most recent call last):
File "t.py", line 12, in ?
thread.start_new_thread( t, (None,None) )
thread.error: can't start new thread

I'm not current on Linux threading models -- do these versions of
Linux create a new process for each thread? If so, could resource
limits be preventing you from creating more threads?

I don't think Linux creates a new process for each thread. Sure seems
like this would have to be something related to python since 2.2 could
create a thousand or more threads. Although I have to admit that it is
odd that the maximum number of threads I can create on RH9 is different
than on FC2.

 

[Jeremy Jones]

If I run this program:




I don't think Linux creates a new process for each thread. Sure seems
like this would have to be something related to python since 2.2 could
create a thousand or more threads. Although I have to admit that it
is odd that the maximum number of threads I can create on RH9 is
different than on FC2.

Versions of RedHat prior to 9 did something that made it appear that
each thread was a process. I don't know all the ins and outs (I think
it had something to do with user-space vs. kernel-space), but when the
Native Posix Thread Library (NPTL) appeard in RH9, each thread did not
show up as its own process (say, in top). I think it's probably an OS
limitation. I kicked off the same script on Windows XP that I kicked
off on FC2 and XP was able to generate over 2000 threads as opposed to
around 400 for FC2. I may be wrong, but I think Python relies on the
underlying OS threading libraries for everything threading related.
Which is probably why the recent change in either 2.2 or 2.3 for a dummy
threading module that simulates threading on systems that do not support
threading. I know my babbling isn't getting you any closer to a
solution.....


Jeremy Jones

 

[Kirby Angell]

around 400 for FC2. I may be wrong, but I think Python relies on the
underlying OS threading libraries for everything threading related.
Which is probably why the recent change in either 2.2 or 2.3 for a dummy
threading module that simulates threading on systems that do not support
threading. I know my babbling isn't getting you any closer to a
solution.....

No, your babbling has gotten me much closer. I'm not sure what changed
between Python 2.2 and Python 2.3, but the problem is the stack size
allocated to each thread. Apparently with glibc on RH9 something along
the order of 8 MB of virtual memory is allocated for the stack of each
thread. 256 threads takes us to the virtual memory limit of the process
itself. Glibc on FC2 must be using a different default stack size, or
the virtual memory limit is higher; I've only recently started running
FC2 on our test servers so I'm not sure which it is.

The only way to change the stack size is to define THREAD_STACK_SIZE in
the thread_pthread.h header. There is an ifdef for the Mac that sets
this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
and recompiled Python. I can now create 16k threads although I'm not
sure yet whether our application will run; some tuning is probably in
order. I'm going to get the 2.2 source code and see what is different
between the two and maybe get an idea of what a reasonable stack size is.

There is a very nice write-up on the basic issue here:

http://www.kegel.com/stackcheck/

Thanks for your help.

 

[Andrew MacIntyre]

No, your babbling has gotten me much closer. I'm not sure what changed
between Python 2.2 and Python 2.3, but the problem is the stack size
allocated to each thread. Apparently with glibc on RH9 something along
the order of 8 MB of virtual memory is allocated for the stack of each
thread. 256 threads takes us to the virtual memory limit of the process
itself. Glibc on FC2 must be using a different default stack size, or
the virtual memory limit is higher; I've only recently started running
FC2 on our test servers so I'm not sure which it is.

The only way to change the stack size is to define THREAD_STACK_SIZE in
the thread_pthread.h header. There is an ifdef for the Mac that sets
this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
and recompiled Python. I can now create 16k threads although I'm not
sure yet whether our application will run; some tuning is probably in
order. I'm going to get the 2.2 source code and see what is different
between the two and maybe get an idea of what a reasonable stack size is.

ISTR that the default stacksize for the LinuxThreads threading
implementation (prior to NPTL) was 1MB.

It is also worth noting that more recent versions of gcc (3.x) are
generating larger stack frames than older versions; something that has
affected building the SRE module until it was made non-recursive for
Python 2.4. This has probably encouraged the bump in the default stack
size.

 

[G. S. Hayes]

Versions of RedHat prior to 9 did something that made it appear that
each thread was a process.

Older versions of ps/top showed all threads instead of just processes.

You can still see them with the right flags (e.g. the H option to
ps, "ps auxH").

I think it's probably an OS limitation. I kicked off the same script on
Windows XP that I kicked off on FC2 and XP was able to generate over
2000 threads as opposed to around 400 for FC2.

I have no problems creating 2000 threads in FC2 from C. haven't tried
it in python. Resource limits are the first issue that comes to mind
for me, followed by kernel versions (esp. if you're running a 2.2 or
earlier kernel for some reason).

 

[Kirby Angell]

Just to finish off this thread...

You can set the constant in the header file I mentioned, but you can
also just do (on Linux):

ulimit -s <stacksize>

to set the stack size that will be used. I can control how many threads
my Python apps can create by setting this to different values. You'll
probably also want to set the number of files (ulimit -n) to something
different if you are creating a socket in each of those threads.

--
Thank you,

Kirby Angell
Get notified anytime your website goes down!
http://www.alertra.com
key: 9004F4C0
fingerprint: DD7E E88D 7F50 2A1E 229D 836A DB5B A751 9004 F4C0