How to increase number of threads per process?

[Ronan Viernes]

Hi,

I have created a python script (see below) to count the maximum number
of threads per process (by starting new threads continuously until it
breaks).

######
#testThread.py
import thread, sys

def main():
print "Main Thread:", thread.get_ident()
count = 0;
try:
while 1:
thread.start_new_thread(test,(`count`,))
count = count + 1;
except:
print "Total Threads:", count
print "Exiting Main Thread:", thread.get_ident()
raise

def test(input=None):
print "count:", thread.get_ident(), input
while 1: #keep thread alive until it breaks
pass

if __name__ == "__main__":
main()

#####

Results:

1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
Max Threads = 1024

2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
Max Threads = 1024

3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
Max Threads = 256

4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
Max Threads = 512

Note:
For all setup, SuSE Linux threads-max=14336 and max_map_count=65536

Questions:
1. How to determine the number of threads? Is it something
configurable?
2. Why do the results above differ in output?
3. How to increase the maximum number of threads per process?


Thanks and hope to hear soon.
Ronan

 

[Steve]

Ronan,

Hi,

I have created a python script (see below) to count the maximum number
of threads per process (by starting new threads continuously until it
breaks).

######
#testThread.py
import thread, sys

def main():
print "Main Thread:", thread.get_ident()
count = 0;
try:
while 1:
thread.start_new_thread(test,(`count`,))
count = count + 1;
except:
print "Total Threads:", count
print "Exiting Main Thread:", thread.get_ident()
raise

def test(input=None):
print "count:", thread.get_ident(), input
while 1: #keep thread alive until it breaks
pass

if __name__ == "__main__":
main()

#####

Results:

1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
Max Threads = 1024

2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
Max Threads = 1024

3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
Max Threads = 256

4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
Max Threads = 512

Note:
For all setup, SuSE Linux threads-max=14336 and max_map_count=65536

Questions:
1. How to determine the number of threads? Is it something
configurable?
2. Why do the results above differ in output?
3. How to increase the maximum number of threads per process?

I'm not sure if this is configurable but I was testing out something
similar today (on fedora core 1). I discovered that a structure such as
the following works with any number of thread (I tried it with 2000
threads just now). The main thing I did was introducing a
'time.sleep(0.01)' in the loop that waits for the worker threads to join
the main thread, and adding a try..except clause around where you start
the threads. Basically an exception is never thrown because I run 10
threads at one go, and then sleep for a fraction of a second (it's
usually a good idea to sleep/yield in busy loops). I think the main
thing here is the fact that it takes 10 threads at a time and then gives
it a pause.

Please try this out and let me know what you find.

class A:
cache = {}
counter = 0
def local(self):
A.cache[A.counter*2] = A.counter
A.counter += 1

def print_it(self):
print A.cache, len(A.cache), A.counter


def test():
count = 2000 # I'm pretty sure a larger number should work too

a = A()
threadList = []
for i in xrange(count):
try:
t = threading.Thread(target=a.local, args=())
threadList.append(t)
t.start()

# Adding the following if statement allowed running more
than 256 threads in one go
# Previously it would run only 256 threads if this clause
wasn't there.
if i % 10 == 0 and i != 0:
# setting sys.setcheckinterval to 1 didn't help. Let's try
# voluntarily giving up some cycles -- this should give
# other threads a chance to run
time.sleep(0.01) # this works with time.sleep(0.005)
as well
# end if
except:
continue
# end try
# end for

# Wait for threads to join the main thread
while threading.activeCount()>1:
time.sleep(1)

a.print_it()

test()

Thanks and hope to hear soon.
Ronan

Steve

 

[Ronan Viernes]

Hi Steve,

Thanks for the info. Unfortunately, adding the sleep to my previous
example (modified version below) did not alter the results. Please
take note that I am using thread library (and not threading). The
sample code below is simplified version of our system and of course
each of the thread has other tasks to do rather than staying idle in
there.

What really puzzles me now is, if I change the environment (OS and
Python Versions) accordingly it will give different results. This made
me believe that it is somewhat 'configurable'.

Anyway, I really appreciate the help. Let's wait what the others have
to say.

Regards,
Ronan

 

[John Lenton]

Hi Steve,

Thanks for the info. Unfortunately, adding the sleep to my previous
example (modified version below) did not alter the results. Please
take note that I am using thread library (and not threading). The
sample code below is simplified version of our system and of course
each of the thread has other tasks to do rather than staying idle in
there.

What really puzzles me now is, if I change the environment (OS and
Python Versions) accordingly it will give different results. This made
me believe that it is somewhat 'configurable'.

as far as I know, it's the way your glibc is compiled that makes the difference.