How to increase number of threads per process?

Discussion in 'Python' started by Ronan Viernes, Jul 13, 2004.

  1. Hi,

    I have created a python script (see below) to count the maximum number
    of threads per process (by starting new threads continuously until it
    breaks).

    ######
    #testThread.py
    import thread, sys

    def main():
    print "Main Thread:", thread.get_ident()
    count = 0;
    try:
    while 1:
    thread.start_new_thread(test,(`count`,))
    count = count + 1;
    except:
    print "Total Threads:", count
    print "Exiting Main Thread:", thread.get_ident()
    raise

    def test(input=None):
    print "count:", thread.get_ident(), input
    while 1: #keep thread alive until it breaks
    pass

    if __name__ == "__main__":
    main()

    #####

    Results:

    1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
    Max Threads = 1024

    2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
    Max Threads = 1024

    3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
    Max Threads = 256

    4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
    Max Threads = 512

    Note:
    For all setup, SuSE Linux threads-max=14336 and max_map_count=65536

    Questions:
    1. How to determine the number of threads? Is it something
    configurable?
    2. Why do the results above differ in output?
    3. How to increase the maximum number of threads per process?


    Thanks and hope to hear soon.
    Ronan
     
    Ronan Viernes, Jul 13, 2004
    #1
    1. Advertising

  2. Ronan Viernes

    Steve Guest

    Ronan,

    Ronan Viernes wrote:
    > Hi,
    >
    > I have created a python script (see below) to count the maximum number
    > of threads per process (by starting new threads continuously until it
    > breaks).
    >
    > ######
    > #testThread.py
    > import thread, sys
    >
    > def main():
    > print "Main Thread:", thread.get_ident()
    > count = 0;
    > try:
    > while 1:
    > thread.start_new_thread(test,(`count`,))
    > count = count + 1;
    > except:
    > print "Total Threads:", count
    > print "Exiting Main Thread:", thread.get_ident()
    > raise
    >
    > def test(input=None):
    > print "count:", thread.get_ident(), input
    > while 1: #keep thread alive until it breaks
    > pass
    >
    > if __name__ == "__main__":
    > main()
    >
    > #####
    >
    > Results:
    >
    > 1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
    > Max Threads = 1024
    >
    > 2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
    > Max Threads = 1024
    >
    > 3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
    > Max Threads = 256
    >
    > 4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
    > Max Threads = 512
    >
    > Note:
    > For all setup, SuSE Linux threads-max=14336 and max_map_count=65536
    >
    > Questions:
    > 1. How to determine the number of threads? Is it something
    > configurable?
    > 2. Why do the results above differ in output?
    > 3. How to increase the maximum number of threads per process?
    >


    I'm not sure if this is configurable but I was testing out something
    similar today (on fedora core 1). I discovered that a structure such as
    the following works with any number of thread (I tried it with 2000
    threads just now). The main thing I did was introducing a
    'time.sleep(0.01)' in the loop that waits for the worker threads to join
    the main thread, and adding a try..except clause around where you start
    the threads. Basically an exception is never thrown because I run 10
    threads at one go, and then sleep for a fraction of a second (it's
    usually a good idea to sleep/yield in busy loops). I think the main
    thing here is the fact that it takes 10 threads at a time and then gives
    it a pause.

    Please try this out and let me know what you find.

    class A:
    cache = {}
    counter = 0
    def local(self):
    A.cache[A.counter*2] = A.counter
    A.counter += 1

    def print_it(self):
    print A.cache, len(A.cache), A.counter


    def test():
    count = 2000 # I'm pretty sure a larger number should work too

    a = A()
    threadList = []
    for i in xrange(count):
    try:
    t = threading.Thread(target=a.local, args=())
    threadList.append(t)
    t.start()

    # Adding the following if statement allowed running more
    than 256 threads in one go
    # Previously it would run only 256 threads if this clause
    wasn't there.
    if i % 10 == 0 and i != 0:
    # setting sys.setcheckinterval to 1 didn't help. Let's try
    # voluntarily giving up some cycles -- this should give
    # other threads a chance to run
    time.sleep(0.01) # this works with time.sleep(0.005)
    as well
    # end if
    except:
    continue
    # end try
    # end for

    # Wait for threads to join the main thread
    while threading.activeCount()>1:
    time.sleep(1)

    a.print_it()

    test()


    >
    > Thanks and hope to hear soon.
    > Ronan


    Steve
     
    Steve, Jul 13, 2004
    #2
    1. Advertising

  3. Hi Steve,

    Thanks for the info. Unfortunately, adding the sleep to my previous
    example (modified version below) did not alter the results. Please
    take note that I am using thread library (and not threading). The
    sample code below is simplified version of our system and of course
    each of the thread has other tasks to do rather than staying idle in
    there.

    What really puzzles me now is, if I change the environment (OS and
    Python Versions) accordingly it will give different results. This made
    me believe that it is somewhat 'configurable'.

    Anyway, I really appreciate the help. Let's wait what the others have
    to say.

    > > ######
    > > #testThread.py
    > > import thread, sys
    > >
    > > def main():
    > > print "Main Thread:", thread.get_ident()
    > > count = 0;
    > > try:
    > > while 1:
    > > thread.start_new_thread(test,(`count`,))
    > > count = count + 1;
    > > if count % 10 == 0 and count != 0:
    > > time.sleep(0.01)
    > > except:
    > > print "Total Threads:", count
    > > print "Exiting Main Thread:", thread.get_ident()
    > > raise
    > >
    > > def test(input=None):
    > > print "count:", thread.get_ident(), input
    > > while 1: #keep thread alive until it breaks
    > > pass
    > >
    > > if __name__ == "__main__":
    > > main()
    > >
    > > #####
    > >
    > > Results:
    > >
    > > 1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
    > > Max Threads = 1024
    > >
    > > 2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
    > > Max Threads = 1024
    > >
    > > 3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
    > > Max Threads = 256
    > >
    > > 4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
    > > Max Threads = 512
    > >
    > > Note:
    > > For all setup, SuSE Linux threads-max=14336 and max_map_count=65536
    > >
    > > Questions:
    > > 1. How to determine the number of threads? Is it something
    > > configurable?
    > > 2. Why do the results above differ in output?
    > > 3. How to increase the maximum number of threads per process?
    > >


    Regards,
    Ronan
     
    Ronan Viernes, Jul 14, 2004
    #3
  4. Ronan Viernes

    John Lenton Guest

    On 13 Jul 2004 19:27:21 -0700, Ronan Viernes <> wrote:
    > Hi Steve,
    >
    > Thanks for the info. Unfortunately, adding the sleep to my previous
    > example (modified version below) did not alter the results. Please
    > take note that I am using thread library (and not threading). The
    > sample code below is simplified version of our system and of course
    > each of the thread has other tasks to do rather than staying idle in
    > there.
    >
    > What really puzzles me now is, if I change the environment (OS and
    > Python Versions) accordingly it will give different results. This made
    > me believe that it is somewhat 'configurable'.


    as far as I know, it's the way your glibc is compiled that makes the difference.

    --
    John Lenton () -- Random fortune:
    bash: fortune: command not found
     
    John Lenton, Jul 14, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?S01aX3N0YXRl?=

    Quick one - Is SESSION per browser instance or per IP Address?

    =?Utf-8?B?S01aX3N0YXRl?=, Apr 4, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    5,944
    gerry
    Apr 10, 2006
  2. Razvan
    Replies:
    1
    Views:
    437
    tony vee
    Sep 10, 2004
  3. Ronnan-Ian Viernes
    Replies:
    0
    Views:
    428
    Ronnan-Ian Viernes
    Jul 14, 2004
  4. John Lenton
    Replies:
    0
    Views:
    554
    John Lenton
    Jul 14, 2004
  5. Jason Kendall
    Replies:
    1
    Views:
    167
    Patrice
    Jun 1, 2005
Loading...

Share This Page