Maximum number of threads

Discussion in 'Python' started by Kirby Angell, Sep 29, 2004.

  1. Kirby Angell

    Kirby Angell Guest

    We are porting our applications from Python 2.2 on RH9 to Python 2.3 on
    RH9. One of our apps used to be able to create threads with wild
    abandon, 800 or more if necessary. With Python 2.3 we are hitting a
    hard limit of 254 on RH9 and around 400 on FC2.

    What is the limiting factor for the number of threads we can create? Is
    there a way to increase it?

    -- Kirby
    Kirby Angell, Sep 29, 2004
    #1
    1. Advertising

  2. Kirby Angell

    Michael Fuhr Guest

    Kirby Angell <> writes:

    > We are porting our applications from Python 2.2 on RH9 to Python 2.3 on
    > RH9. One of our apps used to be able to create threads with wild
    > abandon, 800 or more if necessary. With Python 2.3 we are hitting a
    > hard limit of 254 on RH9 and around 400 on FC2.


    What happens when you hit this "hard limit"? If there's an error
    message then it might hint at the problem.

    > What is the limiting factor for the number of threads we can create? Is
    > there a way to increase it?


    I'm not current on Linux threading models -- do these versions of
    Linux create a new process for each thread? If so, could resource
    limits be preventing you from creating more threads?

    --
    Michael Fuhr
    http://www.fuhr.org/~mfuhr/
    Michael Fuhr, Sep 29, 2004
    #2
    1. Advertising

  3. Kirby Angell

    Kirby Angell Guest

    Michael Fuhr wrote:
    > What happens when you hit this "hard limit"? If there's an error
    > message then it might hint at the problem.


    If I run this program:

    import thread, time

    def t(parm1,parm2):
    time.sleep( 1000 )

    tc = 0
    try:
    while 1:
    thread.start_new_thread( t, (None,None) )
    tc += 1
    time.sleep( 0.05 )
    finally:
    print tc


    This is the result:
    $ python t.py
    404
    Traceback (most recent call last):
    File "t.py", line 12, in ?
    thread.start_new_thread( t, (None,None) )
    thread.error: can't start new thread

    > I'm not current on Linux threading models -- do these versions of
    > Linux create a new process for each thread? If so, could resource
    > limits be preventing you from creating more threads?


    I don't think Linux creates a new process for each thread. Sure seems
    like this would have to be something related to python since 2.2 could
    create a thousand or more threads. Although I have to admit that it is
    odd that the maximum number of threads I can create on RH9 is different
    than on FC2.
    Kirby Angell, Sep 29, 2004
    #3
  4. Kirby Angell

    Jeremy Jones Guest

    Kirby Angell wrote:

    > Michael Fuhr wrote:
    >
    >> What happens when you hit this "hard limit"? If there's an error
    >> message then it might hint at the problem.

    >
    >
    > If I run this program:
    >
    >

    <snip>

    >> I'm not current on Linux threading models -- do these versions of
    >> Linux create a new process for each thread? If so, could resource
    >> limits be preventing you from creating more threads?

    >
    >
    > I don't think Linux creates a new process for each thread. Sure seems
    > like this would have to be something related to python since 2.2 could
    > create a thousand or more threads. Although I have to admit that it
    > is odd that the maximum number of threads I can create on RH9 is
    > different than on FC2.



    Versions of RedHat prior to 9 did something that made it appear that
    each thread was a process. I don't know all the ins and outs (I think
    it had something to do with user-space vs. kernel-space), but when the
    Native Posix Thread Library (NPTL) appeard in RH9, each thread did not
    show up as its own process (say, in top). I think it's probably an OS
    limitation. I kicked off the same script on Windows XP that I kicked
    off on FC2 and XP was able to generate over 2000 threads as opposed to
    around 400 for FC2. I may be wrong, but I think Python relies on the
    underlying OS threading libraries for everything threading related.
    Which is probably why the recent change in either 2.2 or 2.3 for a dummy
    threading module that simulates threading on systems that do not support
    threading. I know my babbling isn't getting you any closer to a
    solution.....


    Jeremy Jones
    Jeremy Jones, Sep 29, 2004
    #4
  5. Kirby Angell

    Kirby Angell Guest

    Jeremy Jones wrote:
    >
    > around 400 for FC2. I may be wrong, but I think Python relies on the
    > underlying OS threading libraries for everything threading related.
    > Which is probably why the recent change in either 2.2 or 2.3 for a dummy
    > threading module that simulates threading on systems that do not support
    > threading. I know my babbling isn't getting you any closer to a
    > solution.....
    >


    No, your babbling has gotten me much closer. I'm not sure what changed
    between Python 2.2 and Python 2.3, but the problem is the stack size
    allocated to each thread. Apparently with glibc on RH9 something along
    the order of 8 MB of virtual memory is allocated for the stack of each
    thread. 256 threads takes us to the virtual memory limit of the process
    itself. Glibc on FC2 must be using a different default stack size, or
    the virtual memory limit is higher; I've only recently started running
    FC2 on our test servers so I'm not sure which it is.

    The only way to change the stack size is to define THREAD_STACK_SIZE in
    the thread_pthread.h header. There is an ifdef for the Mac that sets
    this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
    and recompiled Python. I can now create 16k threads although I'm not
    sure yet whether our application will run; some tuning is probably in
    order. I'm going to get the 2.2 source code and see what is different
    between the two and maybe get an idea of what a reasonable stack size is.

    There is a very nice write-up on the basic issue here:

    http://www.kegel.com/stackcheck/

    Thanks for your help.
    Kirby Angell, Sep 29, 2004
    #5
  6. On Wed, 29 Sep 2004, Kirby Angell wrote:

    > No, your babbling has gotten me much closer. I'm not sure what changed
    > between Python 2.2 and Python 2.3, but the problem is the stack size
    > allocated to each thread. Apparently with glibc on RH9 something along
    > the order of 8 MB of virtual memory is allocated for the stack of each
    > thread. 256 threads takes us to the virtual memory limit of the process
    > itself. Glibc on FC2 must be using a different default stack size, or
    > the virtual memory limit is higher; I've only recently started running
    > FC2 on our test servers so I'm not sure which it is.
    >
    > The only way to change the stack size is to define THREAD_STACK_SIZE in
    > the thread_pthread.h header. There is an ifdef for the Mac that sets
    > this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
    > and recompiled Python. I can now create 16k threads although I'm not
    > sure yet whether our application will run; some tuning is probably in
    > order. I'm going to get the 2.2 source code and see what is different
    > between the two and maybe get an idea of what a reasonable stack size is.


    ISTR that the default stacksize for the LinuxThreads threading
    implementation (prior to NPTL) was 1MB.

    It is also worth noting that more recent versions of gcc (3.x) are
    generating larger stack frames than older versions; something that has
    affected building the SRE module until it was made non-recursive for
    Python 2.4. This has probably encouraged the bump in the default stack
    size.

    -------------------------------------------------------------------------
    Andrew I MacIntyre "These thoughts are mine alone..."
    E-mail: (pref) | Snail: PO Box 370
    (alt) | Belconnen ACT 2616
    Web: http://www.andymac.org/ | Australia
    Andrew MacIntyre, Sep 30, 2004
    #6
  7. Kirby Angell

    G. S. Hayes Guest

    Jeremy Jones <> wrote in message news:<>...
    > > I don't think Linux creates a new process for each thread. Sure seems
    > > like this would have to be something related to python since 2.2 could

    >
    > Versions of RedHat prior to 9 did something that made it appear that
    > each thread was a process.


    Older versions of ps/top showed all threads instead of just processes.

    You can still see them with the right flags (e.g. the H option to
    ps, "ps auxH").

    > I think it's probably an OS limitation. I kicked off the same script on
    > Windows XP that I kicked off on FC2 and XP was able to generate over
    > 2000 threads as opposed to around 400 for FC2.


    I have no problems creating 2000 threads in FC2 from C. haven't tried
    it in python. Resource limits are the first issue that comes to mind
    for me, followed by kernel versions (esp. if you're running a 2.2 or
    earlier kernel for some reason).
    G. S. Hayes, Sep 30, 2004
    #7
  8. Kirby Angell

    Kirby Angell Guest

    Just to finish off this thread...

    You can set the constant in the header file I mentioned, but you can
    also just do (on Linux):

    ulimit -s <stacksize>

    to set the stack size that will be used. I can control how many threads
    my Python apps can create by setting this to different values. You'll
    probably also want to set the number of files (ulimit -n) to something
    different if you are creating a socket in each of those threads.

    --
    Thank you,

    Kirby Angell
    Get notified anytime your website goes down!
    http://www.alertra.com
    key: 9004F4C0
    fingerprint: DD7E E88D 7F50 2A1E 229D 836A DB5B A751 9004 F4C0
    Kirby Angell, Oct 4, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mayur
    Replies:
    2
    Views:
    993
    Natty Gur
    Jul 2, 2004
  2. mark
    Replies:
    6
    Views:
    439
    Daniel Pitts
    Nov 27, 2006
  3. Paul Sijben

    maximum number of threads

    Paul Sijben, Jan 10, 2007, in forum: Python
    Replies:
    12
    Views:
    810
    Ganesan Rajagopal
    Jan 12, 2007
  4. hg
    Replies:
    4
    Views:
    627
    Jeremy Sanders
    Jan 10, 2007
  5. phanhuyich
    Replies:
    4
    Views:
    245
Loading...

Share This Page