Re: maximum number of threads

Discussion in 'Python' started by hg, Jan 10, 2007.

  1. hg

    hg Guest

    Jean-Paul Calderone wrote:

    > On Wed, 10 Jan 2007 12:11:59 -0200, Felipe Almeida Lessa
    > <> wrote:
    >>On 1/10/07, Laurent Pointal <> wrote:
    >>>This is a system configurable limit (up to a maximum).
    >>>
    >>>See ulimit man pages.
    >>>
    >>>test
    >>>
    >>> ulimit -a
    >>>
    >>>to see what are the current limits, and try with
    >>>
    >>> ulimit -u 2000
    >>>
    >>>to modify the maximum number of user process (AFAIK each thread use a
    >>>process entry on Linux)

    >>
    >>I don't think it's only this.

    >
    > Indeed you are correct. The actual limit you are hitting is the size
    > of your address space. Each thread is allocated 8MB of stack. 382
    > threads consumes about 3GB of address space. Even though most of this
    > memory isn't actually allocated, the address space is still used up. So,
    > when you try to create the 383rd thread, the kernel can't find anyplace
    > to put its stack. So you can't create it.
    >
    > Try reducing your stack size or reducing the number of threads you create.
    > There's really actually almost no good reason to have this many threads,
    > even though it's possible.
    >
    > exarkun@charm:~$ python Desktop/test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > Exception raised: can't start new thread
    >
    > Biggest number of threads: 382
    > exarkun@charm:~$ ulimit -Ss 4096
    > exarkun@charm:~$ python Desktop/test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > 400
    > 450
    > 500
    > 550
    > 600
    > 650
    > 700
    > 750
    > Exception raised: can't start new thread
    >
    > Biggest number of threads: 764
    > exarkun@charm:~$
    >
    > Jean-Paul



    Would increasing the swap size do it also then ?

    hg
    hg, Jan 10, 2007
    #1
    1. Advertising

  2. On Wed, 10 Jan 2007 12:11:59 -0200, Felipe Almeida Lessa <> wrote:
    >On 1/10/07, Laurent Pointal <> wrote:
    >>This is a system configurable limit (up to a maximum).
    >>
    >>See ulimit man pages.
    >>
    >>test
    >>
    >> ulimit -a
    >>
    >>to see what are the current limits, and try with
    >>
    >> ulimit -u 2000
    >>
    >>to modify the maximum number of user process (AFAIK each thread use a
    >>process entry on Linux)

    >
    >I don't think it's only this.


    Indeed you are correct. The actual limit you are hitting is the size
    of your address space. Each thread is allocated 8MB of stack. 382
    threads consumes about 3GB of address space. Even though most of this
    memory isn't actually allocated, the address space is still used up. So,
    when you try to create the 383rd thread, the kernel can't find anyplace
    to put its stack. So you can't create it.

    Try reducing your stack size or reducing the number of threads you create.
    There's really actually almost no good reason to have this many threads,
    even though it's possible.

    exarkun@charm:~$ python Desktop/test.py
    50
    100
    150
    200
    250
    300
    350
    Exception raised: can't start new thread

    Biggest number of threads: 382
    exarkun@charm:~$ ulimit -Ss 4096
    exarkun@charm:~$ python Desktop/test.py
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    550
    600
    650
    700
    750
    Exception raised: can't start new thread

    Biggest number of threads: 764
    exarkun@charm:~$

    Jean-Paul
    Jean-Paul Calderone, Jan 10, 2007
    #2
    1. Advertising

  3. hg

    Paul Sijben Guest

    All thanks for all the input! This was very informative.

    Looks like I indeed need stackless as my code benefits from being
    concurrently designed.

    Paul

    Jean-Paul Calderone wrote:
    > On Wed, 10 Jan 2007 12:11:59 -0200, Felipe Almeida Lessa
    > <> wrote:
    >> On 1/10/07, Laurent Pointal <> wrote:
    >>> This is a system configurable limit (up to a maximum).
    >>>
    >>> See ulimit man pages.
    >>>
    >>> test
    >>>
    >>> ulimit -a
    >>>
    >>> to see what are the current limits, and try with
    >>>
    >>> ulimit -u 2000
    >>>
    >>> to modify the maximum number of user process (AFAIK each thread use a
    >>> process entry on Linux)

    >>
    >> I don't think it's only this.

    >
    > Indeed you are correct. The actual limit you are hitting is the size
    > of your address space. Each thread is allocated 8MB of stack. 382
    > threads consumes about 3GB of address space. Even though most of this
    > memory isn't actually allocated, the address space is still used up. So,
    > when you try to create the 383rd thread, the kernel can't find anyplace
    > to put its stack. So you can't create it.
    >
    > Try reducing your stack size or reducing the number of threads you create.
    > There's really actually almost no good reason to have this many threads,
    > even though it's possible.
    >
    > exarkun@charm:~$ python Desktop/test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > Exception raised: can't start new thread
    > Biggest number of threads: 382
    > exarkun@charm:~$ ulimit -Ss 4096
    > exarkun@charm:~$ python Desktop/test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > 400
    > 450
    > 500
    > 550
    > 600
    > 650
    > 700
    > 750
    > Exception raised: can't start new thread
    > Biggest number of threads: 764
    > exarkun@charm:~$
    > Jean-Paul
    Paul Sijben, Jan 10, 2007
    #3
  4. hg

    Paul Sijben Guest

    All thanks for all the input! This was very informative.

    Looks like I indeed need stackless as my code benefits from being
    concurrently designed.

    Paul

    Jean-Paul Calderone wrote:
    > On Wed, 10 Jan 2007 12:11:59 -0200, Felipe Almeida Lessa
    > <> wrote:
    >> On 1/10/07, Laurent Pointal <> wrote:
    >>> This is a system configurable limit (up to a maximum).
    >>>
    >>> See ulimit man pages.
    >>>
    >>> test
    >>>
    >>> ulimit -a
    >>>
    >>> to see what are the current limits, and try with
    >>>
    >>> ulimit -u 2000
    >>>
    >>> to modify the maximum number of user process (AFAIK each thread use a
    >>> process entry on Linux)

    >>
    >> I don't think it's only this.

    >
    > Indeed you are correct. The actual limit you are hitting is the size
    > of your address space. Each thread is allocated 8MB of stack. 382
    > threads consumes about 3GB of address space. Even though most of this
    > memory isn't actually allocated, the address space is still used up. So,
    > when you try to create the 383rd thread, the kernel can't find anyplace
    > to put its stack. So you can't create it.
    >
    > Try reducing your stack size or reducing the number of threads you create.
    > There's really actually almost no good reason to have this many threads,
    > even though it's possible.
    >
    > exarkun@charm:~$ python Desktop/test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > Exception raised: can't start new thread
    > Biggest number of threads: 382
    > exarkun@charm:~$ ulimit -Ss 4096
    > exarkun@charm:~$ python Desktop/test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > 400
    > 450
    > 500
    > 550
    > 600
    > 650
    > 700
    > 750
    > Exception raised: can't start new thread
    > Biggest number of threads: 764
    > exarkun@charm:~$
    > Jean-Paul
    Paul Sijben, Jan 10, 2007
    #4
  5. Jean-Paul Calderone wrote:

    > Indeed you are correct. The actual limit you are hitting is the size
    > of your address space. Each thread is allocated 8MB of stack. 382
    > threads consumes about 3GB of address space. Even though most of this
    > memory isn't actually allocated, the address space is still used up. So,
    > when you try to create the 383rd thread, the kernel can't find anyplace
    > to put its stack. So you can't create it.


    Interesting. That's why I can get over 3000 on my x86-64 machine... Much
    more address space.

    --
    Jeremy Sanders
    http://www.jeremysanders.net/
    Jeremy Sanders, Jan 10, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mayur
    Replies:
    2
    Views:
    1,006
    Natty Gur
    Jul 2, 2004
  2. Kirby Angell

    Maximum number of threads

    Kirby Angell, Sep 29, 2004, in forum: Python
    Replies:
    7
    Views:
    497
    Kirby Angell
    Oct 4, 2004
  3. mark
    Replies:
    6
    Views:
    440
    Daniel Pitts
    Nov 27, 2006
  4. Paul Sijben

    maximum number of threads

    Paul Sijben, Jan 10, 2007, in forum: Python
    Replies:
    12
    Views:
    814
    Ganesan Rajagopal
    Jan 12, 2007
  5. phanhuyich
    Replies:
    4
    Views:
    258
Loading...

Share This Page