maximum number of threads

Discussion in 'Python' started by Paul Sijben, Jan 10, 2007.

  1. Paul Sijben

    Paul Sijben Guest

    I have a server in Python 2.5 that generates a lot of threads. It is
    running on a linux server (Fedora Core 6).

    The server quickly runs out of threads.

    I am seeing the following error.

    File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__
    MercuryObject.__init__(self,mylink)
    File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__
    self.start()
    File "/usr/local/lib/python2.5/threading.py", line 434, in start
    _start_new_thread(self.__bootstrap, ())
    error: can't start new thread


    Does anyone know what it going on here and how I can ensure that I have
    all the threads I need?

    Paul
     
    Paul Sijben, Jan 10, 2007
    #1
    1. Advertising

  2. At Wednesday 10/1/2007 04:38, Paul Sijben wrote:

    >I have a server in Python 2.5 that generates a lot of threads. It is
    >running on a linux server (Fedora Core 6).
    >The server quickly runs out of threads.
    >
    > File "/usr/local/lib/python2.5/threading.py", line 434, in start
    > _start_new_thread(self.__bootstrap, ())
    >error: can't start new thread
    >
    >Does anyone know what it going on here and how I can ensure that I have
    >all the threads I need?


    Simply you can't, as you can't have 10000 open files at once.
    Computer resources are not infinite.
    Do you really need so many threads? Above a certain threshold, the
    program total execution time may increase very quickly.


    --
    Gabriel Genellina
    Softlab SRL






    __________________________________________________
    Preguntá. Respondé. Descubrí.
    Todo lo que querías saber, y lo que ni imaginabas,
    está en Yahoo! Respuestas (Beta).
    ¡Probalo ya!
    http://www.yahoo.com.ar/respuestas
     
    Gabriel Genellina, Jan 10, 2007
    #2
    1. Advertising

  3. On 1/10/07, Gabriel Genellina <> wrote:
    > At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
    > >Does anyone know what it going on here and how I can ensure that I have
    > >all the threads I need?

    >
    > Simply you can't, as you can't have 10000 open files at once.
    > Computer resources are not infinite.
    > Do you really need so many threads? Above a certain threshold, the
    > program total execution time may increase very quickly.


    Maybe Stackless could help the OP?
    http://www.stackless.com/

    --
    Felipe.
     
    Felipe Almeida Lessa, Jan 10, 2007
    #3
  4. Paul Sijben

    Paul Sijben Guest

    Gabriel Genellina wrote:
    >
    > Simply you can't, as you can't have 10000 open files at once. Computer
    > resources are not infinite.


    sure but *how* fast they run out is the issue here

    > Do you really need so many threads?


    I might be able to do with a few less but I still need many.

    I have done a quick test.

    on WinXP I can create 1030 threads
    on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

    now the 301 is rather low I'd say.

    Paul
     
    Paul Sijben, Jan 10, 2007
    #4
  5. Paul Sijben

    Paul Sijben Guest

    Gabriel Genellina wrote:
    >
    > Simply you can't, as you can't have 10000 open files at once. Computer
    > resources are not infinite.


    sure but *how* fast they run out is the issue here

    > Do you really need so many threads?


    I might be able to do with a few less but I still need many.

    I have done a quick test.

    on WinXP I can create 1030 threads
    on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

    now the 301 is rather low I'd say.

    Paul
     
    Paul Sijben, Jan 10, 2007
    #5
  6. Paul Sijben

    Paul Sijben Guest

    Felipe Almeida Lessa wrote:

    > Maybe Stackless could help the OP?
    > http://www.stackless.com/
    >


    thanks I will look into it!
     
    Paul Sijben, Jan 10, 2007
    #6
  7. Paul Sijben

    Paul Sijben Guest

    Felipe Almeida Lessa wrote:

    > Maybe Stackless could help the OP?
    > http://www.stackless.com/
    >


    thanks I will look into it!
     
    Paul Sijben, Jan 10, 2007
    #7
  8. Paul Sijben a écrit :
    > Gabriel Genellina wrote:
    >> Simply you can't, as you can't have 10000 open files at once. Computer
    >> resources are not infinite.

    >
    > sure but *how* fast they run out is the issue here
    >
    >> Do you really need so many threads?

    >
    > I might be able to do with a few less but I still need many.
    >
    > I have done a quick test.
    >
    > on WinXP I can create 1030 threads
    > on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)
    >
    > now the 301 is rather low I'd say.


    This is a system configurable limit (up to a maximum).

    See ulimit man pages.

    test

    ulimit -a

    to see what are the current limits, and try with

    ulimit -u 2000

    to modify the maximum number of user process (AFAIK each thread use a
    process entry on Linux)

    >
    > Paul
    >
     
    Laurent Pointal, Jan 10, 2007
    #8
  9. On 1/10/07, Laurent Pointal <> wrote:
    > This is a system configurable limit (up to a maximum).
    >
    > See ulimit man pages.
    >
    > test
    >
    > ulimit -a
    >
    > to see what are the current limits, and try with
    >
    > ulimit -u 2000
    >
    > to modify the maximum number of user process (AFAIK each thread use a
    > process entry on Linux)


    I don't think it's only this.

    ---
    $ ulimit -a
    core file size (blocks, -c) 0
    data seg size (kbytes, -d) unlimited
    max nice (-e) 20
    file size (blocks, -f) unlimited
    pending signals (-i) unlimited
    max locked memory (kbytes, -l) unlimited
    max memory size (kbytes, -m) unlimited
    open files (-n) 1024
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) unlimited
    max rt priority (-r) unlimited
    stack size (kbytes, -s) 8192
    cpu time (seconds, -t) unlimited
    max user processes (-u) unlimited
    virtual memory (kbytes, -v) unlimited
    file locks (-x) unlimited
    ---

    Well, unlimited number user processes. But:

    ---
    $ python test.py
    50
    100
    150
    200
    250
    300
    350
    Exception raised: can't start new thread

    Biggest number of threads: 382
    ---

    The test.py script is attached.

    --
    Felipe.
     
    Felipe Almeida Lessa, Jan 10, 2007
    #9
  10. On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
    > ---
    > $ python test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > Exception raised: can't start new thread
    >
    > Biggest number of threads: 382
    > ---
    >
    > The test.py script is attached.


    So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
    system with 8G of ram and I got

    <lots of output>
    Exception raised: can't start new thread
    Biggest number of threads: 32274
     
    William Heymann, Jan 10, 2007
    #10
  11. Felipe Almeida Lessa wrote:

    > $ python test.py
    > 50
    > 100
    > 150
    > 200
    > 250
    > 300
    > 350
    > Exception raised: can't start new thread


    I tried your script on a PII 300 MHz and only 150 MB. I broke it of when it
    reached more as 1,25 million. ;-}
     
    Cecil Westerhof, Jan 11, 2007
    #11
  12. "William Heymann" <> wrote:


    > So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
    > system with 8G of ram and I got
    >
    > <lots of output>
    > Exception raised: can't start new thread
    > Biggest number of threads: 32274
    >


    This almost looks as if the number of threads is a sixteen bit signed int...

    - Hendrik
     
    Hendrik van Rooyen, Jan 12, 2007
    #12
  13. >>>>> Felipe Almeida Lessa <> writes:
    >>
    >> to modify the maximum number of user process (AFAIK each thread use a
    >> process entry on Linux)


    > I don't think it's only this.


    It isn't that at all. The default Linux POSIX threads stack size is
    8MB. Linux user space is 3GB (Kernel is mapped at upper 1GB).

    382 * 8 = 3056MB.

    Basically, you're running out of address space. I don't know if you have any
    control at python level. In C you can call pthread_attr_setstacksize().

    Ganesan

    --
    Ganesan Rajagopal
     
    Ganesan Rajagopal, Jan 12, 2007
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mayur
    Replies:
    2
    Views:
    1,032
    Natty Gur
    Jul 2, 2004
  2. Kirby Angell

    Maximum number of threads

    Kirby Angell, Sep 29, 2004, in forum: Python
    Replies:
    7
    Views:
    515
    Kirby Angell
    Oct 4, 2004
  3. mark
    Replies:
    6
    Views:
    455
    Daniel Pitts
    Nov 27, 2006
  4. hg
    Replies:
    4
    Views:
    645
    Jeremy Sanders
    Jan 10, 2007
  5. phanhuyich
    Replies:
    4
    Views:
    291
Loading...

Share This Page