maximum number of threads

P

Paul Sijben

I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).

The server quickly runs out of threads.

I am seeing the following error.

File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__
MercuryObject.__init__(self,mylink)
File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__
self.start()
File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread


Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Paul
 
G

Gabriel Genellina

I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).
The server quickly runs out of threads.

File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread

Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.


--
Gabriel Genellina
Softlab SRL






__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas
 
F

Felipe Almeida Lessa

Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.

Maybe Stackless could help the OP?
http://www.stackless.com/
 
P

Paul Sijben

Gabriel said:
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.

sure but *how* fast they run out is the issue here
Do you really need so many threads?

I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul
 
P

Paul Sijben

Gabriel said:
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.

sure but *how* fast they run out is the issue here
Do you really need so many threads?

I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul
 
L

Laurent Pointal

Paul Sijben a écrit :
sure but *how* fast they run out is the issue here


I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
 
F

Felipe Almeida Lessa

This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)

I don't think it's only this.

---
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
---

Well, unlimited number user processes. But:

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
 
W

William Heymann

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382

So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
 
C

Cecil Westerhof

Felipe said:
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

I tried your script on a PII 300 MHz and only 150 MB. I broke it of when it
reached more as 1,25 million. ;-}
 
H

Hendrik van Rooyen

So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274

This almost looks as if the number of threads is a sixteen bit signed int...

- Hendrik
 
G

Ganesan Rajagopal

I don't think it's only this.

It isn't that at all. The default Linux POSIX threads stack size is
8MB. Linux user space is 3GB (Kernel is mapped at upper 1GB).

382 * 8 = 3056MB.

Basically, you're running out of address space. I don't know if you have any
control at python level. In C you can call pthread_attr_setstacksize().

Ganesan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top