ulimit stack size and python threads

G

Greg Lindahl

I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.

-- greg
 
M

MRAB

Greg said:
I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.
I'm only guessing, but could it be a 32-bit limit somewhere? Have you
tried, say, 1GB, which would be within a 32-bit limit?
 
G

Greg Lindahl

I'm only guessing, but could it be a 32-bit limit somewhere? Have you
tried, say, 1GB, which would be within a 32-bit limit?

Indeed, ulimit -s 1000000 (a bit smaller than 1 GB) does work, but it
doesn't solve my problem, since I want to set the limit higher than 1
GB.

-- greg
 
M

MRAB

Greg said:
Indeed, ulimit -s 1000000 (a bit smaller than 1 GB) does work, but it
doesn't solve my problem, since I want to set the limit higher than 1
GB.
How much higher? You could try just under 4GB (unsigned 32-bit) and just
under 2GB (signed 32-bit).
 
M

Martin v. Löwis

Why is Python overloading the meaning of the ulimit -s like this?

Why do you think Python is overloading the meaning of that? I ensure
you it isn't - it doesn't actively care what the limits are.

Regards,
Martin
 
G

Greg Lindahl

How much higher? You could try just under 4GB (unsigned 32-bit) and just
under 2GB (signed 32-bit).

I'd like to set it to be about 1/2 the memory size of my server, which
happens to end up being 4 gbytes. And no, slightly less than 4 gb
doesn't work.

But even if that worked, I'd be worried that python is doing something
bad with the ulimit -s value under the covers.

-- greg
 
M

Martin v. Löwis

But even if that worked, I'd be worried that python is doing something
bad with the ulimit -s value under the covers.

Again: it definitely isn't.

Regards,
Martin
 
A

Andrew MacIntyre

Greg said:
I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.

The Python interpreter itself (absent a call to resource.setrlimit())
does nothing with resource limits - it just uses the environment it is
loaded into.

In the absence of effective alternative solutions, it may be possible to
achieve the effect you desire by calling resource.setrlimit() in your
Python installation's site.py, effectively over-riding the system
default.

--
 
G

Greg Lindahl

Why do you think Python is overloading the meaning of that? I ensure
you it isn't - it doesn't actively care what the limits are.

Always crashing because I asked the OS to please not allow a process
to grow too big is what I call overloading the meaning of ulimit -s.
It's quite surprising. Not to mention the poor error message.

-- greg
 
G

Greg Lindahl

I see. I should be blaming the default behavior of pthreads. I did
work on a OpenMP library once, and we worked around this problem, plus
we gave good error messages. Given the number of HPC sites which use
Python, I'd think that Python would have grown similar features. (HPC
sites are more likely to have intermediate-sized stack limits due to
use of Fortran.)

-- greg
 
M

Martin v. Löwis

Always crashing because I asked the OS to please not allow a process
to grow too big is what I call overloading the meaning of ulimit -s.

Please trust that there is no explicit code in the Python interpreter
that tests whether the stack size is 4GB, and then produces an explicit
crash.
It's quite surprising. Not to mention the poor error message.

AFAICT, you didn't even *report* yet what the error message is,
so I don't know whether it is poor.

Regards,
Martin
 
M

Martin v. Löwis

I see. I should be blaming the default behavior of pthreads.

You shouldn't blame anybody. Instead, you should sit down and study
the problem in detail, until you fully understand it. Then you should
start contributing fixes. Never ever should you spread blame.

Regards,
Martin
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top