ulimit stack size and python threads

Discussion in 'Python' started by Greg Lindahl, Jan 8, 2009.

  1. Greg Lindahl

    Greg Lindahl Guest

    I figure this is a FAQ, but I can't find it in any FAQs.

    I want to limit the stacksize on my server.

    If I set it to 8 megs, or unlimited, python is happy.

    If I set it to 4 gigabytes, things like yum (which is a python
    program) crash creating a thread. This is on an x86_64 linux kernel,
    RHEL5, etc etc.

    Why is Python overloading the meaning of the ulimit -s like this?
    There are plenty of real non-python programs with huge stack usage,
    and I'd like my system default stack limit to be less than unlimited
    but much larger than Python will allow.

    -- greg
     
    Greg Lindahl, Jan 8, 2009
    #1
    1. Advertising

  2. Greg Lindahl

    MRAB Guest

    Greg Lindahl wrote:
    > I figure this is a FAQ, but I can't find it in any FAQs.
    >
    > I want to limit the stacksize on my server.
    >
    > If I set it to 8 megs, or unlimited, python is happy.
    >
    > If I set it to 4 gigabytes, things like yum (which is a python
    > program) crash creating a thread. This is on an x86_64 linux kernel,
    > RHEL5, etc etc.
    >
    > Why is Python overloading the meaning of the ulimit -s like this?
    > There are plenty of real non-python programs with huge stack usage,
    > and I'd like my system default stack limit to be less than unlimited
    > but much larger than Python will allow.
    >

    I'm only guessing, but could it be a 32-bit limit somewhere? Have you
    tried, say, 1GB, which would be within a 32-bit limit?
     
    MRAB, Jan 8, 2009
    #2
    1. Advertising

  3. Greg Lindahl

    Greg Lindahl Guest

    > I'm only guessing, but could it be a 32-bit limit somewhere? Have you
    > tried, say, 1GB, which would be within a 32-bit limit?


    Indeed, ulimit -s 1000000 (a bit smaller than 1 GB) does work, but it
    doesn't solve my problem, since I want to set the limit higher than 1
    GB.

    -- greg
     
    Greg Lindahl, Jan 8, 2009
    #3
  4. Greg Lindahl

    MRAB Guest

    Greg Lindahl wrote:
    >> I'm only guessing, but could it be a 32-bit limit somewhere? Have you
    >> tried, say, 1GB, which would be within a 32-bit limit?

    >
    > Indeed, ulimit -s 1000000 (a bit smaller than 1 GB) does work, but it
    > doesn't solve my problem, since I want to set the limit higher than 1
    > GB.
    >

    How much higher? You could try just under 4GB (unsigned 32-bit) and just
    under 2GB (signed 32-bit).
     
    MRAB, Jan 8, 2009
    #4
  5. > Why is Python overloading the meaning of the ulimit -s like this?

    Why do you think Python is overloading the meaning of that? I ensure
    you it isn't - it doesn't actively care what the limits are.

    Regards,
    Martin
     
    Martin v. Löwis, Jan 8, 2009
    #5
  6. Greg Lindahl

    Greg Lindahl Guest

    > How much higher? You could try just under 4GB (unsigned 32-bit) and just
    > under 2GB (signed 32-bit).


    I'd like to set it to be about 1/2 the memory size of my server, which
    happens to end up being 4 gbytes. And no, slightly less than 4 gb
    doesn't work.

    But even if that worked, I'd be worried that python is doing something
    bad with the ulimit -s value under the covers.

    -- greg
     
    Greg Lindahl, Jan 8, 2009
    #6
  7. > But even if that worked, I'd be worried that python is doing something
    > bad with the ulimit -s value under the covers.


    Again: it definitely isn't.

    Regards,
    Martin
     
    Martin v. Löwis, Jan 8, 2009
    #7
  8. Greg Lindahl wrote:
    > I figure this is a FAQ, but I can't find it in any FAQs.
    >
    > I want to limit the stacksize on my server.
    >
    > If I set it to 8 megs, or unlimited, python is happy.
    >
    > If I set it to 4 gigabytes, things like yum (which is a python
    > program) crash creating a thread. This is on an x86_64 linux kernel,
    > RHEL5, etc etc.
    >
    > Why is Python overloading the meaning of the ulimit -s like this?
    > There are plenty of real non-python programs with huge stack usage,
    > and I'd like my system default stack limit to be less than unlimited
    > but much larger than Python will allow.


    The Python interpreter itself (absent a call to resource.setrlimit())
    does nothing with resource limits - it just uses the environment it is
    loaded into.

    In the absence of effective alternative solutions, it may be possible to
    achieve the effect you desire by calling resource.setrlimit() in your
    Python installation's site.py, effectively over-riding the system
    default.

    --
    -------------------------------------------------------------------------
    Andrew I MacIntyre "These thoughts are mine alone..."
    E-mail: (pref) | Snail: PO Box 370
    (alt) | Belconnen ACT 2616
    Web: http://www.andymac.org/ | Australia
     
    Andrew MacIntyre, Jan 8, 2009
    #8
  9. Greg Lindahl

    Greg Lindahl Guest

    > Why do you think Python is overloading the meaning of that? I ensure
    > you it isn't - it doesn't actively care what the limits are.


    Always crashing because I asked the OS to please not allow a process
    to grow too big is what I call overloading the meaning of ulimit -s.
    It's quite surprising. Not to mention the poor error message.

    -- greg
     
    Greg Lindahl, Jan 8, 2009
    #9
  10. Greg Lindahl

    Greg Lindahl Guest

    I see. I should be blaming the default behavior of pthreads. I did
    work on a OpenMP library once, and we worked around this problem, plus
    we gave good error messages. Given the number of HPC sites which use
    Python, I'd think that Python would have grown similar features. (HPC
    sites are more likely to have intermediate-sized stack limits due to
    use of Fortran.)

    -- greg
     
    Greg Lindahl, Jan 9, 2009
    #10
  11. > Always crashing because I asked the OS to please not allow a process
    > to grow too big is what I call overloading the meaning of ulimit -s.


    Please trust that there is no explicit code in the Python interpreter
    that tests whether the stack size is 4GB, and then produces an explicit
    crash.

    > It's quite surprising. Not to mention the poor error message.


    AFAICT, you didn't even *report* yet what the error message is,
    so I don't know whether it is poor.

    Regards,
    Martin
     
    Martin v. Löwis, Jan 9, 2009
    #11
  12. > I see. I should be blaming the default behavior of pthreads.

    You shouldn't blame anybody. Instead, you should sit down and study
    the problem in detail, until you fully understand it. Then you should
    start contributing fixes. Never ever should you spread blame.

    Regards,
    Martin
     
    Martin v. Löwis, Jan 9, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Maciej Kalisiak

    "ulimit -s" has no effect?

    Maciej Kalisiak, Feb 5, 2004, in forum: Python
    Replies:
    16
    Views:
    2,223
    Josiah Carlson
    Feb 12, 2004
  2. Maxim Veksler

    ulimit on open sockets ?

    Maxim Veksler, Apr 10, 2007, in forum: Python
    Replies:
    6
    Views:
    549
    Alex Martelli
    Apr 14, 2007
  3. Jarek Zgoda

    Set ulimit when using subprocess.Popen?

    Jarek Zgoda, Jan 28, 2008, in forum: Python
    Replies:
    3
    Views:
    753
    Rob Wolfe
    Jan 28, 2008
  4. Kalaky
    Replies:
    0
    Views:
    278
    Kalaky
    Oct 26, 2004
  5. Sam Roberts
    Replies:
    1
    Views:
    238
    Yukihiro Matsumoto
    Feb 11, 2005
Loading...

Share This Page