Re: questions regarding stack size use for multi-threaded pythonprograms

Discussion in 'Python' started by Andrew MacIntyre, Nov 14, 2009.

  1. Gabriel Genellina wrote:
    > En Mon, 09 Nov 2009 16:05:31 -0300, Eyal Gordon <>
    > escribió:
    >
    >> background:
    >> we are using python 2.4.3 on CentOS 5.3 with many threads - and our
    >> shell's
    >> default stack size limit is set to 10240KB (i.e. ~10MB).
    >>
    >> we noticed that python's Threading module appears to create threads with
    >> this value as their stack size (we ran a sample program that creates 10
    >> threads and measured its virtual memory size, then reduced the stack size
    >> limit of the shell to 5120KB - and saw that the program's virtual memory
    >> size was reduced by ~50MBs).
    >>
    >> the problem:
    >> our program uses numerous threads, and thus the virtual memory size
    >> gets to
    >> be very large. we would like to reduce the size of the stack to reduce
    >> this
    >> size. we were looking for information about recommendation for the stack
    >> size to use, but found none.

    >
    > You can set the thread stack size (for threads that are going to be
    > created, not existing threads) using threading.stack_size(SIZE)
    > http://docs.python.org/library/threading.html#threading.stack_size


    Sadly for the OP, that capability was introduced in Python 2.5. Prior
    to that, the only way to adjust thread stack size is via a compile time
    option (THREAD_STACK_SIZE). In the absence of that compile time option,
    the platform default is used - on pthread systems (incl Linux) refer to
    the manpage for pthread_attr_setstacksize() for more info, but I believe
    what the OP reports could reasonably be expected for his platform.

    The threading.stack_size() support would not be hard to backport to
    Python 2.4.

    >> questions:
    >> 1. is there some rule-of-thumb for the recommended stack size for python
    >> programs of various sorts?


    There is no rule of thumb because there are too many variables amongst
    the supported platforms. Extensive testing would be required to
    validate any selected size.

    I would suggest starting with 1MB and working down. On a 32bit platform
    I would suggest that 64kb might be adequate for child threads but most
    likely not for the primary thread (the thread that the Python
    interpreter starts in). If your app uses regexes, your stack space
    requirements are likely to be larger.

    FWIW the default thread stack size on Win32 is 1MB which anecdotally
    seems sufficient for a wide variety of applications on that platform.

    {...}

    >> 4. would the size of the stacks (which are probably not really
    >> allocated by
    >> the linux virtual memory sub-system, unless used) have a noticeable
    >> performance effect on a python program? same question regarding the
    >> use of a
    >> large number of threads?

    >
    > I think it doesn't matter, unless you create so many threads as to exhaust
    > the available addressing space (in 32bits, 4GB address space and 10MB per
    > thread means 400 threads maximum).


    The performance of course will degrade once the committed memory in the
    active working set exceeds the available real memory, as the OS will
    start swapping. How the OS treats unused stack space will affect that
    threshold.

    --
    -------------------------------------------------------------------------
    Andrew I MacIntyre "These thoughts are mine alone..."
    E-mail: (pref) | Snail: PO Box 370
    (alt) | Belconnen ACT 2616
    Web: http://www.andymac.org/ | Australia
     
    Andrew MacIntyre, Nov 14, 2009
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ian douglas
    Replies:
    2
    Views:
    987
    Randy Howard
    Jul 30, 2004
  2. Surinder Singh
    Replies:
    1
    Views:
    1,206
    Richard Bos
    Dec 20, 2007
  3. C++Liliput
    Replies:
    3
    Views:
    437
    James Kanze
    Sep 24, 2008
  4. Gabriel Genellina
    Replies:
    0
    Views:
    464
    Gabriel Genellina
    Nov 13, 2009
  5. mk
    Replies:
    1
    Views:
    299
    Jonathan Hartley
    Dec 30, 2009
Loading...

Share This Page