PEP: Adding Priority Scheduling feature to the subprocess

Discussion in 'Python' started by darklord@timehorse.com, Feb 20, 2008.

  1. Guest

    I am working on a PEP and would appreciate comment. The proposal is
    available at

    http://python.timehorse.com/PEP_-_Application_Priority.reST

    and is repeated below:

    ---------

    :pEP: <Unassigned>
    :Title: Adding Priority Scheduling feature to the subprocess module
    :Version: $Rev: 93 $
    :Last Modified: $Date: 2008-02-20 08:37:00 -0500 (Wed, 20 Feb 2008) $
    :Author: Jeffrey C. Jacobs
    :Discussions-To: python at timehorse dot com
    :Status: Draft
    :Type: Standards Track
    :Content-Type: text/x-rst
    :Created: 19-Feb-2008
    :python-Version: 2.5.1
    :post-History: <None>

    ----------

    Introduction
    ~~~~~~~~~~~~

    The ``subprocess`` module is intended to create a platform-independent
    method of *spawning* or *popening* or *execing* an OS-level sub-
    process. This module provides the ability to spawn processes on a
    variety of platforms by creating an instance of the
    ``subprocess.Popen`` class.

    Currently, only the Windows version of the ``Popen.__init__`` method
    can set the OS priority of a spawned process directly using the
    ``createFlags`` parameter. Clients operating under UNIX must use some
    version of the ``preexec_fn`` to either wrap a C-call to the
    ``setpriority`` POSIX function or do so by prepending ``nice <u>`` to
    the ``args`` or in the ``executable``, which would only work for sub-
    processes spawned from a shell. Under a Windows environment, however,
    the ``preexec_fn`` parameter is ignored and there is no ``nice`` shell
    program. This thus precludes the easy setting of a spawned process's
    priority in the UNIX environment, as well as creates a disconnect in
    terms of the way priorities are set between Windows and UNIX
    platforms, which somewhat defeats the purpose of ``subprocess.py`` in
    terms of being relatively platform-independent.

    In addition to these issues, because Windows processes are referenced
    by handles, not by process IDs, although this would divide the UNIX-
    specific and Windows specific code, the Handle for the windows would
    none the less be useful in the ``win32process`` module and therefore
    it is proposed that this handle be officially exposed from the
    ``Popen`` instance via the name ``handle``. Currently, this property
    is unofficially accessible via the name ``_handle``, so this proposes
    to rename that value and make the name official for python running
    under Windows. The value of ``handle`` for other platforms would then
    be ``None``. In addition to officially exposing the Handle to
    Process, it is proposed that the Main Thread Handle, also internally
    exposed within the ``Popen.__init__`` process, be externally exposed
    via the name ``hthread`` for use by ``win32process`` and the priority
    methods.

    Because UNIX uses priorities between +20 and -20 and Windows, via
    Process and Thread priorities, allows settings between 0 and 31, a
    uniform setting for each system should be derived. This would be
    accomplished by giving process priority in terms of a floating-point
    value between 0.0 and 1.0 for lowest and highest possible priority,
    respectively. This priority would be added to the ``Popen``
    constructor as an optional parameter, with a default value of
    ``None``. If and only if something other than ``None`` is set, that
    value would override any setting in ``createFlags``, and the Process
    Priority flags of ``createFlags`` would be masked off under the
    Windows environment. In all cases a non-``None`` setting would cause
    the spawned process to run at the indicated priority converted to the
    platform-specific units.

    The Macintosh OS X / BSD subsystem should be compatible with the UNIX-
    specific sections of this proposal.

    Specifics
    ~~~~~~~~~

    The following steps would accomplish this proposal:

    1) Add a C-Python wrapper for the POSIX ``getpriority`` /
    ``setpriority`` methods to the ``os`` module. These methods would
    only be available in a UNIX environment and would mesh well with the
    existing file/process functions specified therein. The
    ``win32process`` module already exposes the corresponding Windows
    methods.

    2) Officially expose the Process Handle as ``handle`` and the
    Process's Main Thread Handle as ``hThread`` within the ``Popen``
    instance interface.

    3) Add ``setPriority`` / ``getPriority`` methods to the ``Popen``
    class which will use the platform-specific priority function either
    from ``os`` for UNIX or ``win32process`` for Windows. The input
    priority to ``setPriority`` will be checked and raise a ``ValueError``
    exception for values not in the range 0.0 to 1.0.

    The 0.0 to 1.0 scale will be converted to the 32-point Windows
    scale by linear mapping, e.g. for a floating-point priority *p*, the
    Windows priority is given by:

    ``int(p*31 + .5)``

    Windows is limited to a non-linear, 7-point scale for process
    priorities, with thread priorities making up the difference relative
    to the 32-point internal priority scale. Thus, the Windows versions
    of these functions will use a combination of ``SetPriorityClass`` and
    ``SetThreadPriority`` in order to set the requested process priority,
    using the ``handle`` and ``hThread`` properties of the ``Popen``
    instance.

    The 0.0 to 1.0 scale will be converted to the 41-point UNIX scale
    (values between +20 and -20, inclusive) via an inverted linear
    mapping, e.g. for a floating-point priority *p*, the UNIX priority is
    given by:

    ``20 - int(p*40 + .5)``

    Under the UNIX platform, only the super-user (root) is allowed to
    set priorities below 0 (*p* > 0.5). This condition will not be
    checked in the ``setPriority`` code; it is the responsibility of the
    caller to ensure the script making the call to ``setpriority`` has
    sufficient user privileges, i.e. via the ``os.setuid`` function.

    The conversion from the platform-specific Priority scale to the
    floating-point scale are used by ``getPriority`` and are as follows
    for a given UNIX priority *u* and Windows priority *w*:

    ``(20 - u) / 40.0``

    ``w / 31.0``

    As with the ``setPriority`` method for windows, the ``getPriority``
    method shall use a combination of ``GetPriorityClass`` /
    ``GetThreadPriority`` to determine the true Windows priority based on
    the 32-point scale before converting to floating point. The specific
    numeric conversion for the Windows ``getPriority`` method is given in
    `Table 1`_.

    See the `Appendix`_ for details about the 32-point Windows scale.
    Note that some priorities have equivalent settings. In this case, raw
    Process Priority Class (without regard for Foreground and Background
    Process settings) will have preferential setting and Thread Priority
    Levels shall be used to make up the difference. E.g. priorities from
    1 to 4 will have ``IDLE_PRIORITY_CLASS``, 5 to 7 will have
    ``BELOW_NORMAL_PRIORITY_CLASS``, 8 and 9 will have
    ``NORMAL_PRIORITY_CLASS``, 10 to 12 will have
    ``ABOVE_NORMAL_PRIORITY_CLASS``, 13 to 15 will have
    ``HIGH_PRIORITY_CLASS`` and 16 to 31 will be accessed through
    ``REALTIME_PRIORITY_CLASS``, potentially with a non-standard Thread
    Priority level. Also note that Priority 0 is reserved for the Zero-
    Page Thread, which is responsible for zeroizing memory when no other
    process is running. It is impossible for a user process to have this
    priority. Therefore, on Windows, any priority set request below about
    0.016 will be treated as if it was requesting Windows Priority Level
    1.

    4) Add a priority parameter to the ``Popen.__init__`` method that
    shall be the final parameter of the function and default to ``None``
    in order to preserve backwards compatibility. When priority is not
    ``None``, it shall be used in order to set the priority of the spawned
    process either at the time of the creation, or soon after. The
    setting of priority may be accomplished via a ``nice <u>`` prefix to
    shell commands (``shell = True``) or via a call to ``setpriority``
    after the process is spawned. When a priority is set via the Windows
    version, a mask is applied to the ``createFlags`` parameter to remove
    any priority related bits. The calculated priority is then used to
    determine a new priority in the 7-point Windows Process scale and set
    the corresponding flag in the ``createFlags`` parameter. Then
    ``SetThreadPriority`` shall be called on the returned Main Thread
    Handle with the remaining priority points to further adjust the
    priority toward the 32-point scale.

    Opened Questions
    ~~~~~~~~~~~~~~~~

    1) Should ``Popen.__init__`` for UNIX check for shell-based spawns and
    automatically apply ``nice`` or should it always first spawn the
    process and then set its priority via ``setpriority`` once created.
    Note, the later technique would technically not be available using
    ``preexec_fn``.

    2) The ``startinfo`` parameter may also need adjusting to set whether
    the process is spawned as a background or foreground process (See
    `Table 1`_).

    3) Should windows use the Priority Boost APIs:
    ``SetProcessPriorityBoost``, ``GetProcessPriorityBoost``,
    ``SetThreadPriorityBoost`` and ``GetThreatPriorityBoost``?

    4) How does one determine if a given Windows process with handle
    ``handle`` is running as a *Background* or a *Foreground* process at
    any given time, for use in the Windows version of ``getPriority``?

    5) With the current version of ``subprocess`` I have been unable to
    spawn a process with a higher priority than ``NORMAL_PRIORITY_CLASS``
    under Windows; python is always running as ``NORMAL_PRIORITY_CLASS``.

    Appendix
    ~~~~~~~~

    ... _Table 1:

    +----+----------------------------------
    +-------------------------------+
    | Pt | Process Priority Class | Thread Priority
    Level |
    +====+==================================
    +===============================+
    | 1 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_IDLE |
    +----+----------------------------------
    +-------------------------------+
    | 1 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_IDLE |
    +----+----------------------------------
    +-------------------------------+
    | 1 | NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_IDLE |
    +----+----------------------------------
    +-------------------------------+
    | 1 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_IDLE |
    +----+----------------------------------
    +-------------------------------+
    | 1 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_IDLE |
    +----+----------------------------------
    +-------------------------------+
    | 2 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 3 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 4 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 4 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 5 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 5 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 5 | Background NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 6 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 6 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 6 | Background NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 7 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 7 | Background NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 7 | Foreground NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 8 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 8 | NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 8 | Foreground NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 8 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 9 | NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 9 | Foreground NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 9 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 10 | Foreground NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 10 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 11 | Foreground NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 11 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 11 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 12 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 12 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 13 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 14 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 15 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 15 | HIGH_PRIORITY_CLASS |
    THREAD_PRIORITY_TIME_CRITICAL |
    +----+----------------------------------
    +-------------------------------+
    | 15 | IDLE_PRIORITY_CLASS |
    THREAD_PRIORITY_TIME_CRITICAL |
    +----+----------------------------------
    +-------------------------------+
    | 15 | BELOW_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_TIME_CRITICAL |
    +----+----------------------------------
    +-------------------------------+
    | 15 | NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_TIME_CRITICAL |
    +----+----------------------------------
    +-------------------------------+
    | 15 | ABOVE_NORMAL_PRIORITY_CLASS |
    THREAD_PRIORITY_TIME_CRITICAL |
    +----+----------------------------------
    +-------------------------------+
    | 16 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_IDLE |
    +----+----------------------------------
    +-------------------------------+
    | 17 | REALTIME_PRIORITY_CLASS | -7 |
    +----+----------------------------------
    +-------------------------------+
    | 18 | REALTIME_PRIORITY_CLASS | -6 |
    +----+----------------------------------
    +-------------------------------+
    | 19 | REALTIME_PRIORITY_CLASS | -5 |
    +----+----------------------------------
    +-------------------------------+
    | 20 | REALTIME_PRIORITY_CLASS | -4 |
    +----+----------------------------------
    +-------------------------------+
    | 21 | REALTIME_PRIORITY_CLASS | -3 |
    +----+----------------------------------
    +-------------------------------+
    | 22 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST |
    +----+----------------------------------
    +-------------------------------+
    | 23 | REALTIME_PRIORITY_CLASS |
    THREAD_PRIORITY_BELOW_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 24 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 25 | REALTIME_PRIORITY_CLASS |
    THREAD_PRIORITY_ABOVE_NORMAL |
    +----+----------------------------------
    +-------------------------------+
    | 26 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST |
    +----+----------------------------------
    +-------------------------------+
    | 27 | REALTIME_PRIORITY_CLASS | 3 |
    +----+----------------------------------
    +-------------------------------+
    | 28 | REALTIME_PRIORITY_CLASS | 4 |
    +----+----------------------------------
    +-------------------------------+
    | 29 | REALTIME_PRIORITY_CLASS | 5 |
    +----+----------------------------------
    +-------------------------------+
    | 30 | REALTIME_PRIORITY_CLASS | 6 |
    +----+----------------------------------
    +-------------------------------+
    | 31 | REALTIME_PRIORITY_CLASS |
    THREAD_PRIORITY_TIME_CRITICAL |
    +----+----------------------------------
    +-------------------------------+

    Table 1: 32-Point Windows Priority Scale [#MSPriorityTable]_.

    -------

    ... [#MSPriorityTable] From `Microsoft Windows Scheduling Priorities
    <http://msdn2.microsoft.com/en-us/library/ms685100.aspx>`_.
    , Feb 20, 2008
    #1
    1. Advertising

  2. Terry Reedy Guest

    Re: Adding Priority Scheduling feature to the subprocess

    | Because UNIX uses priorities between +20 and -20 and Windows, via
    | Process and Thread priorities, allows settings between 0 and 31, a
    | uniform setting for each system should be derived. This would be
    | accomplished by giving process priority in terms of a floating-point
    | value between 0.0 and 1.0 for lowest and highest possible priority,
    | respectively.

    I would rather that the feature use the -20 to 20 priorities and map that
    appropriately to the Windows range on Windows.
    Terry Reedy, Feb 21, 2008
    #2
    1. Advertising

  3. TimeHorse Guest

    Re: Adding Priority Scheduling feature to the subprocess

    On Feb 20, 10:15 pm, "Terry Reedy" <> wrote:
    > | Because UNIX uses priorities between +20 and -20 and Windows, via
    > | Process and Thread priorities, allows settings between 0 and 31, a
    > | uniform setting for each system should be derived.  This would be
    > | accomplished by giving process priority in terms of a floating-point
    > | value between 0.0 and 1.0 for lowest and highest possible priority,
    > | respectively.
    >
    > I would rather that the feature use the -20 to 20 priorities and map that
    > appropriately to the Windows range on Windows.


    The problem as I see it is that -20 to +20 is only just over 5 bits of
    precision and I can easily imagine an OS with many more than just 5
    bits to specify a process priority. Of course, the os.getpriority and
    os.setpriority, being specific to UNIX, WOULD use the -20 to +20
    scale, it's just the generic subprocess that would not. But for a
    generic priority, I like floating point because it gives 52 bits of
    precision on most platforms. This would allow for the most
    flexibility. Also, 0.0 to 1.0 is in some ways more intuitive to new
    programmers because it can be modeled as ~0% CPU usage vs. ~100% CPU
    usage, theoretically. Users not familiar with UNIX might OTHO be
    confused by the idea that a lower priority number constitutes a
    "higher priority".

    Of course, the scale used for p in Popen(...).setPriority(p) is really
    not an important issue to me as long as it makes sense in the context
    of priorities. Given that os.setpriority and Popen(...).setPriority
    have virtually the same name, it would probably be better to rename
    the later to something a bit less prone to confusion. Alternatively,
    it would not be unreasonable to design setPriority (and getPriority
    correspondingly) such that under UNIX it takes 1 parameter, -20 to +20
    and under Windows it takes 2 parameters, second one optional, where
    the Windows API priorities are directly passed to it (for getPriority,
    Windows would return a Tuple pair corresponding to Priority Class and
    Main Thread Priority). However, I personally prefer a unified
    definition for subprocess.py's Priority since there already is or will
    be direct os-level methods to accomplish the same thing in the os-
    native scale.

    Anyway, thanks for the input and I will make a note of it in the PEP.
    Other than the generic Property ranges, do you see any other issues
    with my proposal?

    Jeffrey.
    TimeHorse, Feb 21, 2008
    #3
  4. Re: Adding Priority Scheduling feature to the subprocess

    On Thu, 21 Feb 2008 04:54:01 -0800 (PST), TimeHorse
    <> declaimed the following in comp.lang.python:

    > The problem as I see it is that -20 to +20 is only just over 5 bits of
    > precision and I can easily imagine an OS with many more than just 5
    > bits to specify a process priority. Of course, the os.getpriority and


    Why imagine... AmigaOS ran -128..+127 (though in practice, one never
    went above +20 as the most time critical system processes ran at that
    level; User programs ran at 0, the Workbench [desktop] ran at +1... I
    think file system ran +5 and disk device handlers ran +10)

    On the other side, VMS ran 0..31, with 16..31 being fixed realtime
    priorities, and 0..15 being variable (they don't drop below the process
    base, but can and are boosted higher the longer they are preempted)
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
    Dennis Lee Bieber, Feb 21, 2008
    #4
  5. TimeHorse Guest

    Re: Adding Priority Scheduling feature to the subprocess

    On Feb 21, 1:17 pm, Dennis Lee Bieber <> wrote:
    >         Why imagine... AmigaOS ran -128..+127 (though in practice, one never
    > went above +20 as the most time critical system processes ran at that
    > level; User programs ran at 0, the Workbench [desktop] ran at +1... I
    > think file system ran +5 and disk device handlers ran +10)
    >
    >         On the other side, VMS ran 0..31, with 16..31 being fixed realtime
    > priorities, and 0..15 being variable (they don't drop below the process
    > base, but can and are boosted higher the longer they are preempted)


    Thanks for the info! Actually, many peices of the WinNT kernel were
    based on VMS, interestingly enough (go back to the NT 3.5 days and you
    can see this) and of course Windows 2000, XP and Vista are all derived
    from NT. So I guess one long-time holdover from the VMS days of NT is
    just as you say, it's still true in Windows: processes run 0..15 for
    anything but realtime, even if the thread is boosted to Time Critical,
    but in realtime the priorities go from 16..31.

    Anyway, on the one hand AmigaOS support where -128 -> p = 0.0 and +127
    -> p = 1.0 would be a good example of why simply using a 41 point UNIX
    scale is defecient in representing all possible priorities, but apart
    from the support AmigaOS argument, you bring up another issue which
    may be dangerous: are priorities linear in nature? For instance, a
    with-Focus Windows App running Normal-Normal runs at priority 9, p ~=
    0.29 (as well as likely for VMS), where as UNIX and Amiga have 0 for
    normal processes, p ~= 0.50. In many ways, the "Normal" mode being
    the epoch makes sense, but this clearly cannot be done on a linear
    scale. Perhaps I should modify the PEP to instead having the generic
    priorities from 0.0 to 1.0, to have them go from -1.0 to +1.0, with
    0.0 considered normal priority. Then, the negative and positive
    regions can be considered linearly but not necessairily with the same
    spacing since on Windows -1.0 to 0.0 spans 0 to 9 priorities and 0.0
    to +1.0 spans 10 to 31 priorities. And then, since +20 is the highest
    AmigaOS priority in practice, yet the scale goes up to +127, the would
    mean that from p ~= +0.16 to p ~= -1.0 you get obscenely high
    priorities which do not seem practical.

    I will need to think about this problem so more. I'd hate to think
    there might be a priority scale based on a Normal Distribution! I'm
    already going bald; I can't afford to loose any more hair! :)

    Anyway, thanks for info Dennis; you've given me quite a bit to think
    about!

    Jeffrey.
    TimeHorse, Feb 21, 2008
    #5
  6. Re: Adding Priority Scheduling feature to the subprocess

    TimeHorse <> wrote:
    > Anyway, on the one hand AmigaOS support where -128 -> p = 0.0 and +127
    > -> p = 1.0 would be a good example of why simply using a 41 point UNIX
    > scale is defecient in representing all possible priorities, but apart
    > from the support AmigaOS argument, you bring up another issue which
    > may be dangerous: are priorities linear in nature?


    Interestingly enough this was changed in recent linux kernels.
    Process levels in linus kernels are logarithmic now, whereas before
    they weren't (but I wouldn't like to say exactly what!).

    If I run two CPU intensive jobs to busy out both my CPUs, then using
    kernel 2.6.22, run

    $ python -c 'from time import time
    t = time()
    for i in xrange(10000000):
    pass
    print time()-t'
    1.36508607864

    $ nice -n 10 python -c 'from time import time
    t = time()
    for i in xrange(10000000):
    pass
    print time()-t'
    4.27783703804

    $ nice -n 20 python -c 'from time import time
    t = time()
    for i in xrange(10000000):
    pass
    print time()-t'
    36.9293899536

    You can see that the levels are not linear! A nice 10 job gets 32% of
    the CPU of a nice 0 job, but a nice 20 job gets only 4%.

    I think you are on to a loser here trying to normalise it across
    OSes unfortunately :-(

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
    Nick Craig-Wood, Feb 22, 2008
    #6
  7. TimeHorse Guest

    Re: Adding Priority Scheduling feature to the subprocess

    On Feb 22, 4:30 am, Nick Craig-Wood <> wrote:
    > Interestingly enough this was changed in recent linux kernels.
    > Process levels in linus kernels are logarithmic now, whereas before
    > they weren't (but I wouldn't like to say exactly what!).


    Wow! That's a VERY good point. I ran a similar test on Windows with
    the 'start' command which is similar to nice but you need to specify
    the Priority Class by name, e.g.

    start /REALTIME python.exe bench1.py

    Now, in a standard operating system you'd expect some variance between
    runs, and I did find that. So I wrote a script to compute the Mode
    (but not the Standard Deviation as I didn't have time for it) for each
    Priority Class, chosen each run at random, accumulated the running
    value for each one. Now, when I read the results, I really wish I'd
    computed the Chi**2 to calculate the Standard Deviation because the
    results all appeared within very close relation to one another, as if
    the Priority Class had overall very little effect. In fact, I would
    be willing to guess that say NORMAL and ABOVENORMAL lie with one
    Standard Deviation of one another!

    That having been said, the tests all ran in about 10 seconds so it may
    be that the process was too simple to show any statistical results. I
    know for instance that running ffmpeg as NORMAL or REALTIME makes a
    sizable difference.

    So, I concede the "Unified Priority" may indeed be dead in the water,
    but I am thinking of giving it once last go with the following
    suggestion:

    0.0 == Zero-Page (Windows, e.g. 0) / +20 (Unix)
    1.0 == Normal (Foreground) Priority (Windows, e.g. 9) / 0 (Unix)
    MAX_PRIORITY == Realtime / Time Critical (Windows, e.g. 31) / -20
    (Unix)

    With the value of MAX_PRIORITY TBD. Now, 0.0 would still represent
    (relatively) 0% CPU usage, but now 1.0 would represent 100% of
    'Normal' priority. I would still map 0.0 - 1.0 linearly over the
    scale corresponding to the given operating system (0 - 9, Window; +20
    - 0, Unix), but higher priorities would correspond to > 1.0 values.

    The idea here is that most user will only want to lower priority, not
    raise it, so it makes lowering pretty intuitive. As for the linear
    mapping, I would leave a note in the documentation that although the
    scale is "linear", the operating system may choose to behave as if the
    scale is linear and that the user should consult the documentation for
    their OS to determine specific behavior. This is similar to the
    documentation of the file time-stamps in os.stat, since their
    granularity differs based on OS. Most users, I should think, would
    just want to make their spawn "slower" and use the scale do determine
    "how much" in a relative fashion rather than expecting hard-and-fast
    numbers for the actually process retardation.

    Higher than Normal priorities may OTHO, be a bit harder to deal with.
    It strikes me that maybe the best approach is to make MAX_PRIORITY
    operating system dependent, specifically 31 - 9 + 1.0 = +23.0 for
    Windows and -20 - 0 + 1.0 = +21.0 for Unix. This way, again the
    priorities map linearly and in this case 1:1. I think for most users,
    they would choose a "High Priority" relative to MAX_PRIORITY or just
    choose a small increment about 1.0 to add just a little boost.

    Of course, the 2 biggest problems with this approach are, IMHO, a) the
    < Normal scale is percent but the > Normal scale is additive.
    However, there is no "Simple" definition of MAX_PRIORITY, so I think
    using the OS's definition is natural. b) This use of the priority
    scale may be confusing to Unix users, since 1.0 now represents
    "Normal" and +21, not +/-20 represents Max Priority. However, the
    definition of MAX_PRIORITY would be irrelevant to the definition of
    setPriority and getPriority, since each would, in my proposal, compute
    for p > 1.0:

    Windows: 9 + int((p - 1) / (MAX_PRIORITY - 1) * 22 + .5)
    Unix: -int((p - 1) / (MAX_PRIORITY - 1) * 20 + .5)

    Anyway, that's how I'd propose to do the nitty-gritty. But, more than
    anything, I think the subprocess 'priority' methods should use a
    priority scheme that is easy to explain. And by that, I propose:

    1.0 represents normal priority, 100%. Any priority less than 1
    represents a below normal priority, down to 0.0, the lowest possible
    priority or 0%. Any priority above 1.0 represents an above normal
    priority, with MAX_PRIORITY being the highest priority level available
    for a given os.

    Granted, it's not much simpler than 0 is normal, -20 is highest and
    +20 is lowest, except in so far as it being non-intuitive to consider
    a lower priority number representing a higher priority. Certainly, we
    could conform all systems to the +20.0 to -20.0 floating point system,
    but I prefer not to bias the methods and honestly feel percentage is
    more intuitive.

    So, how does that sound to people? Is that more palatable?

    Thanks again for all the input!

    Jeffrey.
    TimeHorse, Feb 25, 2008
    #7
  8. Re: Adding Priority Scheduling feature to the subprocess

    TimeHorse <> wrote:
    > On Feb 22, 4:30 am, Nick Craig-Wood <> wrote:
    > > Interestingly enough this was changed in recent linux kernels.
    > > Process levels in linus kernels are logarithmic now, whereas before
    > > they weren't (but I wouldn't like to say exactly what!).

    >
    > Wow! That's a VERY good point. I ran a similar test on Windows with
    > the 'start' command which is similar to nice but you need to specify
    > the Priority Class by name, e.g.
    >
    > start /REALTIME python.exe bench1.py
    >
    > Now, in a standard operating system you'd expect some variance between
    > runs, and I did find that. So I wrote a script to compute the Mode
    > (but not the Standard Deviation as I didn't have time for it) for each
    > Priority Class, chosen each run at random, accumulated the running
    > value for each one. Now, when I read the results, I really wish I'd
    > computed the Chi**2 to calculate the Standard Deviation because the
    > results all appeared within very close relation to one another, as if
    > the Priority Class had overall very little effect. In fact, I would
    > be willing to guess that say NORMAL and ABOVENORMAL lie with one
    > Standard Deviation of one another!
    >
    > That having been said, the tests all ran in about 10 seconds so it may
    > be that the process was too simple to show any statistical results. I
    > know for instance that running ffmpeg as NORMAL or REALTIME makes a
    > sizable difference.


    You need to run N x tasks at normal priority which just use up CPU, eg

    python -c "while 1: pass"

    N needs to be the number of CPUs that you have.

    If you don't do that then your time test prog will just run at the
    full CPU speed and not test the scheduling at all!

    > So, I concede the "Unified Priority" may indeed be dead in the water,
    > but I am thinking of giving it once last go with the following
    > suggestion:
    >
    > 0.0 == Zero-Page (Windows, e.g. 0) / +20 (Unix)
    > 1.0 == Normal (Foreground) Priority (Windows, e.g. 9) / 0 (Unix)
    > MAX_PRIORITY == Realtime / Time Critical (Windows, e.g. 31) / -20
    > (Unix)
    >
    > With the value of MAX_PRIORITY TBD. Now, 0.0 would still represent
    > (relatively) 0% CPU usage, but now 1.0 would represent 100% of
    > 'Normal' priority. I would still map 0.0 - 1.0 linearly over the
    > scale corresponding to the given operating system (0 - 9, Window; +20
    > - 0, Unix), but higher priorities would correspond to > 1.0 values.
    >
    > The idea here is that most user will only want to lower priority, not
    > raise it, so it makes lowering pretty intuitive. As for the linear
    > mapping, I would leave a note in the documentation that although the
    > scale is "linear", the operating system may choose to behave as if the
    > scale is linear and that the user should consult the documentation for
    > their OS to determine specific behavior. This is similar to the
    > documentation of the file time-stamps in os.stat, since their
    > granularity differs based on OS. Most users, I should think, would
    > just want to make their spawn "slower" and use the scale do determine
    > "how much" in a relative fashion rather than expecting hard-and-fast
    > numbers for the actually process retardation.
    >
    > Higher than Normal priorities may OTHO, be a bit harder to deal with.
    > It strikes me that maybe the best approach is to make MAX_PRIORITY
    > operating system dependent, specifically 31 - 9 + 1.0 = +23.0 for
    > Windows and -20 - 0 + 1.0 = +21.0 for Unix. This way, again the
    > priorities map linearly and in this case 1:1. I think for most users,
    > they would choose a "High Priority" relative to MAX_PRIORITY or just
    > choose a small increment about 1.0 to add just a little boost.
    >
    > Of course, the 2 biggest problems with this approach are, IMHO, a) the
    > < Normal scale is percent but the > Normal scale is additive.
    > However, there is no "Simple" definition of MAX_PRIORITY, so I think
    > using the OS's definition is natural. b) This use of the priority
    > scale may be confusing to Unix users, since 1.0 now represents
    > "Normal" and +21, not +/-20 represents Max Priority. However, the
    > definition of MAX_PRIORITY would be irrelevant to the definition of
    > setPriority and getPriority, since each would, in my proposal, compute
    > for p > 1.0:
    >
    > Windows: 9 + int((p - 1) / (MAX_PRIORITY - 1) * 22 + .5)
    > Unix: -int((p - 1) / (MAX_PRIORITY - 1) * 20 + .5)
    >
    > Anyway, that's how I'd propose to do the nitty-gritty. But, more than
    > anything, I think the subprocess 'priority' methods should use a
    > priority scheme that is easy to explain. And by that, I propose:
    >
    > 1.0 represents normal priority, 100%. Any priority less than 1
    > represents a below normal priority, down to 0.0, the lowest possible
    > priority or 0%. Any priority above 1.0 represents an above normal
    > priority, with MAX_PRIORITY being the highest priority level available
    > for a given os.
    >
    > Granted, it's not much simpler than 0 is normal, -20 is highest and
    > +20 is lowest, except in so far as it being non-intuitive to consider
    > a lower priority number representing a higher priority. Certainly, we
    > could conform all systems to the +20.0 to -20.0 floating point system,
    > but I prefer not to bias the methods and honestly feel percentage is
    > more intuitive.
    >
    > So, how does that sound to people? Is that more palatable?


    I think it might give people the idea that if you set it to 0.1 you'll
    get 1/10 of the cpu than normal. Likewise 2.0 etc. There is
    something to be said for the deliberately somewhat arbitrary nice
    levels in the unix world!

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
    Nick Craig-Wood, Feb 25, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Christoph Becker-Freyseng

    PEP for new modules (I read PEP 2)

    Christoph Becker-Freyseng, Jan 15, 2004, in forum: Python
    Replies:
    3
    Views:
    360
    Gerrit Holl
    Jan 16, 2004
  2. tharaka
    Replies:
    1
    Views:
    251
    Diez B. Roggisch
    Oct 18, 2005
  3. Lie
    Replies:
    25
    Views:
    718
    Dafydd Hughes
    Dec 18, 2007
  4. Marcel Müller
    Replies:
    3
    Views:
    533
    Marcel Müller
    Apr 27, 2009
  5. hiral
    Replies:
    2
    Views:
    583
    Jean-Michel Pichavant
    May 5, 2010
Loading...

Share This Page