K
Kirk Strauser
I was thinking about how a lot of Lisp proponents claim that Lisp is
inherently parallelizable because its functions don't have (or are not
supposed to have) side effects, and therefore the compiler can easily tell
which calls may be run in parallel instead of strictly serially. I'm not a
Lisp expert so I can't say whether that's true or not, but it seems like an
interesting idea for Python.
Suppose that Python had a new decorator, say "@parallelizable". Functions
so decorated would be eligible for multi-processed or multi-threaded
execution by such native constructs as list comprehensions, or the map()
function. Their logic could be something along the lines of:
1) Is every function in the "function expression" of map() or a list
comprehension decorated with @parallelizable? If not, use the normal
implementation.
2) Spawn an appropriate number of child threads and queue values to be
processed.
3) As results return, either store them in the result list (for map()), or
insert them into their place in the output queue to be consumed by users of
a list comprehension.
In #2, "appropriate" could default to something like 2 times the number of
CPUs detected in the system (which could be found the first time it was
needed so that programs that don't use this functionality don't pay for
finding that information). I could imagine that it'd be useful to pass
hints to the decorator to more explicitly size the thread pool, such as:
@parallelizable(number=100)
def lightweightfunction:
"""Launch as many parallel copies as you can"""
@parallelizable(perproc=1)
def heavierfunction:
"""Only run one copy per CPU"""
@parallelizable(number=2)
def bigslowfunction:
"""This benefits from parallel execution, but don't kill the system"""
Would something like this have merit? I like the idea because it's
completely optional; if you don't use it, you never have to deal with the
side effects. However, if you take care to make your functions loosely
coupled and not dependent on execution order, you could get a nice speedup
every single time you give Python permission to schedule your repeated
function calls.
inherently parallelizable because its functions don't have (or are not
supposed to have) side effects, and therefore the compiler can easily tell
which calls may be run in parallel instead of strictly serially. I'm not a
Lisp expert so I can't say whether that's true or not, but it seems like an
interesting idea for Python.
Suppose that Python had a new decorator, say "@parallelizable". Functions
so decorated would be eligible for multi-processed or multi-threaded
execution by such native constructs as list comprehensions, or the map()
function. Their logic could be something along the lines of:
1) Is every function in the "function expression" of map() or a list
comprehension decorated with @parallelizable? If not, use the normal
implementation.
2) Spawn an appropriate number of child threads and queue values to be
processed.
3) As results return, either store them in the result list (for map()), or
insert them into their place in the output queue to be consumed by users of
a list comprehension.
In #2, "appropriate" could default to something like 2 times the number of
CPUs detected in the system (which could be found the first time it was
needed so that programs that don't use this functionality don't pay for
finding that information). I could imagine that it'd be useful to pass
hints to the decorator to more explicitly size the thread pool, such as:
@parallelizable(number=100)
def lightweightfunction:
"""Launch as many parallel copies as you can"""
@parallelizable(perproc=1)
def heavierfunction:
"""Only run one copy per CPU"""
@parallelizable(number=2)
def bigslowfunction:
"""This benefits from parallel execution, but don't kill the system"""
Would something like this have merit? I like the idea because it's
completely optional; if you don't use it, you never have to deal with the
side effects. However, if you take care to make your functions loosely
coupled and not dependent on execution order, you could get a nice speedup
every single time you give Python permission to schedule your repeated
function calls.