P
Paul Boddie
Announcing the release of pprocess 0.3 (previously known as parallel/
pprocess), available from...
http://www.python.org/pypi/pprocess/0.3
Note that only POSIX-like platforms are supported in this release,
although improvements in portability and all other areas are welcome.
What is it?
-----------
The pprocess module provides several mechanisms for running Python
code concurrently in several processes. On systems with multiple CPUs
or multicore CPUs, processes should take advantage of as many CPUs or
cores as the operating system permits.
Several different abstractions are supported by the module, including
channels (and communicating sequential processes), queues and a
parallel map function. The module gives developers a certain amount of
flexibility in developing new abstractions, and a motivation behind
the development of the module was, in fact, to discover which
abstractions are most convenient when writing parallel-aware programs.
Tutorial
--------
In order to get a feel for the abstractions and mechanisms provided by
the module, a tutorial has been made available; it illustrates the
process of converting certain kinds of sequential programs into
parallel programs:
http://www.boddie.org.uk/python/pprocess/tutorial.html
Future Work
-----------
The module has been tested with non-trivial programs (see the PyGmy
raytracer example in the distribution), but performance results have
been varied: pprocess ensures the utilisation of many processors but
could arguably show better scalability. It is envisaged that the focus
for future releases involve improvements to the interprocess
communications mechanisms employed, along with an investigation into
more effective ways of distributing computations in specific programs.
In the context of recent discussions about Python concurrency support
[1], this module is intended as a demonstrator for various API
conveniences, with an aim of encouraging standardisation and
potentially the eventual standard library adoption of a useful
parallel processing module.
[1] http://jessenoller.com/2007/09/10/interesting-read-tear-down-that-gil/
pprocess), available from...
http://www.python.org/pypi/pprocess/0.3
Note that only POSIX-like platforms are supported in this release,
although improvements in portability and all other areas are welcome.
What is it?
-----------
The pprocess module provides several mechanisms for running Python
code concurrently in several processes. On systems with multiple CPUs
or multicore CPUs, processes should take advantage of as many CPUs or
cores as the operating system permits.
Several different abstractions are supported by the module, including
channels (and communicating sequential processes), queues and a
parallel map function. The module gives developers a certain amount of
flexibility in developing new abstractions, and a motivation behind
the development of the module was, in fact, to discover which
abstractions are most convenient when writing parallel-aware programs.
Tutorial
--------
In order to get a feel for the abstractions and mechanisms provided by
the module, a tutorial has been made available; it illustrates the
process of converting certain kinds of sequential programs into
parallel programs:
http://www.boddie.org.uk/python/pprocess/tutorial.html
Future Work
-----------
The module has been tested with non-trivial programs (see the PyGmy
raytracer example in the distribution), but performance results have
been varied: pprocess ensures the utilisation of many processors but
could arguably show better scalability. It is envisaged that the focus
for future releases involve improvements to the interprocess
communications mechanisms employed, along with an investigation into
more effective ways of distributing computations in specific programs.
In the context of recent discussions about Python concurrency support
[1], this module is intended as a demonstrator for various API
conveniences, with an aim of encouraging standardisation and
potentially the eventual standard library adoption of a useful
parallel processing module.
[1] http://jessenoller.com/2007/09/10/interesting-read-tear-down-that-gil/