What are you trying to accomplish?
What are you trying to accomplish? While it's possible to do, I can't
believe it's going to be very safe.
I am trying to implement a completely new concurrency abstraction for
Python. It is modeled on OpenMP instead of Java threads (cf. threading
and multiprocessing). I am planning to use thread and multiprocessing
(or os.fork) as backends. Doing this with os.fork is child's play, but
it would not work on Windows.
I believe Python context managers are excellent for this purpose. I
believe the OpenMP way of doing concurrency fits better with the mind
than Java threading, and thus are more pythonic. Algorithms can be
written as sequential, and transformed to parallel code with the
simple insertion of a few directives. It makes debugging easier,
because one can debug the sequential code, and it makes it easier to
avoid the problems with deadlocks, race conditions, livelocks, etc.
Just ask any C or Fortran programmer how much easier it is to use
OpenMP instead of Win32 or Posix threads.
What I have in mind is an API that would look approximately like this
(OpenMP pragmas for C on top, proposed Python equivalent below):
#pragma omp parallel
with Pool() as pool:
#pragma omp for
for item in pool.parallel(<iterable>):
#pragma omp for shedule(guided)
for item in pool.parallel(<iterable>, shed='guided'):
#pragma omp parallel for
with Pool() as pool:
for item in pool.parallel(<iterable>):
#pragma omp barrier
pool.barrier()
#pragma omp section
with pool.section():
#pragma omp parallel sections
with Pool() as pool:
with pool.section():
fun1(*args, **kwargs)
with pool.section():
fun2(*args, **kwargs)
#pragma omp master
if pool.master:
#pragma omp critical
#pragma omp atomic
with pool.lock:
#pragma omp single
with pool.single():
#pragma omp ordered
with ordered(k):
Here is a toy example of what I have in mind. Say you would want to
compute the DFT of some signal (real apps would use an FFT in C for
this, but never mind that). In Python using an O(n**2) algorithm, this
would look like something like this:
def real_dft(x):
''' DFT for a real valued sequence x '''
r = []
N = len(x)
M = N//2 + 1 if N%2 else N//2
for n in range(M):
s = 0j
for k in range(N):
tmp = 2*pi*k*n/N
s += x[k] * (cos(tmp) - 1j*sin(tmp))
r.append(s)
return r
Then, one could 'magically' transform this algorithm into to a
parallel one simply by inserting directives from the 'OpenMP' module:
def real_dft(x):
''' DFT for a real valued sequence x '''
''' parallelized '''
r = []
N = len(x)
M = N//2 + 1 if N%2 else N//2
with Pool() as pool:
parallel_iter = pool.parallel(range(M)):
for n in parallel_iter:
s = 0j
for k in range(N):
tmp = 2*pi*k*n/N
s += x[k] * (cos(tmp) - 1j*sin(tmp))
with parallel_iter.ordered(n):
r.append(s)
return r
The idea is that 'parallelizing' a sequential algorithm like this is
much easier than writing a parallel one from scratch using the
abstractions in threading or multiprocessing.
The problem is that the __enter__ method of the Pool object (the
context manager) must spawn multiple threads, each executing
sys._getframe().f_back, because the 'with Pool() as pool:' should be
executed by multiple threads in parallel. These slave threads will be
killed when they enter the __exit__ method of its context manager.
Most of this OpenMP API has already been implemented. I just need the
machinery to make the slave threads come alive
I'll take a look at your code now
Regards,
Sturla Molden
