Language Enhancement Idea to help with multi-processing (your opinionsplease)

[Martin P. Hellwig]

First of all let me say that I have no authority or knowledge of
language design or multi-processing except from a user point of view,
having a decade or so experience.
I would like your opinion and appreciate any feedback and value any
hints to documentation, procedures or related ramblings .

I was wondering if there could be an advantage to add another control
flow statement.
For the purpose of this writing let's say "ooo" which stands for 'out of
order'.

For example;

def do_something():
a = 4
b = 2
c = 1
ooo:
a += 1
b += 2
c += 3
print(a, b, c)

What I would expect to happen that all statements within the ooo block
may be executed out
of order. The block itself waits till all statements are returned before
continuing.

What do you think?

 

[Chris Angelico]

What I would expect to happen that all statements within the ooo block may
be executed out
of order. The block itself waits till all statements are returned before
continuing.

In a statically-typed language such as C, this can be done by an
optimizing compiler, without any hints from the programmer. But in
Python, what you're looking for is either a guarantee from the
programmer that none of the statements has a side effect that affects
any other, or that you're prepared to wear it and have a horrendous
debugging job if anyone monkey-patches your code or you do anything
that makes a, b, or c into something more complex than an integer.

ChrisA

 

[Stefan Behnel]

Martin P. Hellwig, 13.10.2011 14:35:

I was wondering if there could be an advantage to add another control flow
statement.

Changes at that level must be very well justified, are often rejected for
the reason of being not too complicated to write in some other form and are
close to impossible to get accepted when requiring a new keyword.

Also, the right place to discuss (and, more importantly, search for
previous) ideas about language changes is the python-ideas mailing list.

For the purpose of this writing let's say "ooo" which stands for 'out of
order'.

For example;

def do_something():
a = 4
b = 2
c = 1
ooo:
a += 1
b += 2
c += 3
print(a, b, c)

What I would expect to happen that all statements within the ooo block may
be executed out
of order. The block itself waits till all statements are returned before
continuing.

This looks like a rather special case. What if you need more than one
statement to accomplish a single step? What if the number of parallel tasks
is not fixed at coding time? I don't think there are many problems that you
can solve with this feature.

Also: the GIL will not allow you to take a major advantage from the
parallel execution of a set of statements, as only one of the statements
can use the interpreter at a time.

And, on a related note: Cython has freshly gained support for parallel
loops based on OpenMP, so you may be able to solve your problem with Cython
instead of using plain Python.

Stefan

 

[jkn]

FWIW, this looks rather like the 'PAR' construct of Occam to me.

http://en.wikipedia.org/wiki/Occam_(programming_language)

J^n

 

[Martin P. Hellwig]

Martin P. Hellwig, 13.10.2011 14:35:

Changes at that level must be very well justified, are often rejected
for the reason of being not too complicated to write in some other form
and are close to impossible to get accepted when requiring a new keyword.

Also, the right place to discuss (and, more importantly, search for
previous) ideas about language changes is the python-ideas mailing list.



This looks like a rather special case. What if you need more than one
statement to accomplish a single step?

Aah yes haven't thought about that, was more thinking like wrap multiple
steps in a function and put that as one line, but yeah that kind of
stuff gets very unpythonic very quickly.

What if the number of parallel
tasks is not fixed at coding time? I don't think there are many problems
that you can solve with this feature.

Also: the GIL will not allow you to take a major advantage from the
parallel execution of a set of statements, as only one of the statements
can use the interpreter at a time.

Well I was more or less thinking in the line that the interpreter when
parsing such a statement can, if there are multiple cpu's available,
fire the appropriate amount of other interpreters and do all the passing
and locking of data for you.

And, on a related note: Cython has freshly gained support for parallel
loops based on OpenMP, so you may be able to solve your problem with
Cython instead of using plain Python.

Well funny enough I don't really have a problem with it, I am a happy
consumer of multiprocessing, but I was just wondering if there would be
a way to do it simpler and more pythonic.

Stefan

Thanks for your feedback it was very enlightening although for what the
idea is concerned disappointing as it more or less makes it obvious that
such a thing is a no-go.

Cheers,

Martin

 

[Steven D'Aprano]

I was wondering if there could be an advantage to add another control
flow statement.
For the purpose of this writing let's say "ooo" which stands for 'out of
order'. [...]
What I would expect to happen that all statements within the ooo block
may be executed out
of order. The block itself waits till all statements are returned before
continuing.

Why do you think this needs to be a language statement?

You can have that functionality *right now*, without waiting for a syntax
update, by use of the multiprocessing module, or a third party module.

http://docs.python.org/library/multiprocessing.html
http://wiki.python.org/moin/ParallelProcessing

There's no need for forcing language changes on everyone, whether they need
it or not, for features that can easily be implemented as library code.

 

[Carl Banks]

Why do you think this needs to be a language statement?

You can have that functionality *right now*, without waiting for a syntax
update, by use of the multiprocessing module, or a third party module.

http://docs.python.org/library/multiprocessing.html
http://wiki.python.org/moin/ParallelProcessing

There's no need for forcing language changes on everyone, whether they need
it or not, for features that can easily be implemented as library code.

This goes a little beyond a simple threading mechanism, though. It's more like guidance to the compiler that you don't care what order these are executed in; the compiler is then free to take advantage of this advice howeverit like. That could be to spawn threads, but it could also compile instructions to optimize pipelining and cacheing. The compiler could also ignoreit. But you can see that, fully realized, syntax like that can do much more than can be done with library code.

Obviously that extra capability is a very long way off for being useful in CPython.


Carl Banks

 

[Carl Banks]

What I would expect to happen that all statements within the ooo block
may be executed out
of order. The block itself waits till all statements are returned before
continuing.

What do you think?

The statement is kind of limiting as a unit of uncaring. What if you have two statements that you do want to be executed in order, but still don't care what order they are executed in relative to other sets of two statements? Better would be to have a set of blocks that the compiler is free to execute asynchronously relative to each other (I'll call it async).

async:
a += 1
f *= a
async:
b += 1
e *= b
async:
c += 1
d *= c


There is utterly no chance of this syntax entering Python.


Carl Banks

 

[Steven D'Aprano]

This goes a little beyond a simple threading mechanism, though.

The multiprocessing module is not a simple threading mechanism, and neither
are the libraries listed in the ParallelProcessing page.

 

[Westley Martínez]

This goes a little beyond a simple threading mechanism, though. It's more like guidance to the compiler that you don't care what order these are executed in; the compiler is then free to take advantage of this advice however it like. That could be to spawn threads, but it could also compile instructions to optimize pipelining and cacheing. The compiler could also ignore it. But you can see that, fully realized, syntax like that can do much more than can be done with library code.

Obviously that extra capability is a very long way off for being useful in CPython.

While we're at it, let's throw in the register keyword.

 

[alex23]

But you can see that, fully realized, syntax like that can do much more
than can be done with library code.

Well sure, but imaginary syntax can do _anything_. That doesn't mean
it's possible within CPython.

 

[Chris Rebert]

Well sure, but imaginary syntax can do _anything_. That doesn't mean
it's possible within CPython.

But it's The Future now! Where are my jetpack and `dwim` statement,
dammit?!

Cheers,
Chris

 

[alex23]

def do_something():
     a = 4
     b = 2
     c = 1
     ooo:
         a += 1
         b += 2
         c += 3
     print(a, b, c)

What I would expect to happen that all statements within the ooo block
may be executed out
of order. The block itself waits till all statements are returned before
continuing.

What do you think?

You can do this right now with Python 3.2+ and concurrent.futures:

from concurrent.futures import ThreadPoolExecutor
from functools import partial
import time

class DoSomething:
a = 4
b = 2
c = 1

def add(self, prop, val):
cur = getattr(self, prop)
time.sleep(cur)
print('Adding %d to %s' % (val, prop))
setattr(self, prop, cur + val)

def __init__(self):
with ThreadPoolExecutor(max_workers=3) as pool:
pool.submit(self.add, 'a', 1)
pool.submit(self.add, 'b', 2)
pool.submit(self.add, 'c', 3)
print(self.a, self.b, self.c)

DoSomething()

Here we call 'ooo' the ThreadPoolExecutor context manager

 

[alex23]

from functools import partial

You can ignore this, sorry, leftover from earlier code

 

[Chris Angelico]

But it's The Future now! Where are my jetpack and `dwim` statement,
dammit?!  

If you can imagine something your language can't do, you're not using
enough preprocessors.

ChrisA
who, frustrated by Java about ten years ago, started running his .java
files through the C preprocessor... and it can get a lot worse than
that

 

[Grant Edwards]

On Thu, Oct 13, 2011 at 11:56:20PM -0700, Carl Banks wrote:

While we're at it, let's throw in the register keyword.

Yah! C compilers have been ignoring it for ages, and I miss it.

 

[Carl Banks]

Well sure, but imaginary syntax can do _anything_. That doesn't mean
it's possible within CPython.

Hey, thanks for backing me up on that sentiment.


Carl Banks

 

[John Ladasky]

You can do this right now with Python 3.2+ and concurrent.futures:

from concurrent.futures import ThreadPoolExecutor

You may have finally sold me on struggling through the upgrade from
Python 2.6! I've been doing reasonably well with the Multiprocessing
module, but it looks like the syntax of ThreadPoolExecutor offers some
significant improvements.

Here's hoping that numpy, scipy, and matplotlib are all Python 3.x-
compatible by now...