Java-style futures in Python - only better

[Brian Quinlan]

Hey all,

I've been working on an Java-style futures implementation in Python.
Futures are a way of representing asynchronous operations e.g.
operations that are run in another thread or process. The are are a easy
but powerful way of parallelizing sequential operations. The also
provide a consistent interface across implementations e.g. they can
provide the same interface to threading and to multiprocessing.

For example:

def is_prime(n):
"Return True iff n is prime"
...

def check_primes(numbers):
return map(is_prime, numbers)

Could be parallelized as:

def check_primes(numbers):
# ProcessPoolExecutor will create one worker process
# per CPU if called without arguments. Using threads
# is valueless because of the GIL.
with futures.ProcessPoolExecutor() as executor:
return executor.map(is_prime, numbers)

A more complex example:

def load_url(url, timeout):
return urllib.request.urlopen(url, timeout=timeout).read()

### Download the content of some URLs - ignore failures.
def download_urls(urls, timeout=60):
url_to_content = {}
for url in urls:
try:
url_to_content = load_url(url, timeout=timeout) ...ould be very much appreciated! Cheers, Brian

 

[Brian Quinlan]

Brian,

Since the word "future" is part of the Python lingo:

A future statement is a directive to the compiler that a particular
module should be compiled using syntax or semantics that will be
available in a specified future release of Python. The future statement
is intended to ease migration to future versions of Python that
introduce incompatible changes to the language. It allows use of the new
features on a per-module basis before the release in which the feature
becomes standard.

Have you given thought to the use of another word?

I named the module "futures" (plural) to try to reduce the potential
confusion with the "__futures__" module.

The concept of a future is fairly well known in CS
[http://en.wikipedia.org/wiki/Future_(programming)] so giving it a
completely different name would be a bit annoying.

Cheers,
Brian

Colin W.

Hey all,

I've been working on an Java-style futures implementation in Python.
Futures are a way of representing asynchronous operations e.g.
operations that are run in another thread or process. The are are a
easy but powerful way of parallelizing sequential operations. The also
provide a consistent interface across implementations e.g. they can
provide the same interface to threading and to multiprocessing.

For example:

def is_prime(n):
"Return True iff n is prime"
...

def check_primes(numbers):
return map(is_prime, numbers)

Could be parallelized as:

def check_primes(numbers):
# ProcessPoolExecutor will create one worker process
# per CPU if called without arguments. Using threads
# is valueless because of the GIL.
with futures.ProcessPoolExecutor() as executor:
return executor.map(is_prime, numbers)

A more complex example:

def load_url(url, timeout):
return urllib.request.urlopen(url, timeout=timeout).read()

### Download the content of some URLs - ignore failures.
def download_urls(urls, timeout=60):
url_to_content = {}
for url in urls:
try:
url_to_content = load_url(url, timeout=timeout) e... appreciated! Cheers, Brian [/QUOTE][/QUOTE]