Parallel/distributed generator

[George Sakkis]

I'm looking for any existing packages or ideas on how to implement the
equivalent of a generator (in the Python sense, i.e.
http://www.python.org/dev/peps/pep-0255/) in a parallel/distributed
way. As a use case, imagine a function that generates a range of
primes. I'd like to be able to do something along the following lines:

def iterprimes(start=1, end=None):
# ...
yield prime


# rpc-related initialization
....
rpc_proxy = some_rpc_lib(iterprimes, start=1e6, end=1e12)
for prime in proxy:
print prime

Is there any module out there that does anything close to this ?

George

 

[Diez B. Roggisch]

I'm looking for any existing packages or ideas on how to implement the
equivalent of a generator (in the Python sense, i.e.
http://www.python.org/dev/peps/pep-0255/) in a parallel/distributed
way. As a use case, imagine a function that generates a range of
primes. I'd like to be able to do something along the following lines:

def iterprimes(start=1, end=None):
# ...
yield prime


# rpc-related initialization
...
rpc_proxy = some_rpc_lib(iterprimes, start=1e6, end=1e12)
for prime in proxy:
print prime

Is there any module out there that does anything close to this ?

Have you tried using pyro to just return a generator? After all, it's just
an object with a next()-method, that raises a certain exception. I can't
see why pyro shouldn't be able to handle that.

diez

 

[Paul McGuire]

DOLT!- Hide quoted text -

- Show quoted text -

I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.

-- Paul

 

[George Sakkis]

I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.

-- Paul

No, he probably sent it from Google groups as I did, which claimed
that "your message has not been sent, please try again later". Go
figure.

George

 

[half.italian]

I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.
I thought you were making a joke about parallel processing.

-- Paul

Damn computers!

 

[Simon Brunning]

I'm looking for any existing packages or ideas on how to implement the
equivalent of a generator (in the Python sense, i.e.
http://www.python.org/dev/peps/pep-0255/) in a parallel/distributed
way.

Kamaelia? http://tinyurl.com/35fjbr

--
Cheers,
Simon B.
(e-mail address removed)
http://www.brunningonline.net/simon/blog/
GTalk: simon.brunning | MSN: small_values | Yahoo: smallvalues