cloning generator iterators

[Bernhard Mulder]

I have something like the following (using Python 2.5 syntax):

class adapt(object):
def __init__(self):
self.iterator = self.generator()
self.n = 0

def generator(self):
while True:
while True:
if (yield 1):
if (yield 1):
break
self.n = 2
while (yield self.n):
self.n += 1

def clone(self):
clone = self.__class__()
clone.n = self.n

# This assignment does not work!
clone.iterator.gi_frame.f_lasti = clone.iterator.gi_frame.f_lasti

a = adapt()
c = a.clone()

c should be an independent clone of a, *including the program counter*.

However, I have trouble adjusting the pc. The assignment above does not
work since f_lasti is a readonly attribute.

c1 = copy.copy(a)
c1.iterator = copy.copy(a.iterator)

also does not work, since a.iterator can not be copied.

Question: What is the best way to implement this clone operation?

Solutions I came up with:

1. Don't use generators, and program a state based approached directly,
2. Keep track of the values that where sent to the generator iterator,
and feed the same values to the clone.

I would like to keep the clone operation fairly independent from the
generator, so that I can try out different generators (the generator
above is probably not optimal).

2. is somewhat unsatisfactory, since with the clone operation become
more expensive with long histories.


Motivation:

Abstractly speaking, I have a list of elements, and the list as a
whole has property P. Now some of these elements can be deleted, and
the list still has property P. The goal of the program is to delete as
many elements of the list as possible without loosing property P.

Now assume the following additional two conditions:

1. Testing for property P is time consuming.

2. Elements which can be deleted often come in 'clusters', that is
elements at index i, i+1, i+2, i+3, ... can all be deleted without
loosing property p.

I am trying to address 1. and 2. by the above generator which I use to
determine how many elements to delete. After two successful deletions,
I delete 2, 3, 4, ... elements until the list does not show property P
anymore.

Now the generator helps, but it is still taking (too) long. On
multi-core machines (soon coming to a desktop near you) I can reduce
the elapsed time by testing three lists simultaneously: the second list
is constructed under the assumption that the first list has property
P, and the third list is constructed under the assumption that the
first list does *not* have property P.

My idea was to clone the class constructing the lists to test, which
includes a clone of the adapt class sketched above. One of the two
clones (for list 2 and 3) will be thrown away depending on the outcome
of the test of list 1.

 

[Bernhard Mulder]

Oops. Got the indentation wrong. Here is the corrected version:

def generator(self):
while True:
while True:
if (yield 1):
if (yield 1):
break
self.n = 2
while (yield self.n):
self.n += 1

 

[Michael]

[ attempt to clone/fork a generator ]

You can do this, but you can't pickle the results. (If you want pickling,
use Stackless - I've not tried pickling generators in stackless because I
don't use stackless, but it looks pretty clear you can pickle them there)

Question: What is the best way to implement this clone operation?

However you should be able to do what you want using the small extension
"statesaver" which you can grab from
* http://www.gosubway.org/install/statesaver.c

The statesaver allows you to clone generators in the way that you want.

FWIW, I can see lots of situations where this would be useful - mainly in
the area of dealing with search spaces (After all, this effectively allows
you to fork the generator).


Michael.