Augmented generators?

[Andrew Koenig]

Can anyone think of an easy technique for creating an object that acts like
a generator but has additional methods?

For example, it might be nice to be able to iterate through an associative
container without having to index it for each element. Right now, I can say

i = iter(d)

and then repeatedly calling i.next() gives us the keys for the elements.
But to get the corresponding value requires us to look up the key.

Of course one could define a generator that yields key-value pairs, along
the following lines:

def kviter(d):
for i in d:
yield i, d

to hide the lookup. But this yields a tuple even when you don't want it.
In other words, I must now write

for k, v in kviter(d):
# whatever

and I can think of situations in which I don't really want both the key and
the value all the time.

So what I really want is something like this:

it = augiter(d)
for i in it:
if <some condition on i>:
foo(it.value())

In other words, I want "it" to support both the next and value methods (or
next and something else)

Of course I can write such a beast as a class, but that prevents me from
taking advantage of the yield statement in its implementation.

So my question is: Can you think of an easy way to write something that
looks like a generator (using yield), but can also incorporate methods other
than next?

 

[Paul Rubin]

Can anyone think of an easy technique for creating an object that acts like
a generator but has additional methods?

For example, it might be nice to be able to iterate through an associative
container without having to index it for each element.

Normally you'd define a class and give it __iter__ and next operations.
I guess that can get messy. Does it qualify as easy for your purposes?

Of course I can write such a beast as a class, but that prevents me from
taking advantage of the yield statement in its implementation.

You can make an internal function that's a generator with a yield
statement (or a generator expression instead of a function, if simple
enough). The class's 'next' method would loop through the generator
and return each value from it.

Let me try your example:

class kviter:
def __init__(self, d):
self.d = d
def __iter__(self):
def iter1(d, s=self):
for k in d:
# lambda not really needed here, but you wanted value to
# be callable instead of just an attribute
s.value = (lambda r=d[k]: r)
yield k
return iter1(self.d)

a = {1:5,2:6,3:7}
it = kviter(a)
for b in it:
print b, it.value()

1 5
2 6
3 7

 

[Lonnie Princehouse]

AFAIK there's no way to have "yield" produce anything other than a
generator.
You could achieve the syntax you want with a decorator, although under
the hood it would still be iterating over a (key,value) tuples so
there's not really any point.

class GeneratorTupleWrapper:
def __init__(self, g):
self.g = g
def next(self):
key, self._value = g.next()
return key
def value(self):
return self._value

def magic_generator (func):
def wrapper(*a, **b):
return GeneratorTupleWrapper(func(*a, **b))

@magic_generator
def kviter(d):
for i in d:
yield i, d

it = kviter(d)
for key in it:
if <something>:
foo(it.value())




Also: dict.iteritems() enumerates (key,value) tuples for dictionaries,
so kviter isn't necessary if d is a dictionary. I don't know the
internal mechanism used by dict for iteritems, but it's a fair bet that
it's more efficient than performing a lookup for every single key.

 

[Raymond Hettinger]

Can anyone think of an easy technique for creating an object that acts like
a generator but has additional methods?

Perhaps the enable_attributes() recipe will help:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/164044

The recipe imbues generators with classlike attribute reads and writes.
It shouldn't be hard to adapt the technique for attaching methods.


Raymond

 

[Bengt Richter]

Can anyone think of an easy technique for creating an object that acts like
a generator but has additional methods?

For example, it might be nice to be able to iterate through an associative
container without having to index it for each element. Right now, I can say

i = iter(d)

and then repeatedly calling i.next() gives us the keys for the elements.
But to get the corresponding value requires us to look up the key.

Of course one could define a generator that yields key-value pairs, along
the following lines:

def kviter(d):
for i in d:
yield i, d

to hide the lookup. But this yields a tuple even when you don't want it.
In other words, I must now write

for k, v in kviter(d):
# whatever

and I can think of situations in which I don't really want both the key and
the value all the time.

So what I really want is something like this:

it = augiter(d)
for i in it:
if <some condition on i>:
foo(it.value())

In other words, I want "it" to support both the next and value methods (or
next and something else)

Of course I can write such a beast as a class, but that prevents me from
taking advantage of the yield statement in its implementation.

It does? (this just occurred to me, so there may be some gotcha ;-)
... def __init__(self, d): self.d = d
... def __iter__(self):
... for k in self.d: self.value = self.d[k]; yield k
... ... if i%2: print i, it.value
...
1 b
3 d

You could of course store self._value and def value(self):return self._value to be closer to your syntax)

So my question is: Can you think of an easy way to write something that
looks like a generator (using yield), but can also incorporate methods other
than next?

See above.

Regards,
Bengt Richter

 

[Bengt Richter]

Sorry to reply this way. I saw this on google, but neither this nor my previous post
has shown up yet on my news server. Wonder what all this delay is lately ;-/

From: Paul Rubin <http://[email protected]>
Subject: Re: Augmented generators?
Date: 10 Jan 2006 11:03:39 -0800
Message-ID: <[email protected]>

Normally you'd define a class and give it __iter__ and next operations.
I guess that can get messy. Does it qualify as easy for your purposes?

You can make an internal function that's a generator with a yield
statement (or a generator expression instead of a function, if simple
enough). The class's 'next' method would loop through the generator
and return each value from it.

Let me try your example:

This suffers from the same problem as my first go (an exhausted iterator
is not supposed to restart).
... def __init__(self, d):
... self.d = d
... def __iter__(self):
... def iter1(d, s=self):
... for k in d:
... # lambda not really needed here, but you wanted value to
... # be callable instead of just an attribute
... s.value = (lambda r=d[k]: r)
... yield k
... return iter1(self.d)
...

Using my example with your iterator:
... if i%2: print i, it.value()
...
1 b
3 d

Ok, but this shouldn't happen at this point:
... if i%2: print i, it.value()
...
1 b
3 d

Whereas,
... def __init__(self, d):
... self._it = self._gen(d)
... def __iter__(self): return self
... def _gen(self, d):
... for k, self._v in d.items(): yield k
... def next(self): return self._it.next()
... def value(self): return self._v
... ... if i%2: print i, it.value()
...
1 b
3 d ... if i%2: print i, it.value()
...
Letting __init__ create the generator instance by calling a bound
method coded with yield makes integrating the latter style pretty easy,
even though you still need __iter__ and next methods.

Hm, you could factor that out into a base class though:
(then you just need the convention that a derived class must define at least
the _gen method. Then augment to taste)
... def __init__(self, *args, **kw):
... self._it = self._gen(*args, **kw)
... def __iter__(self): return self
... def next(self): return self._it.next()
... ... def _gen(self, d):
... for k, self._v in d.items(): yield k
... def value(self): return self._v
... ... if i%2: print i, it.value()
...
1 b
3 d ... print i, it.value()
... ... print i, it.value()
...
0 a
1 b
2 c
3 d

Regards,
Bengt Richter