revive a generator

Discussion in 'Python' started by Yingjie Lan, Oct 20, 2011.

1. Yingjie LanGuest

Hi,

it seems a generator expression can be used only once:

>>> g = (x*x for x in range(3))
>>> for x in g: print x

0
1
4
>>> for x in g: print x #nothing printed
>>>

Is there any way to revive g here?

Yingjie

Yingjie Lan, Oct 20, 2011

2. Paul RudinGuest

Yingjie Lan <> writes:

> Hi,
>
> it seems a generator expression can be used only once:
>
>>>> g = (x*x for x in range(3))
>>>> for x in g: print x

> 0
> 1
> 4
>>>> for x in g: print x #nothing printed
>>>>

>
> Is there any way to revive g here?
>

Generators are like that - you consume them until they run out of
values. You could have done [x*x for x in range(3)] and then iterated
over that list as many times as you wanted.

A generator doesn't have to remember all the values it generates so it
can be more memory efficient that a list. Also it can, for example,
generate an infinite sequence.

Paul Rudin, Oct 20, 2011

3. Yingjie LanGuest

----- Original Message -----
> From: Paul Rudin <>
> To:
> Cc:
> Sent: Thursday, October 20, 2011 10:28 PM
> Subject: Re: revive a generator
>
> Yingjie Lan <> writes:
>
>> Hi,
>>
>> it seems a generator expression can be used only once:
>>
>>>>> g = (x*x for x inrange(3))
>>>>> for x in g: print x

>> 0
>> 1
>> 4
>>>>> for x in g: print x #nothing printed
>>>>>

>>
>> Is there any way torevive g here?
>>

>
> Generators are like that - you consume them until they run out of
> values. You could have done [x*x for x in range(3)] and then iterated
> over that list as many times as you wanted.
>
> A generator doesn't have to remember all the values it generates so it
> can be more memory efficient that a list. Also it can, for example,
> generate an infinite sequence.
>
>

Thanks a lot to all who answered my question.
I am still not sure why should we enforce that
a generator can not be reused after an explicit
request to revive it?

Yingjie Lan, Oct 21, 2011
4. Chris AngelicoGuest

On Fri, Oct 21, 2011 at 12:46 PM, Yingjie Lan <> wrote:
>
> Thanks a lot to all who answered my question.
> I am still not sure why should we enforce that
> a generator can not be reused after an explicit
> request to revive it?

Here's an example of an explicit request to revive the generator:

>>> g = (x*x for x in range(3))
>>> for x in g: print x

0
1
4
>>> g = (x*x for x in range(3)) # revive the generator
>>> for x in g: print x #now this will work

0
1
4

ChrisA

Chris Angelico, Oct 21, 2011
5. Paul RudinGuest

Yingjie Lan <> writes:

> ----- Original Message -----
>> From: Paul Rudin <>

>> Generators are like that - you consume them until they run out of
>> values. You could have done [x*x for x in range(3)] and then iterated
>> over that list as many times as you wanted.
>>
>> A generator doesn't have to remember all the values it generates so it
>> can be more memory efficient that a list. Also it can, for example,
>> generate an infinite sequence.
>>
>>

> Thanks a lot to all who answered my question.Â
> I am still not sure why should we enforce thatÂ
> a generator can not be reused after an explicitÂ
> request to revive it?

The language has no explicit notion of a request to "revive" a
generator. You could use the same syntax to make a new generator that
yeilds the same values as the one you started with if that's what you
want.

As we've already discussed if you want to iterate several times over the
same values then it probably makes sense to compute them and store them
in e.g. a list (although there are always trade-offs between storage use
and the cost of computing things again).

Paul Rudin, Oct 21, 2011
6. Yingjie LanGuest

----- Original Message -----

> From: Paul Rudin <>
> To:
> Cc:
> Sent: Friday, October 21, 2011 3:27 PM
> Subject: Re: revive a generator
>
>
> The language has no explicit notion of a request to "revive" a
> generator. You could use the same syntax to make a new generator that
> yeilds the same valuesas the one you started with if that's what you
> want.
>
> As we've already discussed if you want to iterate several times over the
> same values then it probably makes sense to compute them and store them
> in e..g. a list (although there are always trade-offs between storage use
> and the cost of computing things again).
>
>

What if the generator involves a variable from another scope,
and before re-generating, thevariable changed its value.
Also, the generator could be passed in as anargument,
so that we don't know its exact expression.

>>> vo = 34
>>> g = (vo*x for x in range(3))
>>> def myfun(g):

for i in g: print(i)
vo  += 3
revive(g) #best if revived automatically
for i in g: print(i)
myfun(g)

Yingjie

Yingjie Lan, Oct 21, 2011
7. Yingjie LanGuest

----- Original Message -----

> From: Paul Rudin <>
> The language has no explicit notion of a request to "revive" a
> generator. You could use the same syntax to make a new generator that
> yeilds the same values as the one you started with if that's what you
> want.
>
> As we've already discussed if you want to iterate several times over the
> same values then it probably makes sense to compute them and store them
> in e.g. a list (although there are always trade-offs between storage use
> and the cost of computing things again).
>

Oops,my former reply has the code indentation messed up
by the mail system.. Here is a reformatted one:

What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value.
Also, the generator could be passed in as an argument,
so thatwe don't know its exact expression.

>>> vo = 34
>>> g = (vo*xfor x in range(3))
>>> def myfun(g):

for i in g: print(i)
vo  += 3
revive(g) #best if revived automatically
for i in g: print(i)
>>> myfun(g)

Yingjie

Yingjie Lan, Oct 21, 2011
8. Paul RudinGuest

Yingjie Lan <> writes:

>
>
> What if the generator involves a variable from another scope,
> and before re-generating, the variable changed its value.
> Also, the generator could be passed in as an argument,
> so that we don't know its exact expression.
>
>>>> vo = 34
>>>>Â g = (vo*x for x in range(3))
>>>> def myfun(g):

> Â  Â  Â  Â  Â  Â  for i in g: print(i)
> Â  Â  Â  Â  Â  Â Â vo Â += 3
> Â  Â  Â  Â  Â  Â Â revive(g) #best if revived automatically
> Â  Â  Â  Â  Â  Â Â for i in g: print(i)
>>>> myfun(g)

>
>

I'm not really sure whether you intend g to yield the original values
after your "revive" or new values based on the new value of vo. But
still you can make a class that supports the iterator protocol and does
whatever you want (but you can't use the generator expression syntax).

If you want something along these lines you should probably read up on
the .send() part of the generator protocol.

As an aside you shouldn't really write code that uses a global in that
way.. it'll end up biting you eventually.

Anyway... we can speculate endlessly about non-existent language
constructs, but I think we've probably done this one to death.

Paul Rudin, Oct 21, 2011
9. Chris AngelicoGuest

On Fri, Oct 21, 2011 at 7:02 PM, Yingjie Lan <> wrote:
> What if the generator involves a variable from another scope,
> and before re-generating, the variable changed its value.
> Also, the generator could be passed in as an argument,
> so that we don't know its exact expression.
>

There's actually no way to know that the generator's even
deterministic. Try this, for instance:

>>> g=(input("Enter value %d or blank to stop: "%n) for n in range(1,11))
>>> for s in g:

if not s: break
print("Processing input: "+s)

It may not be particularly useful, but it's certainly legal. And this
generator cannot viably be restarted. The only way is to cast it to
list first, but that doesn't work when you have to stop reading
expressions from the generator part way.

What you could perhaps do is wrap the generator in something that
saves its values:

>>> class restartable(object):

def __init__(self,gen):
self.gen=gen
self.yielded=[]
self.iter=iter(self.yielded)
def restart(self):
self.iter=iter(self.yielded)
def __iter__(self):
return self
def __next__(self): # Remove the underscores for Python 2
try:
return self.iter.__next__()
except StopIteration:
pass
ret=self.gen.__next__()
self.yielded.append(ret)
return ret

>>> h=restartable(g)
>>> for i in h:

if not i: break
print("Using: ",i)
>>> h.restart()
>>> for i in h:

if not i: break
print("Using: ",i)

Complicated, but what this does is returns a value from its saved list
if there is one, otherwise returns a value from the original
generator. It can be restarted as many times as necessary, and any
time you read "past the end" of where you've read so far, the original
generator will be called upon.

Actually, this model might be useful for a repeatable random-number
generator. But those are more efficiently restarted by means of
reseeding the PRNG.

ChrisA

Chris Angelico, Oct 21, 2011
10. Yingjie LanGuest

----- Original Message -----

> From: Paul Rudin <>
>
> I'm not really sure whether you intend g to yield the originalvalues
> after your "revive" or new values based on the new value of vo.  But
> still you can make a class that supports the iterator protocol and does
> whatever you want (but you can't use the generator expression syntax).
>
> If you want something along these lines you should probably read up on
> the .send() part of the generator protocol.
>
> As an aside you shouldn't really write code that uses a global in that
> way... it'll end up biting you eventually.
>
> Anyway... we can speculate endlessly about non-existent language
> constructs, but I think we've probably done this one to death.
> --

Maybe no new language construct is needed:
just define that x.send() revives a generator.

Yingjie

Yingjie Lan, Oct 21, 2011
11. Yingjie LanGuest

----- Original Message -----
> From: Chris Angelico <>
> To:
> Cc:
> Sent: Friday, October21, 2011 4:27 PM
> Subject: Re: revive a generator
>
> On Fri, Oct 21, 2011 at 7:02 PM, Yingjie Lan <> wrote:
>> What if the generator involves a variable from another scope,
>> and before re-generating, the variable changed its value.
>> Also, the generator could be passed in as an argument,
>> so that we don't know its exact expression.
>>

>
> There's actually no way to know that the generator's even
> deterministic. Try this, for instance:
>
>>>> g=(input("Enter value %d or blank to stop: "%n) for n in

> range(1,11))
>>>> for s ing:

>     if not s: break
>     print("Processing input: "+s)
>
> It may not be particularly useful, but it's certainly legal. And this
> generator cannot viably be restarted.

Depends on what you want. If you want ten more inputs from user,
reviving this generator iscertainly a good thing to do.

> The only way is to cast it to
> list first, but that doesn't work when you have to stop reading
> expressions from the generator part way.
>
> What you could perhaps do is wrap the generator in something that
> saves its values:
>
>>>> class restartable(object):

>     def __init__(self,gen):
>         self.gen=gen
>         self.yielded=[]
>         self.iter=iter(self.yielded)
>     def restart(self):
>         self.iter=iter(self.yielded)
>     def__iter__(self):
>         return self
>     def __next__(self): # Remove the underscores for Python 2
>         try:
>             return self.iter.__next__()
>         except StopIteration:
>             pass
>         ret=self.gen.__next__()
>         self.yielded.append(ret)
>         return ret
>
>>>>h=restartable(g)
>>>> for i in h:

>     if not i: break
>     print("Using: ",i)
>>>> h.restart()
>>>> for i in h:

>     if not i: break
>     print("Using: ",i)
>
> Complicated, but what this does is returns a value from its saved list
> if there is one, otherwise returns a value from the original
> generator. It canbe restarted as many times as necessary, and any
> time you read "past the end" of where you've read so far, the
> original
> generator will be called upon.
>
> Actually, this model might be useful for a repeatable random-number
> generator. But those are more efficiently restarted bymeans of
> reseeding the PRNG.
>

Sure. Or you would like to have the next few random numbers with
the same PRNG.

These two cases seem to be strong use cases for reviving a generator.

Yingjie

Yingjie Lan, Oct 21, 2011
12. Dave AngelGuest

On 10/20/2011 10:09 PM, Yingjie Lan wrote:
> <snip>
>
> What if the generator is passed in as an argument
> when you are writing a function? That is, the expression
> is not available?
>
> Secondly, it would be nice to automatically revive it.
> For example, when another for-statement or other
> equivalent is applied to it.
>
> Yingjie

That last point would definitely be incompatible. It's quite useful to
be able to peel some values off a generator, then use a for loop to get
the remainder. Simplest example I can think of would be to read a file
where the first line(s) represent header information, and the body is
obtainable with a simple loop. I wouldn't be surprised if the csv
module does that.

Not arguing whether an explicit revival is useful or not. Although as
others have pointed out, not all generators could accomplish it, and in
some cases not unambiguously.

--

DaveA

Dave Angel, Oct 21, 2011
13. Steven D'ApranoGuest

On Thu, 20 Oct 2011 19:09:42 -0700, Yingjie Lan wrote:

>> Here's an example of an explicit request to revive the generator:

>
>
>>>>> g = (x*x for x in range(3))
>>>>> for x in g: print x

>> 0
>> 1
>> 4
>>>>> g = (x*x for x in range(3)) # revive the generator for x in g:
>>>>> print x #now this will work

>> 0
>> 1
>> 4
>>
>> ChrisA

>
>
> What if the generator is passed in as an argument when you are writing a
> function? That is, the expression is not available?

If the expression is not available, how do you expect to revive it? The
expression is gone, it no longer exists. As you said in another post:

"What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value."

Exactly. In general, you *can't* revive general iterators. It simply
isn't possible. The variables that defined it might be gone. They might
be non-deterministic: random numbers, data from the Internet or a file
system that has changed, or user input. Trying to enforce the rule
"iterators must support restarting" is foolish: it can't be done. You use
an iterator when you want to iterate over something *once*, that is why
they exist. If you want to iterate over it twice, don't use an iterator,
use a sequence.

Forcing all iterators to save their data, on the off-chance that maybe
somebody might want to iterate over it twice, defeats the purpose of an
iterator.

> Secondly, it would be nice to automatically revive it. For example, when
> another for-statement or other equivalent is applied to it.

Nice? No, it would be horrible. It goes against the basic concept of an
iterator: iterators should be memory efficient, generating values lazily.
If you want an iterable sequence that you can iterate over multiple
times, then use a list, or a custom iterable class.

If you want a socket wrench, use a socket wrench. Don't insist that
hammers have to have a socket wrench attachment.

--
Steven

Steven D'Aprano, Oct 21, 2011
14. Ian KellyGuest

On Fri, Oct 21, 2011 at 2:02 AM, Yingjie Lan <> wrote:
> Oops, my former reply has the code indentation messed up
> by the mail system. Here is a reformatted one:
>
>
> What if the generator involves a variable from another scope,
> and before re-generating, the variable changed its value.
> Also, the generator could be passed in as an argument,
> so that we don't know its exact expression.

In the former case, use a named generator function and call it twice
to create two generators. In the latter case, don't pass in the
generator as an argument. Pass in a callable that constructs the

vo = 34
def mygen():
for x in range(3):
yield vo * x

def myfun(g):
global vo
for i in g(): print(i)
vo += 3
for i in g(): print(i)

myfun(mygen)

Cheers,
Ian

Ian Kelly, Oct 21, 2011
15. Terry ReedyGuest

Here is a class that creates a re-iterable from any callable, such as a
generator function, that returns an iterator when called, + captured
arguments to be given to the function.

class reiterable():
def __init__(self, itercall, *args, **kwds):
self.f = itercall # callable that returns an iterator
self.args = args
self.kwds = kwds
def __iter__(self):
return self.f(*self.args, **self.kwds)

def squares(n):
for i in range(n):
yield i*i

sq3 = reiterable(squares, 3)

for i in sq3: print(i)
for i in sq3: print(i)
>>>

0
1
4
0
1
4

--
Terry Jan Reedy

Terry Reedy, Oct 21, 2011
16. Steven D'ApranoGuest

On Fri, 21 Oct 2011 16:25:47 -0400, Terry Reedy wrote:

> Here is a class that creates a re-iterable from any callable, such as a
> generator function, that returns an iterator when called, + captured
> arguments to be given to the function.
>
> class reiterable():
> def __init__(self, itercall, *args, **kwds):
> self.f = itercall # callable that returns an iterator
> self.args = args
> self.kwds = kwds
> def __iter__(self):
> return self.f(*self.args, **self.kwds)
>
> def squares(n):
> for i in range(n):
> yield i*i
>
> sq3 = reiterable(squares, 3)

We can do that even more simply, using a slightly different interface.

>>> from functools import partial
>>> sq3 = partial(squares, 3)

sq3 is now an iterator factory. You can't iterate over it directly, but
it's easy to restart: just call it to return a fresh iterator.

>>> list(sq3())

[0, 1, 4]
>>> list(sq3())

[0, 1, 4]

--
Steven

Steven D'Aprano, Oct 22, 2011
17. Carl BanksGuest

On Thursday, October 20, 2011 6:23:50 AM UTC-7, Yingjie Lan wrote:
> Hi,
>
> it seems a generator expression can be used only once:
>
> >>> g = (x*x for x in range(3))
> >>> for x in g: print x

> 0
> 1
> 4
> >>> for x in g: print x #nothing printed
> >>>

>
> Is there any way to revive g here?

Revive is the wrong word for what you want. Once an iterator (be it a generator or some other kind of iterator) is done, it's done. What you are asking for is, given a generator, to create a new generator from the same expression/function that created the original generator. This is not reviving,but recreating.

I have two objections to this: a major ideological one and a minor practical one.

The practical drawback to allowing generators to be recreated is that it forces all generators to carry around a reference to the code object that created it.

if random.random() > 5:
g = (x*x for x in xrange(3))
else:
g = (x+x for x in xrange(3))
for y in g:
print x
revive(g) # which generator expression was it?
# need to carry around a reference to be able to tell
for y in g:
print x

Carrying a reference to a code object in turn carries around any closures used in the generator expression or function, so it can potentially keep a large amount of data alive. Given that the vast majority of generators would never be recreated, this is quite wasteful.

My ideological objection is that it forces the programmer to be wary of theeffects of recreation. Right now, if someone writes a generator expression, they can rely on the fact that it can only be iterated through once (pertime the generator expression is evaluated). But if you allow a downstream user to recreate the generator at will, then the writer will always have to be wary of adverse side-effects if the generator is iterated through twice.

So, although I can see it being occasionally useful, I'm going to opine that it is more trouble than it's worth.

Carl Banks

Carl Banks, Oct 22, 2011
18. alex23Guest

On Oct 21, 11:46 am, Yingjie Lan <> wrote:
> I am still not sure why should we enforce that
> a generator can not be reused after an explicit
> request to revive it?

No one is "enforcing" anything, you're simply resisting implementing
this yourself. Consider the following generator:

import random
def randgen():
for _ in xrange(10):
yield random.choice(['Hi','Lo'])

>>> [x for x in randgen()]

['Hi', 'Hi', 'Lo', 'Hi', 'Lo', 'Lo', 'Lo', 'Lo', 'Hi', 'Hi']

What would you expect when you reset that generator? A newly
randomised set of values, or the _same_ set of values?

What if the generator is reading from an external source, like
temperature values? Once revived, should it return the exact same
sequence it originally did, or should it retrieve new values?

Now, no matter which decision you made, why is your expectation of
behaviour the right one? Why should the generator protocol support

If you need your generator to be re-usable, make a factory that
creates it.

alex23, Oct 25, 2011
19. alex23Guest

On Oct 21, 12:09 pm, Yingjie Lan <> wrote:
> Secondly, it would be nice to automatically revive it.

Sure, it's always nice when your expectation of a language feature
exactly matches with its capabilities.

When it doesn't, you suck it up and code around it.

Because at the very least it's a hell of a lot quicker than waiting
for the language to change for you.

alex23, Oct 25, 2011