revive a generator

P

Paul Rudin

Yingjie Lan said:
Hi,

it seems a generator expression can be used only once:


Is there any way to revive g here?

Generators are like that - you consume them until they run out of
values. You could have done [x*x for x in range(3)] and then iterated
over that list as many times as you wanted.

A generator doesn't have to remember all the values it generates so it
can be more memory efficient that a list. Also it can, for example,
generate an infinite sequence.
 
Y

Yingjie Lan

----- Original Message -----
From: Paul Rudin <[email protected]>
To: (e-mail address removed)
Cc:
Sent: Thursday, October 20, 2011 10:28 PM
Subject: Re: revive a generator

Yingjie Lan said:
Hi,

it seems a generator expression can be used only once:


Is there any way torevive g here?

Generators are like that - you consume them until they run out of
values. You could have done [x*x for x in range(3)] and then iterated
over that list as many times as you wanted.

A generator doesn't have to remember all the values it generates so it
can be more memory efficient that a list. Also it can, for example,
generate an infinite sequence.
Thanks a lot to all who answered my question. 
I am still not sure why should we enforce that 
a generator can not be reused after an explicit 
request to revive it?
 
C

Chris Angelico

Thanks a lot to all who answered my question.
I am still not sure why should we enforce that
a generator can not be reused after an explicit
request to revive it?

Here's an example of an explicit request to revive the generator:
0
1
4

ChrisA
 
P

Paul Rudin

Yingjie Lan said:
----- Original Message -----
From: Paul Rudin <[email protected]>
Generators are like that - you consume them until they run out of
values. You could have done [x*x for x in range(3)] and then iterated
over that list as many times as you wanted.

A generator doesn't have to remember all the values it generates so it
can be more memory efficient that a list. Also it can, for example,
generate an infinite sequence.
Thanks a lot to all who answered my question. 
I am still not sure why should we enforce that 
a generator can not be reused after an explicit 
request to revive it?

The language has no explicit notion of a request to "revive" a
generator. You could use the same syntax to make a new generator that
yeilds the same values as the one you started with if that's what you
want.

As we've already discussed if you want to iterate several times over the
same values then it probably makes sense to compute them and store them
in e.g. a list (although there are always trade-offs between storage use
and the cost of computing things again).
 
Y

Yingjie Lan

----- Original Message -----
From: Paul Rudin <[email protected]>
To: (e-mail address removed)
Cc:
Sent: Friday, October 21, 2011 3:27 PM
Subject: Re: revive a generator


The language has no explicit notion of a request to "revive" a
generator. You could use the same syntax to make a new generator that
yeilds the same valuesas the one you started with if that's what you
want.

As we've already discussed if you want to iterate several times over the
same values then it probably makes sense to compute them and store them
in e..g. a list (although there are always trade-offs between storage use
and the cost of computing things again).


What if the generator involves a variable from another scope,
and before re-generating, thevariable changed its value.
Also, the generator could be passed in as anargument,
so that we don't know its exact expression.
for i in g: print(i)
vo  += 3
revive(g) #best if revived automatically
for i in g: print(i)
myfun(g)

Yingjie
 
Y

Yingjie Lan

----- Original Message -----
From: Paul Rudin <[email protected]>
The language has no explicit notion of a request to "revive" a
generator. You could use the same syntax to make a new generator that
yeilds the same values as the one you started with if that's what you
want.

As we've already discussed if you want to iterate several times over the
same values then it probably makes sense to compute them and store them
in e.g. a list (although there are always trade-offs between storage use
and the cost of computing things again).

Oops,my former reply has the code indentation messed up 
by the mail system.. Here is a reformatted one:


What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value.
Also, the generator could be passed in as an argument,
so thatwe don't know its exact expression.
            for i in g: print(i)
            vo  += 3
            revive(g) #best if revived automatically
            for i in g: print(i)

Yingjie
 
P

Paul Rudin

Yingjie Lan said:
What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value.
Also, the generator could be passed in as an argument,
so that we don't know its exact expression.

            for i in g: print(i)
            vo  += 3
            revive(g) #best if revived automatically
            for i in g: print(i)


I'm not really sure whether you intend g to yield the original values
after your "revive" or new values based on the new value of vo. But
still you can make a class that supports the iterator protocol and does
whatever you want (but you can't use the generator expression syntax).

If you want something along these lines you should probably read up on
the .send() part of the generator protocol.

As an aside you shouldn't really write code that uses a global in that
way.. it'll end up biting you eventually.

Anyway... we can speculate endlessly about non-existent language
constructs, but I think we've probably done this one to death.
 
C

Chris Angelico

What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value.
Also, the generator could be passed in as an argument,
so that we don't know its exact expression.

There's actually no way to know that the generator's even
deterministic. Try this, for instance:
if not s: break
print("Processing input: "+s)

It may not be particularly useful, but it's certainly legal. And this
generator cannot viably be restarted. The only way is to cast it to
list first, but that doesn't work when you have to stop reading
expressions from the generator part way.

What you could perhaps do is wrap the generator in something that
saves its values:
def __init__(self,gen):
self.gen=gen
self.yielded=[]
self.iter=iter(self.yielded)
def restart(self):
self.iter=iter(self.yielded)
def __iter__(self):
return self
def __next__(self): # Remove the underscores for Python 2
try:
return self.iter.__next__()
except StopIteration:
pass
ret=self.gen.__next__()
self.yielded.append(ret)
return ret
if not i: break
print("Using: ",i)if not i: break
print("Using: ",i)

Complicated, but what this does is returns a value from its saved list
if there is one, otherwise returns a value from the original
generator. It can be restarted as many times as necessary, and any
time you read "past the end" of where you've read so far, the original
generator will be called upon.

Actually, this model might be useful for a repeatable random-number
generator. But those are more efficiently restarted by means of
reseeding the PRNG.

ChrisA
 
Y

Yingjie Lan

----- Original Message -----
From: Paul Rudin <[email protected]>

I'm not really sure whether you intend g to yield the originalvalues
after your "revive" or new values based on the new value of vo.  But
still you can make a class that supports the iterator protocol and does
whatever you want (but you can't use the generator expression syntax).

If you want something along these lines you should probably read up on
the .send() part of the generator protocol.

As an aside you shouldn't really write code that uses a global in that
way... it'll end up biting you eventually.

Anyway... we can speculate endlessly about non-existent language
constructs, but I think we've probably done this one to death.
--


Maybe no new language construct is needed:
just define that x.send() revives a generator.

Yingjie
 
Y

Yingjie Lan

----- Original Message -----
From: Chris Angelico <[email protected]>
To: (e-mail address removed)
Cc:
Sent: Friday, October21, 2011 4:27 PM
Subject: Re: revive a generator



There's actually no way to know that the generator's even
deterministic. Try this, for instance:

    if not s: break
    print("Processing input: "+s)

It may not be particularly useful, but it's certainly legal. And this
generator cannot viably be restarted. 

Depends on what you want. If you want ten more inputs from user,
reviving this generator iscertainly a good thing to do.
The only way is to cast it to
list first, but that doesn't work when you have to stop reading
expressions from the generator part way.

What you could perhaps do is wrap the generator in something that
saves its values:
    def __init__(self,gen):
        self.gen=gen
        self.yielded=[]
        self.iter=iter(self.yielded)
    def restart(self):
        self.iter=iter(self.yielded)
    def__iter__(self):
        return self
    def __next__(self): # Remove the underscores for Python 2
        try:
            return self.iter.__next__()
        except StopIteration:
            pass
        ret=self.gen.__next__()
        self.yielded.append(ret)
        return ret
    if not i: break
    print("Using: ",i)    if not i: break
    print("Using: ",i)

Complicated, but what this does is returns a value from its saved list
if there is one, otherwise returns a value from the original
generator. It canbe restarted as many times as necessary, and any
time you read "past the end" of where you've read so far, the
original
generator will be called upon.

Actually, this model might be useful for a repeatable random-number
generator. But those are more efficiently restarted bymeans of
reseeding the PRNG.


Sure. Or you would like to have the next few random numbers with 
the same PRNG. 

These two cases seem to be strong use cases for reviving a generator.

Yingjie
 
D

Dave Angel

<snip>

What if the generator is passed in as an argument
when you are writing a function? That is, the expression
is not available?

Secondly, it would be nice to automatically revive it.
For example, when another for-statement or other
equivalent is applied to it.

Yingjie
That last point would definitely be incompatible. It's quite useful to
be able to peel some values off a generator, then use a for loop to get
the remainder. Simplest example I can think of would be to read a file
where the first line(s) represent header information, and the body is
obtainable with a simple loop. I wouldn't be surprised if the csv
module does that.

Not arguing whether an explicit revival is useful or not. Although as
others have pointed out, not all generators could accomplish it, and in
some cases not unambiguously.
 
S

Steven D'Aprano

What if the generator is passed in as an argument when you are writing a
function? That is, the expression is not available?

If the expression is not available, how do you expect to revive it? The
expression is gone, it no longer exists. As you said in another post:

"What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value."

Exactly. In general, you *can't* revive general iterators. It simply
isn't possible. The variables that defined it might be gone. They might
be non-deterministic: random numbers, data from the Internet or a file
system that has changed, or user input. Trying to enforce the rule
"iterators must support restarting" is foolish: it can't be done. You use
an iterator when you want to iterate over something *once*, that is why
they exist. If you want to iterate over it twice, don't use an iterator,
use a sequence.

Forcing all iterators to save their data, on the off-chance that maybe
somebody might want to iterate over it twice, defeats the purpose of an
iterator.

Secondly, it would be nice to automatically revive it. For example, when
another for-statement or other equivalent is applied to it.

Nice? No, it would be horrible. It goes against the basic concept of an
iterator: iterators should be memory efficient, generating values lazily.
If you want an iterable sequence that you can iterate over multiple
times, then use a list, or a custom iterable class.

If you want a socket wrench, use a socket wrench. Don't insist that
hammers have to have a socket wrench attachment.
 
I

Ian Kelly

Oops, my former reply has the code indentation messed up
by the mail system. Here is a reformatted one:


What if the generator involves a variable from another scope,
and before re-generating, the variable changed its value.
Also, the generator could be passed in as an argument,
so that we don't know its exact expression.

In the former case, use a named generator function and call it twice
to create two generators. In the latter case, don't pass in the
generator as an argument. Pass in a callable that constructs the
iterator instead. Modifying your example:

vo = 34
def mygen():
for x in range(3):
yield vo * x

def myfun(g):
global vo
for i in g(): print(i)
vo += 3
for i in g(): print(i)

myfun(mygen)

Cheers,
Ian
 
T

Terry Reedy

Here is a class that creates a re-iterable from any callable, such as a
generator function, that returns an iterator when called, + captured
arguments to be given to the function.

class reiterable():
def __init__(self, itercall, *args, **kwds):
self.f = itercall # callable that returns an iterator
self.args = args
self.kwds = kwds
def __iter__(self):
return self.f(*self.args, **self.kwds)

def squares(n):
for i in range(n):
yield i*i

sq3 = reiterable(squares, 3)

for i in sq3: print(i)
for i in sq3: print(i)0
1
4
0
1
4
 
S

Steven D'Aprano

Here is a class that creates a re-iterable from any callable, such as a
generator function, that returns an iterator when called, + captured
arguments to be given to the function.

class reiterable():
def __init__(self, itercall, *args, **kwds):
self.f = itercall # callable that returns an iterator
self.args = args
self.kwds = kwds
def __iter__(self):
return self.f(*self.args, **self.kwds)

def squares(n):
for i in range(n):
yield i*i

sq3 = reiterable(squares, 3)


We can do that even more simply, using a slightly different interface.

sq3 is now an iterator factory. You can't iterate over it directly, but
it's easy to restart: just call it to return a fresh iterator.
list(sq3()) [0, 1, 4]
list(sq3())
[0, 1, 4]
 
C

Carl Banks

Hi,

it seems a generator expression can be used only once:


Is there any way to revive g here?

Revive is the wrong word for what you want. Once an iterator (be it a generator or some other kind of iterator) is done, it's done. What you are asking for is, given a generator, to create a new generator from the same expression/function that created the original generator. This is not reviving,but recreating.

I have two objections to this: a major ideological one and a minor practical one.

The practical drawback to allowing generators to be recreated is that it forces all generators to carry around a reference to the code object that created it.

if random.random() > 5:
g = (x*x for x in xrange(3))
else:
g = (x+x for x in xrange(3))
for y in g:
print x
revive(g) # which generator expression was it?
# need to carry around a reference to be able to tell
for y in g:
print x

Carrying a reference to a code object in turn carries around any closures used in the generator expression or function, so it can potentially keep a large amount of data alive. Given that the vast majority of generators would never be recreated, this is quite wasteful.

My ideological objection is that it forces the programmer to be wary of theeffects of recreation. Right now, if someone writes a generator expression, they can rely on the fact that it can only be iterated through once (pertime the generator expression is evaluated). But if you allow a downstream user to recreate the generator at will, then the writer will always have to be wary of adverse side-effects if the generator is iterated through twice.

So, although I can see it being occasionally useful, I'm going to opine that it is more trouble than it's worth.


Carl Banks
 
A

alex23

I am still not sure why should we enforce that 
a generator can not be reused after an explicit 
request to revive it?

No one is "enforcing" anything, you're simply resisting implementing
this yourself. Consider the following generator:

import random
def randgen():
for _ in xrange(10):
yield random.choice(['Hi','Lo'])
['Hi', 'Hi', 'Lo', 'Hi', 'Lo', 'Lo', 'Lo', 'Lo', 'Hi', 'Hi']

What would you expect when you reset that generator? A newly
randomised set of values, or the _same_ set of values?

What if the generator is reading from an external source, like
temperature values? Once revived, should it return the exact same
sequence it originally did, or should it retrieve new values?

Now, no matter which decision you made, why is your expectation of
behaviour the right one? Why should the generator protocol support
your convience in particular?

If you need your generator to be re-usable, make a factory that
creates it.
 
A

alex23

Secondly, it would be nice to automatically revive it.

Sure, it's always nice when your expectation of a language feature
exactly matches with its capabilities.

When it doesn't, you suck it up and code around it.

Because at the very least it's a hell of a lot quicker than waiting
for the language to change for you.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top