revive a generator

Discussion in 'Python' started by Yingjie Lan, Oct 20, 2011.

  1. Yingjie Lan

    Yingjie Lan Guest

    Hi,

    it seems a generator expression can be used only once:

    >>> g = (x*x for x in range(3))
    >>> for x in g: print x

    0
    1
    4
    >>> for x in g: print x #nothing printed
    >>>


    Is there any way to revive g here?

    Yingjie
    Yingjie Lan, Oct 20, 2011
    #1
    1. Advertising

  2. Yingjie Lan

    Paul Rudin Guest

    Yingjie Lan <> writes:

    > Hi,
    >
    > it seems a generator expression can be used only once:
    >
    >>>> g = (x*x for x in range(3))
    >>>> for x in g: print x

    > 0
    > 1
    > 4
    >>>> for x in g: print x #nothing printed
    >>>>

    >
    > Is there any way to revive g here?
    >


    Generators are like that - you consume them until they run out of
    values. You could have done [x*x for x in range(3)] and then iterated
    over that list as many times as you wanted.

    A generator doesn't have to remember all the values it generates so it
    can be more memory efficient that a list. Also it can, for example,
    generate an infinite sequence.
    Paul Rudin, Oct 20, 2011
    #2
    1. Advertising

  3. Yingjie Lan

    Yingjie Lan Guest

    ----- Original Message -----
    > From: Paul Rudin <>
    > To:
    > Cc:
    > Sent: Thursday, October 20, 2011 10:28 PM
    > Subject: Re: revive a generator
    >
    > Yingjie Lan <> writes:
    >
    >> Hi,
    >>
    >> it seems a generator expression can be used only once:
    >>
    >>>>> g = (x*x for x inrange(3))
    >>>>> for x in g: print x

    >> 0
    >> 1
    >> 4
    >>>>> for x in g: print x #nothing printed
    >>>>>

    >>
    >> Is there any way torevive g here?
    >>

    >
    > Generators are like that - you consume them until they run out of
    > values. You could have done [x*x for x in range(3)] and then iterated
    > over that list as many times as you wanted.
    >
    > A generator doesn't have to remember all the values it generates so it
    > can be more memory efficient that a list. Also it can, for example,
    > generate an infinite sequence.
    >
    >

    Thanks a lot to all who answered my question. 
    I am still not sure why should we enforce that 
    a generator can not be reused after an explicit 
    request to revive it?
    Yingjie Lan, Oct 21, 2011
    #3
  4. On Fri, Oct 21, 2011 at 12:46 PM, Yingjie Lan <> wrote:
    >
    > Thanks a lot to all who answered my question.
    > I am still not sure why should we enforce that
    > a generator can not be reused after an explicit
    > request to revive it?


    Here's an example of an explicit request to revive the generator:

    >>> g = (x*x for x in range(3))
    >>> for x in g: print x

    0
    1
    4
    >>> g = (x*x for x in range(3)) # revive the generator
    >>> for x in g: print x #now this will work

    0
    1
    4

    ChrisA
    Chris Angelico, Oct 21, 2011
    #4
  5. Yingjie Lan

    Paul Rudin Guest

    Yingjie Lan <> writes:

    > ----- Original Message -----
    >> From: Paul Rudin <>


    >> Generators are like that - you consume them until they run out of
    >> values. You could have done [x*x for x in range(3)] and then iterated
    >> over that list as many times as you wanted.
    >>
    >> A generator doesn't have to remember all the values it generates so it
    >> can be more memory efficient that a list. Also it can, for example,
    >> generate an infinite sequence.
    >>
    >>

    > Thanks a lot to all who answered my question. 
    > I am still not sure why should we enforce that 
    > a generator can not be reused after an explicit 
    > request to revive it?


    The language has no explicit notion of a request to "revive" a
    generator. You could use the same syntax to make a new generator that
    yeilds the same values as the one you started with if that's what you
    want.

    As we've already discussed if you want to iterate several times over the
    same values then it probably makes sense to compute them and store them
    in e.g. a list (although there are always trade-offs between storage use
    and the cost of computing things again).
    Paul Rudin, Oct 21, 2011
    #5
  6. Yingjie Lan

    Yingjie Lan Guest

    ----- Original Message -----

    > From: Paul Rudin <>
    > To:
    > Cc:
    > Sent: Friday, October 21, 2011 3:27 PM
    > Subject: Re: revive a generator
    >
    >
    > The language has no explicit notion of a request to "revive" a
    > generator. You could use the same syntax to make a new generator that
    > yeilds the same valuesas the one you started with if that's what you
    > want.
    >
    > As we've already discussed if you want to iterate several times over the
    > same values then it probably makes sense to compute them and store them
    > in e..g. a list (although there are always trade-offs between storage use
    > and the cost of computing things again).
    >
    >



    What if the generator involves a variable from another scope,
    and before re-generating, thevariable changed its value.
    Also, the generator could be passed in as anargument,
    so that we don't know its exact expression.

    >>> vo = 34
    >>> g = (vo*x for x in range(3))
    >>> def myfun(g):

    for i in g: print(i)
    vo  += 3
    revive(g) #best if revived automatically
    for i in g: print(i)
    myfun(g)

    Yingjie
    Yingjie Lan, Oct 21, 2011
    #6
  7. Yingjie Lan

    Yingjie Lan Guest

    ----- Original Message -----

    > From: Paul Rudin <>
    > The language has no explicit notion of a request to "revive" a
    > generator. You could use the same syntax to make a new generator that
    > yeilds the same values as the one you started with if that's what you
    > want.
    >
    > As we've already discussed if you want to iterate several times over the
    > same values then it probably makes sense to compute them and store them
    > in e.g. a list (although there are always trade-offs between storage use
    > and the cost of computing things again).
    >


    Oops,my former reply has the code indentation messed up 
    by the mail system.. Here is a reformatted one:


    What if the generator involves a variable from another scope,
    and before re-generating, the variable changed its value.
    Also, the generator could be passed in as an argument,
    so thatwe don't know its exact expression.

    >>> vo = 34
    >>> g = (vo*xfor x in range(3))
    >>> def myfun(g):

                for i in g: print(i)
                vo  += 3
                revive(g) #best if revived automatically
                for i in g: print(i)
    >>> myfun(g)



    Yingjie
    Yingjie Lan, Oct 21, 2011
    #7
  8. Yingjie Lan

    Paul Rudin Guest

    Yingjie Lan <> writes:

    >
    >
    > What if the generator involves a variable from another scope,
    > and before re-generating, the variable changed its value.
    > Also, the generator could be passed in as an argument,
    > so that we don't know its exact expression.
    >
    >>>> vo = 34
    >>>> g = (vo*x for x in range(3))
    >>>> def myfun(g):

    >             for i in g: print(i)
    >             vo  += 3
    >             revive(g) #best if revived automatically
    >             for i in g: print(i)
    >>>> myfun(g)

    >
    >



    I'm not really sure whether you intend g to yield the original values
    after your "revive" or new values based on the new value of vo. But
    still you can make a class that supports the iterator protocol and does
    whatever you want (but you can't use the generator expression syntax).

    If you want something along these lines you should probably read up on
    the .send() part of the generator protocol.

    As an aside you shouldn't really write code that uses a global in that
    way.. it'll end up biting you eventually.

    Anyway... we can speculate endlessly about non-existent language
    constructs, but I think we've probably done this one to death.
    Paul Rudin, Oct 21, 2011
    #8
  9. On Fri, Oct 21, 2011 at 7:02 PM, Yingjie Lan <> wrote:
    > What if the generator involves a variable from another scope,
    > and before re-generating, the variable changed its value.
    > Also, the generator could be passed in as an argument,
    > so that we don't know its exact expression.
    >


    There's actually no way to know that the generator's even
    deterministic. Try this, for instance:

    >>> g=(input("Enter value %d or blank to stop: "%n) for n in range(1,11))
    >>> for s in g:

    if not s: break
    print("Processing input: "+s)

    It may not be particularly useful, but it's certainly legal. And this
    generator cannot viably be restarted. The only way is to cast it to
    list first, but that doesn't work when you have to stop reading
    expressions from the generator part way.

    What you could perhaps do is wrap the generator in something that
    saves its values:

    >>> class restartable(object):

    def __init__(self,gen):
    self.gen=gen
    self.yielded=[]
    self.iter=iter(self.yielded)
    def restart(self):
    self.iter=iter(self.yielded)
    def __iter__(self):
    return self
    def __next__(self): # Remove the underscores for Python 2
    try:
    return self.iter.__next__()
    except StopIteration:
    pass
    ret=self.gen.__next__()
    self.yielded.append(ret)
    return ret

    >>> h=restartable(g)
    >>> for i in h:

    if not i: break
    print("Using: ",i)
    >>> h.restart()
    >>> for i in h:

    if not i: break
    print("Using: ",i)

    Complicated, but what this does is returns a value from its saved list
    if there is one, otherwise returns a value from the original
    generator. It can be restarted as many times as necessary, and any
    time you read "past the end" of where you've read so far, the original
    generator will be called upon.

    Actually, this model might be useful for a repeatable random-number
    generator. But those are more efficiently restarted by means of
    reseeding the PRNG.

    ChrisA
    Chris Angelico, Oct 21, 2011
    #9
  10. Yingjie Lan

    Yingjie Lan Guest

    ----- Original Message -----

    > From: Paul Rudin <>
    >
    > I'm not really sure whether you intend g to yield the originalvalues
    > after your "revive" or new values based on the new value of vo.  But
    > still you can make a class that supports the iterator protocol and does
    > whatever you want (but you can't use the generator expression syntax).
    >
    > If you want something along these lines you should probably read up on
    > the .send() part of the generator protocol.
    >
    > As an aside you shouldn't really write code that uses a global in that
    > way... it'll end up biting you eventually.
    >
    > Anyway... we can speculate endlessly about non-existent language
    > constructs, but I think we've probably done this one to death.
    > --



    Maybe no new language construct is needed:
    just define that x.send() revives a generator.

    Yingjie
    Yingjie Lan, Oct 21, 2011
    #10
  11. Yingjie Lan

    Yingjie Lan Guest

    ----- Original Message -----
    > From: Chris Angelico <>
    > To:
    > Cc:
    > Sent: Friday, October21, 2011 4:27 PM
    > Subject: Re: revive a generator
    >
    > On Fri, Oct 21, 2011 at 7:02 PM, Yingjie Lan <> wrote:
    >> What if the generator involves a variable from another scope,
    >> and before re-generating, the variable changed its value.
    >> Also, the generator could be passed in as an argument,
    >> so that we don't know its exact expression.
    >>

    >
    > There's actually no way to know that the generator's even
    > deterministic. Try this, for instance:
    >
    >>>> g=(input("Enter value %d or blank to stop: "%n) for n in

    > range(1,11))
    >>>> for s ing:

    >     if not s: break
    >     print("Processing input: "+s)
    >
    > It may not be particularly useful, but it's certainly legal. And this
    > generator cannot viably be restarted. 


    Depends on what you want. If you want ten more inputs from user,
    reviving this generator iscertainly a good thing to do.

    > The only way is to cast it to
    > list first, but that doesn't work when you have to stop reading
    > expressions from the generator part way.
    >
    > What you could perhaps do is wrap the generator in something that
    > saves its values:
    >
    >>>> class restartable(object):

    >     def __init__(self,gen):
    >         self.gen=gen
    >         self.yielded=[]
    >         self.iter=iter(self.yielded)
    >     def restart(self):
    >         self.iter=iter(self.yielded)
    >     def__iter__(self):
    >         return self
    >     def __next__(self): # Remove the underscores for Python 2
    >         try:
    >             return self.iter.__next__()
    >         except StopIteration:
    >             pass
    >         ret=self.gen.__next__()
    >         self.yielded.append(ret)
    >         return ret
    >
    >>>>h=restartable(g)
    >>>> for i in h:

    >     if not i: break
    >     print("Using: ",i)
    >>>> h.restart()
    >>>> for i in h:

    >     if not i: break
    >     print("Using: ",i)
    >
    > Complicated, but what this does is returns a value from its saved list
    > if there is one, otherwise returns a value from the original
    > generator. It canbe restarted as many times as necessary, and any
    > time you read "past the end" of where you've read so far, the
    > original
    > generator will be called upon.
    >
    > Actually, this model might be useful for a repeatable random-number
    > generator. But those are more efficiently restarted bymeans of
    > reseeding the PRNG.
    >



    Sure. Or you would like to have the next few random numbers with 
    the same PRNG. 

    These two cases seem to be strong use cases for reviving a generator.

    Yingjie
    Yingjie Lan, Oct 21, 2011
    #11
  12. Yingjie Lan

    Dave Angel Guest

    On 10/20/2011 10:09 PM, Yingjie Lan wrote:
    > <snip>
    >
    > What if the generator is passed in as an argument
    > when you are writing a function? That is, the expression
    > is not available?
    >
    > Secondly, it would be nice to automatically revive it.
    > For example, when another for-statement or other
    > equivalent is applied to it.
    >
    > Yingjie

    That last point would definitely be incompatible. It's quite useful to
    be able to peel some values off a generator, then use a for loop to get
    the remainder. Simplest example I can think of would be to read a file
    where the first line(s) represent header information, and the body is
    obtainable with a simple loop. I wouldn't be surprised if the csv
    module does that.

    Not arguing whether an explicit revival is useful or not. Although as
    others have pointed out, not all generators could accomplish it, and in
    some cases not unambiguously.


    --

    DaveA
    Dave Angel, Oct 21, 2011
    #12
  13. On Thu, 20 Oct 2011 19:09:42 -0700, Yingjie Lan wrote:

    >> Here's an example of an explicit request to revive the generator:

    >
    >
    >>>>> g = (x*x for x in range(3))
    >>>>> for x in g: print x

    >> 0
    >> 1
    >> 4
    >>>>> g = (x*x for x in range(3)) # revive the generator for x in g:
    >>>>> print x #now this will work

    >> 0
    >> 1
    >> 4
    >>
    >> ChrisA

    >
    >
    > What if the generator is passed in as an argument when you are writing a
    > function? That is, the expression is not available?


    If the expression is not available, how do you expect to revive it? The
    expression is gone, it no longer exists. As you said in another post:

    "What if the generator involves a variable from another scope,
    and before re-generating, the variable changed its value."

    Exactly. In general, you *can't* revive general iterators. It simply
    isn't possible. The variables that defined it might be gone. They might
    be non-deterministic: random numbers, data from the Internet or a file
    system that has changed, or user input. Trying to enforce the rule
    "iterators must support restarting" is foolish: it can't be done. You use
    an iterator when you want to iterate over something *once*, that is why
    they exist. If you want to iterate over it twice, don't use an iterator,
    use a sequence.

    Forcing all iterators to save their data, on the off-chance that maybe
    somebody might want to iterate over it twice, defeats the purpose of an
    iterator.


    > Secondly, it would be nice to automatically revive it. For example, when
    > another for-statement or other equivalent is applied to it.


    Nice? No, it would be horrible. It goes against the basic concept of an
    iterator: iterators should be memory efficient, generating values lazily.
    If you want an iterable sequence that you can iterate over multiple
    times, then use a list, or a custom iterable class.

    If you want a socket wrench, use a socket wrench. Don't insist that
    hammers have to have a socket wrench attachment.


    --
    Steven
    Steven D'Aprano, Oct 21, 2011
    #13
  14. Yingjie Lan

    Ian Kelly Guest

    On Fri, Oct 21, 2011 at 2:02 AM, Yingjie Lan <> wrote:
    > Oops, my former reply has the code indentation messed up
    > by the mail system. Here is a reformatted one:
    >
    >
    > What if the generator involves a variable from another scope,
    > and before re-generating, the variable changed its value.
    > Also, the generator could be passed in as an argument,
    > so that we don't know its exact expression.


    In the former case, use a named generator function and call it twice
    to create two generators. In the latter case, don't pass in the
    generator as an argument. Pass in a callable that constructs the
    iterator instead. Modifying your example:

    vo = 34
    def mygen():
    for x in range(3):
    yield vo * x

    def myfun(g):
    global vo
    for i in g(): print(i)
    vo += 3
    for i in g(): print(i)

    myfun(mygen)

    Cheers,
    Ian
    Ian Kelly, Oct 21, 2011
    #14
  15. Yingjie Lan

    Terry Reedy Guest

    Here is a class that creates a re-iterable from any callable, such as a
    generator function, that returns an iterator when called, + captured
    arguments to be given to the function.

    class reiterable():
    def __init__(self, itercall, *args, **kwds):
    self.f = itercall # callable that returns an iterator
    self.args = args
    self.kwds = kwds
    def __iter__(self):
    return self.f(*self.args, **self.kwds)

    def squares(n):
    for i in range(n):
    yield i*i

    sq3 = reiterable(squares, 3)

    for i in sq3: print(i)
    for i in sq3: print(i)
    >>>

    0
    1
    4
    0
    1
    4


    --
    Terry Jan Reedy
    Terry Reedy, Oct 21, 2011
    #15
  16. On Fri, 21 Oct 2011 16:25:47 -0400, Terry Reedy wrote:

    > Here is a class that creates a re-iterable from any callable, such as a
    > generator function, that returns an iterator when called, + captured
    > arguments to be given to the function.
    >
    > class reiterable():
    > def __init__(self, itercall, *args, **kwds):
    > self.f = itercall # callable that returns an iterator
    > self.args = args
    > self.kwds = kwds
    > def __iter__(self):
    > return self.f(*self.args, **self.kwds)
    >
    > def squares(n):
    > for i in range(n):
    > yield i*i
    >
    > sq3 = reiterable(squares, 3)



    We can do that even more simply, using a slightly different interface.

    >>> from functools import partial
    >>> sq3 = partial(squares, 3)


    sq3 is now an iterator factory. You can't iterate over it directly, but
    it's easy to restart: just call it to return a fresh iterator.

    >>> list(sq3())

    [0, 1, 4]
    >>> list(sq3())

    [0, 1, 4]


    --
    Steven
    Steven D'Aprano, Oct 22, 2011
    #16
  17. Yingjie Lan

    Carl Banks Guest

    On Thursday, October 20, 2011 6:23:50 AM UTC-7, Yingjie Lan wrote:
    > Hi,
    >
    > it seems a generator expression can be used only once:
    >
    > >>> g = (x*x for x in range(3))
    > >>> for x in g: print x

    > 0
    > 1
    > 4
    > >>> for x in g: print x #nothing printed
    > >>>

    >
    > Is there any way to revive g here?


    Revive is the wrong word for what you want. Once an iterator (be it a generator or some other kind of iterator) is done, it's done. What you are asking for is, given a generator, to create a new generator from the same expression/function that created the original generator. This is not reviving,but recreating.

    I have two objections to this: a major ideological one and a minor practical one.

    The practical drawback to allowing generators to be recreated is that it forces all generators to carry around a reference to the code object that created it.

    if random.random() > 5:
    g = (x*x for x in xrange(3))
    else:
    g = (x+x for x in xrange(3))
    for y in g:
    print x
    revive(g) # which generator expression was it?
    # need to carry around a reference to be able to tell
    for y in g:
    print x

    Carrying a reference to a code object in turn carries around any closures used in the generator expression or function, so it can potentially keep a large amount of data alive. Given that the vast majority of generators would never be recreated, this is quite wasteful.

    My ideological objection is that it forces the programmer to be wary of theeffects of recreation. Right now, if someone writes a generator expression, they can rely on the fact that it can only be iterated through once (pertime the generator expression is evaluated). But if you allow a downstream user to recreate the generator at will, then the writer will always have to be wary of adverse side-effects if the generator is iterated through twice.

    So, although I can see it being occasionally useful, I'm going to opine that it is more trouble than it's worth.


    Carl Banks
    Carl Banks, Oct 22, 2011
    #17
  18. Yingjie Lan

    alex23 Guest

    On Oct 21, 11:46 am, Yingjie Lan <> wrote:
    > I am still not sure why should we enforce that 
    > a generator can not be reused after an explicit 
    > request to revive it?


    No one is "enforcing" anything, you're simply resisting implementing
    this yourself. Consider the following generator:

    import random
    def randgen():
    for _ in xrange(10):
    yield random.choice(['Hi','Lo'])

    >>> [x for x in randgen()]

    ['Hi', 'Hi', 'Lo', 'Hi', 'Lo', 'Lo', 'Lo', 'Lo', 'Hi', 'Hi']

    What would you expect when you reset that generator? A newly
    randomised set of values, or the _same_ set of values?

    What if the generator is reading from an external source, like
    temperature values? Once revived, should it return the exact same
    sequence it originally did, or should it retrieve new values?

    Now, no matter which decision you made, why is your expectation of
    behaviour the right one? Why should the generator protocol support
    your convience in particular?

    If you need your generator to be re-usable, make a factory that
    creates it.
    alex23, Oct 25, 2011
    #18
  19. Yingjie Lan

    alex23 Guest

    On Oct 21, 12:09 pm, Yingjie Lan <> wrote:
    > Secondly, it would be nice to automatically revive it.


    Sure, it's always nice when your expectation of a language feature
    exactly matches with its capabilities.

    When it doesn't, you suck it up and code around it.

    Because at the very least it's a hell of a lot quicker than waiting
    for the language to change for you.
    alex23, Oct 25, 2011
    #19
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Martin Maurer
    Replies:
    3
    Views:
    4,773
    Peter
    Apr 19, 2006
  2. TheDustbustr
    Replies:
    1
    Views:
    430
    Sami Hangaslammi
    Jul 25, 2003
  3. Replies:
    9
    Views:
    524
  4. Chris Withers

    Problems with email.Generator.Generator

    Chris Withers, Sep 11, 2006, in forum: Python
    Replies:
    20
    Views:
    1,667
    Max M
    Sep 12, 2006
  5. Terry Reedy

    Generator functions subclass generator?

    Terry Reedy, Jun 18, 2009, in forum: Python
    Replies:
    0
    Views:
    444
    Terry Reedy
    Jun 18, 2009
Loading...

Share This Page