Yield

[Mateuszk87]

Hi.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

thanks in forward

mateusz

 

[Fredrik Lundh]

may someone explain "yield" function, please. how does it actually work
and when do you use it?

it returns a value from a function without actually terminating the
function; when the function is resumed, it'll continue to execute after
the yield.

a function that contains a yield statement is called a "generator", and
is most often used in a for-in loop, or in other contexts that expect a
sequence. the loop is automatically terminated when the function
returns in a usual way:
.... yield 1
.... yield 2
.... yield 3
........ print item
....
1
2
3

>>> sum(gen()) 6
>>> [str(i) for i in gen()]

['1', '2', '3']

you can also use the generator "by hand"; when you call a generator
function, it returns a special "generator object", and then immediately
suspends itself. to run the generator, call its "next" method:
3

when the generator is exhausted, it raises a StopIterator exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

reference information:

http://effbot.org/pyref/yield.htm

hope this helps!

</F>

 

[dwelch91]

Hi.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

thanks in forward

mateusz

http://docs.python.org/ref/yield.html

 

[Fredrik Lundh]

reference information:

also see:

http://effbot.org/pyfaq/what-is-a-generator.htm

</F>

 

[Carsten Haese]

Hi.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

[There is probably a much better explanation of this somewhere on the
net already, but I feel like writing this out myself.]

"yield" is a keyword. In simple terms, if "yield" appears inside a
function, the function becomes a generator that can return multiple
results.

Consider the following trivial example of a function that calculates the
elements of some sequence and prints each one.

def squares(n):
for i in range(n):
print i**2

Now suppose it turns out that in addition to--or instead of--printing
the elements of this sequence, the caller wants to do something
different with them. You could handle this by building a list, and
return the list to the caller to loop over and operate on as they
choose. But if the list is very big, that's not very memory-efficient if
the caller only needs to operate on one element at a time.

This is where yield comes in. If you write this:

def squares(n):
for i in range(n):
yield i**2

squares is now a generator function. If you call squares(100), it won't
calculate any sequence elements immediately. It will instead return an
iterator the caller can loop over:

for el in squares(100):
print el # or do anything else the caller decides

The beauty of this is that you achieve physical separation between
generating elements of a sequence and using them. The generator function
only cares about how to produce the elements, and the caller only cares
about how to use them.

Hope this helps,

Carsten.

 

[Mateuszk87]

thx for the quick answer. i ll have a look.

 

[John Henry]

Thank you. This is very clear. I can see that this is useful in lots
of situations.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

it returns a value from a function without actually terminating the
function; when the function is resumed, it'll continue to execute after
the yield.

a function that contains a yield statement is called a "generator", and
is most often used in a for-in loop, or in other contexts that expect a
sequence. the loop is automatically terminated when the function
returns in a usual way:
... yield 1
... yield 2
... yield 3
...... print item
...
1
2
3

sum(gen()) 6
[str(i) for i in gen()]

['1', '2', '3']

you can also use the generator "by hand"; when you call a generator
function, it returns a special "generator object", and then immediately
suspends itself. to run the generator, call its "next" method:
3

when the generator is exhausted, it raises a StopIterator exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

reference information:

http://effbot.org/pyref/yield.htm

hope this helps!

</F>

 

[olive]

dwelch91 a écrit :

http://docs.python.org/ref/yield.html

This is a perfect example that demonstrate how the actual python is bad
and most of the time useless (at least for me).

We really need mor example !

I would like to thanks Fredrik for his contribution to improve that.

Olivier.

 

[John Machin]

dwelch91 a écrit :

This is a perfect example that demonstrate how the actual python is bad
and most of the time useless (at least for me).

I think that you left the word "documentation" out after "python"

We really need mor example !

I would like to thanks Fredrik for his contribution to improve that.

Call me crazy, but after an admittedly quick read, the version on the
wiki seems to be about word for word with on the docs.python.org version.

Could you please tell us how you think the wiki version is an improvement?

Cheers,
John

 

[Fredrik Lundh]

This is a perfect example that demonstrate how the actual python is bad
and most of the time useless (at least for me).

there's a place for (relatively) formal reference documentation, but
it's hardly ever the right place to learn why things are there, or how
they are supposed to be used.

We really need mor example !

I would like to thanks Fredrik for his contribution to improve that.

thanks!

(for the curious, my current plan is to use

http://effbot.org/zone/idea-seealso.htm

files to link from the official documentation to tutorials, example
collections, and blog articles. I encourage content producers to start
looking into generating such files for their document collections.)

</F>

 

[Fredrik Lundh]

Call me crazy, but after an admittedly quick read, the version on the
wiki seems to be about word for word with on the docs.python.org version.

maybe he was thinking about the article I posted, or the FAQ link I
posted in a followup:

http://effbot.org/pyfaq/what-is-a-generator.htm

which was based on my article and the glossary entry from the tutorial:

http://effbot.org/pytut/glossary.htm#generator

Could you please tell us how you think the wiki version is an improvement?

an "add comment" link? direct URL:s for all concepts in the language?
extensive hyperlinking? semantic retargetable markup?

</F>

 

[John Machin]

maybe he was thinking about the article I posted, or the FAQ link I
posted in a followup:

http://effbot.org/pyfaq/what-is-a-generator.htm

which was based on my article and the glossary entry from the tutorial:

http://effbot.org/pytut/glossary.htm#generator

Quite possibly.

an "add comment" link? direct URL:s for all concepts in the language?
extensive hyperlinking? semantic retargetable markup?

Yes, they're great. Maybe I was thinking about the text contents only

Cheers,
John

 

[Danny Colligan]

Now that we're on the subject, what are the advantages of using
generators over, say, list comprehensions or for loops? It seems to me
that virtually all (I won't say everything) the examples I've seen can
be done just as easily without using generators. For example,
Fredrik's initial example in the post:

a = [1,2,3]
for i in a: print i

....
1
2
3

sum(a) 6
[str(i) for i in a] ['1', '2', '3']

Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

Thanks in advance,

Danny

 

[Richard Brodie]

Now that we're on the subject, what are the advantages of using
generators over, say, list comprehensions or for loops? It seems to me
that virtually all (I won't say everything) the examples I've seen can
be done just as easily without using generators.

The more trivial the example, the harder it is to see the advantage.
Suppose you wanted to sum the first 10000 primes. A quick Google
fins you Wensheng Wang's recipe:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/366178
Just add print sum(primes(10000)), and you're done.

 

[Danny Colligan]

The more trivial the example, the harder it is to see the advantage.

I absoultely agree. Thanks for pointing me out to some real-world
code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes). A generator version would be:
.... if n<2: yield []
.... s=range(3,n+1,2)
.... mroot = n ** 0.5
.... half=(n+1)/2-1
.... i=0
.... m=3
.... while m <= mroot:
.... if s:
.... j=(m*m-3)/2
.... s[j]=0
.... while j<half:
.... s[j]=0
.... j+=m
.... i=i+1
.... m=2*i+3
.... yield 2
.... for x in s:
.... if x: yield x
....Traceback (most recent call last):

Danny Colligan

 

[Carsten Haese]

Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

The memory efficiency is definitely a major advantage. Generators allow
you to efficiently produce, manipulate, and consume sequences of
arbitrary length. The length of the sequence could even be potentially
infinite, which is impossible to handle when you're working with actual
lists.

The memory efficiency aside, it's more elegant to write something like
this:

def squares(n):
for i in range(n):
yield i**2

than something like this:

def squares(n):
result = []
for i in range(n):
result.append(i**2)
return result

-Carsten

 

[Carsten Haese]

The more trivial the example, the harder it is to see the advantage.

I absoultely agree. Thanks for pointing me out to some real-world
code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes). A generator version would be:
... if n<2: yield []
... s=range(3,n+1,2)
... mroot = n ** 0.5
... half=(n+1)/2-1
... i=0
... m=3
... while m <= mroot:
... if s:
... j=(m*m-3)/2
... s[j]=0
... while j<half:
... s[j]=0
... j+=m
... i=i+1
... m=2*i+3
... yield 2
... for x in s:
... if x: yield x

Not quite:

x = primes(1)
x.next() []
x.next() 2
x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

To handle n<2 correctly, you have to "return" instead of "yield []".
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

-Carsten

 

[Richard Brodie]

I absoultely agree. Thanks for pointing me out to some real-world
code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes).

Oops, should have looked at the code more closely. Another example
would be os.walk() in the standard library.

 

[Tim Chase]

I absoultely agree. Thanks for pointing me out to some real-world

Oops, should have looked at the code more closely. Another example
would be os.walk() in the standard library.

I've used a similar scheme for dealing with incoming data files.
Sometimes, I get them Emailed/FTP'ed so they're sitting in a
folder, so I can do a multitude of them by just pointing
os.walk() at them. Other times, I get them on specially
formatted CDs. It's nice to have the clean main-loop logic of:

for file_name in source:
do_stuff(file_name)

and be able to dynamically set "source" to an generator that
either wraps os.walk() for a given directory or repeatedly (1)
reads the file on the CD to know where it should be dumped, (2)
copies files off the CD to their associated dumping ground, (3)
ejects the CD, (4) yields the name of the file to process in the
dumping-ground directory, and (5) prompts the user if there are
more CDs to be processed.

Without "yield" functionality, I'd either have to have two
duplicate loops with similar internal logic (the do_stuff() sort
of code), or I'd have to read in all the files off the CD and
*then* process them all as a list afterwards.

Just one of the places I've recently been thankful for generators.

-tkc

 

[Fredrik Lundh]

Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

the main advantage is that it lets you decouple the generation of data
from the use of data; instead of inlining calculations, or using a pre-
defined operation on their result (e.g printing them or adding them to a
container), you can simply yield them, and let the user use whatever
mechanism she wants to process them. i.e. instead of

for item in range:
calculate result
print result

you'll write

def generate():
for item in range:
yield result

and can then use

for item in generate():
print item

or

list(process(s) for s in generate())

or

sys.stdout.writelines(generate())

or

sum(generate())

etc, without having to change the generator.

you can also do various tricks with "endless" generators, such as the
following pi digit generator, based on an algorithm by Python's grand-
father Lambert Meertens:

def pi():
k, a, b, a1, b1 = 2, 4, 1, 12, 4
while 1:
# Next approximation
p, q, k = k*k, 2*k+1, k+1
a, b, a1, b1 = a1, b1, p*a+q*a1, p*b+q*b1
# Yield common digits
d, d1 = a/b, a1/b1
while d == d1:
yield str(d)
a, a1 = 10*(a%b), 10*(a1%b1)
d, d1 = a/b, a1/b1

import sys
sys.stdout.writelines(pi())

</F>