frange() question

[George Trojan]

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers).
Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment)
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:<type 'generator'>
But I always thought that generators need a keyword "yield". What is
going on here?

George

 

[Tim Golden]

A while ago I found somewhere the following implementation of frange(): ..
..
..
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:

But I always thought that generators need a keyword "yield". What is
going on here?

That's what's known as a generator expression:

http://docs.python.org/ref/genexpr.html
http://www.python.org/dev/peps/pep-0289/

TJG

 

[Stargaming]

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers). Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment) return (limit1 +
n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:
<type 'generator'>
But I always thought that generators need a keyword "yield". What is
going on here?

George

Consider the following:

def foo():
yield 1
def bar():
return foo()

Still, ``type(bar())`` would be a generator.

I don't want to tell you anything wrong because I don't know how
generators are implemented on the C level but it's more like changing
foo's (or frange's, in your example) return value.

HTH,
Stargaming

 

[John J. Lee]

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers).
Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment)
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:
<type 'generator'>

Functions are never generators, senso stricto. There are "generator
functions", which *return* (or yield) generators when you call them.
It's true sometimes people refer to generator functions as simply
"generators", but in examples like the above, it's useful to remember
that they are two different things.

In this case, frange isn't a generator function, because it doesn't
yield. Instead, it returns the result of evaluating a generator
expression (a generator). The generatator expression plays the same
role as a generator function -- calling a generator function gives you
a generator object; evaluating a generator expression gives you a
generator object. There's nothing to stop you returning that
generator object, which makes this function behave just like a regular
generator function.


John

 

[Carsten Haese]

Functions are never generators, senso stricto. There are "generator
functions", which *return* (or yield) generators when you call them.

Actually, a generator function is a function that returns a generator.
The generator, in turn, is an object that wraps the iterator protocol
around a resumable function. When the generator's next() method is
called, it resumes its wrapped function that runs until it yields or
finishes. When the wrapped function yields a value, it is suspended, and
the yielded value is returned as the result of next(). If the wrapped
function finished, next() raises StopIteration.

Generators are a special case of iterators. Any object that implements
the iterator protocol (which among other less important things mandates
a next() method that returns the next value or raises StopIteration) is
an iterator. Generators simply have their special way (by calling into a
resumable function) of implementing the iterator protocol, but any
stateful object that returns a value or raises StopIteration in next()
is an iterator.

It's true sometimes people refer to generator functions as simply
"generators", but in examples like the above, it's useful to remember
that they are two different things.

In this case, frange isn't a generator function, because it doesn't
yield. Instead, it returns the result of evaluating a generator
expression (a generator). The generatator expression plays the same
role as a generator function -- calling a generator function gives you
a generator object; evaluating a generator expression gives you a
generator object. There's nothing to stop you returning that
generator object, which makes this function behave just like a regular
generator function.

Generator expressions still "yield", but you don't see the yield in the
Python source code. Observe:
.... for i in (1,2,3):
.... yield i
.... .... return (i for i in (1,2,3))
.... ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'gi_frame', 'gi_running',
'next', 'send', 'throw']['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'gi_frame', 'gi_running',
'next', 'send', 'throw'] 0 SETUP_LOOP 19 (to 22)
3 LOAD_CONST 4 (4)
6 GET_ITER 10 STORE_FAST 0 (0)
13 LOAD_FAST 0 (0)
16 YIELD_VALUE
17 POP_TOP
18 JUMP_ABSOLUTE 7 0 SETUP_LOOP 18 (to 21)
3 LOAD_FAST 0 (0) 9 STORE_FAST 1 (1)
12 LOAD_FAST 1 (1)
15 YIELD_VALUE
16 POP_TOP
17 JUMP_ABSOLUTE 6 24 RETURN_VALUE

As you can see, except for a minor optimization in the byte code, there
is no discernible difference between the generator that's returned from
a generator expression and the generator that's returned from the
generator function. Note in particular that the byte code for g2
contains a YIELD_VALUE operation just like g1 does.

In fact, generator expressions are merely a short-hand notation for
simple generator functions of a particular form. The generator
expression

g = (expr(x) for x in some_iterable)

produces a generator that is almost indistinguishable from, and
operationally identical to, the generator produced by the following
code:

def __anonymous():
for x in some_iterable:
yield expr(x)
g = __anonymous()
del __anonymous

Hope this helps,

 

[Michael J. Fromberger]

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers).
Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment)
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:
<type 'generator'>
But I always thought that generators need a keyword "yield". What is
going on here?

Hi, George,

The expression returned is a "generator expression",

return (limit1 + n*increment for n in range(count))

Thus, although frange itself is not written as a generator, it does
return a generator as its result. The syntax is like that of list
comprehensions; see:

<http://docs.python.org/ref/genexpr.html>

Cheers,
-M

 

[John J. Lee]

Actually, a generator function is a function that returns a generator.

Read what I wrote again. What makes you begin your sentence with
"Actually", rather than "Putting it another way"?


[...snip more sensible stuff I must assume was directed at the OP...]


John

 

[Carsten Haese]

Read what I wrote again. What makes you begin your sentence with
"Actually", rather than "Putting it another way"?

I was attempting to correct your interjection "(or yield)." A generator
function doesn't yield a generator; it returns a generator that yields
sequence values.

The second half of my post illustrates a difference of opinion about
what constitutes a generator function. You state that frange() is not a
generator function because it doesn't use yield, but it behaves like
one. My point is that it *is* a generator function because the generator
expression is merely syntactic sugar for an equivalent for/yield loop.

Of course, the distinction of whether frange() *is* a generator function
or merely *behaves* as one is immaterial in practice, and we can both be
right in the absence of a formal definition of what a generator function
is. PEP 255 says "A function that contains a yield statement is called a
generator function," but that was written before generator expressions
were introduced.

HTH,

 

[John J. Lee]

I was attempting to correct your interjection "(or yield)." A generator
function doesn't yield a generator; it returns a generator that yields
sequence values.

OK. There's an obvious second language issue here (the first being
the generator / generator function), which I was trying to draw
attention away from, in the interests of explaining the concept, and
the first language issue. Probably a bad idea!

The second half of my post illustrates a difference of opinion about
what constitutes a generator function. You state that frange() is not a
generator function because it doesn't use yield, but it behaves like
one. My point is that it *is* a generator function because the generator
expression is merely syntactic sugar for an equivalent for/yield loop.

Seems to me that's a definitional thing, with no conceptual content,
so "difference of opinion" seems an odd choice of words. It would be
nice to nail the definitions down. Do the Python docs do that?

Of course, the distinction of whether frange() *is* a generator function
or merely *behaves* as one is immaterial in practice, and we can both be
right in the absence of a formal definition of what a generator function
is. PEP 255 says "A function that contains a yield statement is called a
generator function," but that was written before generator expressions
were introduced.

Ah, they do -- thanks. Though I'm now left puzzled why you express
your "difference of opionion" above...


John

 

[Carsten Haese]

Seems to me that's a definitional thing, with no conceptual content,
so "difference of opinion" seems an odd choice of words. It would be
nice to nail the definitions down. Do the Python docs do that?



Ah, they do -- thanks. Though I'm now left puzzled why you express
your "difference of opionion" above...

It's a matter of opinion whether the excerpt from the PEP constitutes
the formal definition of what a generator function is or isn't. A formal
definition needs conditions that are both sufficient and necessary. The
PEP only says that having a yield statement is sufficient for making a
generator function. It doesn't say that a yield statement is necessary
for making a generator function. In other words, it doesn't say that a
function that doesn't contain a yield statement isn't a generator
function.

The language reference is equally wishy-washy: "Using a yield statement
in a function definition is *sufficient* to cause that definition to
create a generator function instead of a normal function." [emphasis
mine]

Again, no indication that a yield statement is necessary for making a
generator function. It then proceeds to saying "When a generator
function is called, it returns an iterator known as a generator
iterator, or more commonly, a generator." That condition seems to be
true for the OP's frange() function, even though it doesn't have a yield
statement.

Until somebody can point out a definition that says unequivocally "an
object is a generator function if and only if ...", it's up in the air
whether frange() is a generator function or merely impersonates a
generator function. Ultimately, though, this is a purely academic
question without a useful answer. The useful conclusion is that a
function can behave like a generator function without a yield statement,
and we have reached that conclusion a long time ago.