yield_all needed in Python

[Steve Holden]

In that case, your brain works nothing like mine.

Well, the fact that different people's brains work differently is hardly
worth remarking in this day and age. Hence my (and other people's)
questioning of the word "needed" in the title of this thread.

Guido has generally observed a parsimony about the introduction of
features such as the one you suggest into Python, and in particular he
is reluctant to add new keywords - even in cases like decorators that
cried out for a keyword rather than the ugly "@" syntax.

In my opinion that is a good thing mostly, not only because it avoids
code breakage (which can't possibly be bad) but also because it tends to
limit the number of ways that different programmers can express the same
idea.

I suspect this is why people have suggested that you were "only going to
save a few keystrokes". Despite your feeling that yield_all makes the
intent of the code more obvious there seems to be a majority in favor of
the simpler expression of the idea with a yield in a loop.

If you think there's a genuine chance this could be usefully added to
Python the solution is obvious: write and submit a PEP.

regards
Steve

 

[Douglas Alan]

Guido has generally observed a parsimony about the introduction of
features such as the one you suggest into Python, and in particular
he is reluctant to add new keywords - even in cases like decorators
that cried out for a keyword rather than the ugly "@" syntax.

In this case, that is great, since I'd much prefer

yield *gen1(arg)

than

yield_all gen1(arg)

anyway, as someone else suggested in this thread (followed by a
demonic laugh). The only reason I mentioned "yield_all" is because
there was a preexisting discussion that used "yield_all".

|>oug

 

[Steven Bethard]

In this case, that is great, since I'd much prefer

yield *gen1(arg)

than

yield_all gen1(arg)

I'm guessing the * syntax is pretty unlikely to win Guido's approval.
There have been a number of requests[1][2][3] for syntax like:

x, y, *rest = iterable

for unpacking a variable sized list (where *rest would work in an
analogous way to what it does in the args of a def.) Guido has
consistently rejected these proposals, e.g.:

"I think it's not worth adding now, and if you don't hear from me again
on this topic it's because I haven't changed my mind..."

My suspicion is that if he doesn't like the * syntax when there's a
close parallel to the argument parsing usage, he's not likely to like it
when there isn't one.

STeVe

[1]http://mail.python.org/pipermail/python-dev/2002-November/030349.html
[2]http://mail.python.org/pipermail/python-dev/2004-August/046684.html
[3]http://mail.python.org/pipermail/python-dev/2004-November/049895.html

 

[Douglas Alan]

I'm guessing the * syntax is pretty unlikely to win Guido's
approval. There have been a number of requests[1][2][3] for syntax
like:

x, y, *rest = iterable

Oh, it is so wrong that Guido objects to the above. Python needs
fully destructuring assignment!

|>oug

 

[Isaac To]

Douglas> If you'll reread what I wrote, you'll see that I'm not
Douglas> concerned with performance, but rather my concern is that
Douglas> I want the syntactic sugar. I'm tired of writing code
Douglas> that looks like

Douglas> def foogen(arg1):
Douglas> def foogen1(arg2):
Douglas> # Some code here
Douglas> # Some code here
Douglas> for e in foogen1(arg3): yield e
Douglas> # Some code here
Douglas> for e in foogen1(arg4): yield e
Douglas> # Some code here
Douglas> for e in foogen1(arg5): yield e
Douglas> # Some code here
Douglas> for e in foogen1(arg6): yield e

How about writing it like the following?

def gen_all(gen):
for e in gen:
yield e

def foogen(arg1):
def foogen1(arg2):
# Some code here
# Some code here
gen_all(arg3)
# Some code here
gen_all(arg4)
# Some code here
gen_all(arg5)
# Some code here
gen_all(arg6)

Regards,
Isaac.

 

[Isaac To]

def gen_all(gen):
for e in gen:
yield e

def foogen(arg1):
def foogen1(arg2):
# Some code here
# Some code here
gen_all(arg3)
^ I mean foogen1(arg3), obviously, and similar for below
# Some code here
gen_all(arg4)
# Some code here
gen_all(arg5)
# Some code here
gen_all(arg6)

Regards,
Isaac.

 

[Douglas Alan]

def gen_all(gen):
for e in gen:
yield e

def foogen(arg1):
def foogen1(arg2):
# Some code here
# Some code here
gen_all(arg3)
^ I mean foogen1(arg3), obviously, and similar for below
# Some code here
gen_all(arg4)
# Some code here
gen_all(arg5)
# Some code here
gen_all(arg6)

Regards,
Isaac.

If you actually try doing this, you will see why I want "yield_all".

|>oug

 

[Isaac To]

Douglas> If you actually try doing this, you will see why I want
Douglas> "yield_all".

Oh... I see your point.

I was about to suggest that the code in my posts before should be made
to work somehow. I mean, if in

def fun1(x):
if not x:
raise MyErr()
...

def fun2():
...
fun1(val)

fun2()

we can expect that main gets the exception thrown by fun1, why in

def fun1(x):
if not x:
yield MyObj()
...

def fun2():
fun1(val)

for a in fun2():
...

we cannot expect MyObj() to be yielded to main? But soon I found that
it is not realistic: there is no way to know that fun2 has generator
semantics. Perhaps that is a short-sightness in not introducing a new
keyword instead of def when defining generators.

Regards,
Isaac.

 

[Nick Coghlan]

In this case, that is great, since I'd much prefer

yield *gen1(arg)

If you do write a PEP, try to get genexp syntax supported by the yield keyword.

That is, the following currently triggers a syntax error:
def f():
yield x for x in gen1(arg)

I wouldn't mind seeing it successively yield the values returned by gen1()
instead. To my mind that better conveys what's going on than putting the yield
statement inside the for loop (it should also provide the opportunity to
optimise the next() calls by temporarily swapping the outer generator's frame
for the inner generator's frame, and swapping them back only when the inner
generator is exhausted)

That syntax error I mentioned makes it backwards compatible, too (existing code
must have parentheses around the genexp, which will not by altered if the yield
keyword gains a native genexp syntax).

If anyone thinks that this would result in a couple of parentheses making too
much difference, consider this:

Py> [x for x in []]
[]
Py> [(x for x in [])]
[<generator object at 0x009E6698>]

Cheers,
Nick.

 

[Steven Bethard]

I'm guessing the * syntax is pretty unlikely to win Guido's
approval. There have been a number of requests[1][2][3] for syntax
like:

x, y, *rest = iterable

Oh, it is so wrong that Guido objects to the above. Python needs
fully destructuring assignment!

Yeah, there are a lot of folks that like this idea, but most of us are
willing to conceded that Guido's intuition for these kind of things is
generally pretty good. This would be convenient for me occasionally,
but I certainly wouldn't need it that often...

STeVe

 

[Jeremy Bowers]

If you do write a PEP, try to get genexp syntax supported by the yield keyword.

That is, the following currently triggers a syntax error:
def f():
yield x for x in gen1(arg)

Hmmmm.

At first I liked this, but the reason that is a syntax error is that it is
"supposed" to be

def f():
yield (x for x in gen1(arg))

which today on 2.4 returns a generator instance which will in turn
yield one generator instance from the genexp, and I am quite uncomfortable
with the difference between the proposed behaviors with and without the
parens.

Which sucks, because at first I really liked it

We still would need some syntax to say "yield this 'in place' rather than
as an object".

Moreover, since "yield" is supposed to be analogous to "return", what does

return x for x in gen1(arg)

do? Both "it returns a list" and "it returns a generator" have some
arguments in their favor.

And I just now note that any * syntax, indeed, any syntax at all will
break this.

You know, given the marginal gains this gives anyway, maybe it's best off
to just observe that in the event that this is really important, it's
possible to hand-code the short-circuiting without too much work, and let
people write a recipe or something.

def genwrap(*generators):
while generators:
try:
returnedValue = generators[-1].next()
if hasattr(returnedValue, 'next'):
generators.append(returnedValue)
continue
yield returnedValue
except StopIteration:
generators.pop()

Not tested at all because the wife is calling and I gotta go

 

[Terry Reedy]

In this case, that is great, since I'd much prefer

yield *gen1(arg)

than

yield_all gen1(arg)

I'm guessing the * syntax is pretty unlikely to win Guido's approval.
There have been a number of requests[1][2][3] for syntax like:

x, y, *rest = iterable

for unpacking a variable sized list (where *rest would work in an
analogous way to what it does in the args of a def.) Guido has
consistently rejected these proposals, e.g.:

"I think it's not worth adding now, and if you don't hear from me again
on this topic it's because I haven't changed my mind..."

My suspicion is that if he doesn't like the * syntax when there's a close
parallel to the argument parsing usage, he's not likely to like it when
there isn't one.

Hmm. My impression is that Guido did not like x,*y=iterable because he
does *not* see it as a 'close parallel' but as a strained analogy. To me,
yield *iterable is closer to the use in function calling. It would mean
'unpack in time' rather than 'unpack in space' but that is natural (to me,
anyway) since that is what generators are about.

In any case, I also like this better than yield_all and would estimate that
it a higher chance of accceptance, even if still under 50%. Part of the
justification could be that the abbreviated form is much easier to speed up
than the expanded form. If the ref count of the iterator is just 1, then
it might be reasonable to assume that iterator.next will not be called from
anywhere else. (IE, I can't think of an exception, but know that there
might be one somehow.)

Terry J. Reedy


Terry J. Reedy

 

[Steven Bethard]

Hmm. My impression is that Guido did not like x,*y=iterable because he
does *not* see it as a 'close parallel' but as a strained analogy. To me,
yield *iterable is closer to the use in function calling. It would mean
'unpack in time' rather than 'unpack in space' but that is natural (to me,
anyway) since that is what generators are about.

I don't see the * in
yield *iterable
being much closer to the use in argument unpacking. Note what happens
to iterables after *:

py> def gen():
.... yield 1
.... yield 2
.... print "complete!"
....
py> def f(*args):
.... print args
....
py> f(*gen())
complete!
(1, 2)

The entire iterable is immediately exhausted. So if
yield *iterable
is supposed to parallel argument unpacking, I would expect that it would
also immediately exhaust the iterable, e.g. it would be equivalent to:
for item in tuple(iterable):
yield item
which I don't think is what the OP wants.

I'm certain I could get used to the syntax. I'm only suggesting that I
don't find it very intuitive. (And I *have* thought a lot about
argument unpacking -- see my older threads about *args being an iterable
instead of a tuple.)

STeVe

 

[Douglas Alan]

If you do write a PEP, try to get genexp syntax supported by the
yield keyword.

That is, the following currently triggers a syntax error:
def f():
yield x for x in gen1(arg)

Wouldn't

yield *(x for x in gen1(arg))

be sufficient, and would already be supported by the proposal at
hand?

Also, with the syntax you suggest, it's not automatically clear
whether you want to yield the generator created by the generator
expression or the values yielded by the expression. The "*" makes
this much more explicit, if you ask me, without hindering readability.

|>oug

 

[Skip Montanaro]

Jeremy> At first I liked this, but the reason that is a syntax error is
Jeremy> that it is "supposed" to be

Jeremy> def f():
Jeremy> yield (x for x in gen1(arg))

Jeremy> which today on 2.4 returns a generator instance which will in
Jeremy> turn yield one generator instance from the genexp, and I am
Jeremy> quite uncomfortable with the difference between the proposed
Jeremy> behaviors with and without the parens.

Jeremy> Which sucks, because at first I really liked it

Jeremy> We still would need some syntax to say "yield this 'in place'
Jeremy> rather than as an object".

def f():
yield from (x for x in gen1(arg))

Skip

 

[Francis Girard]

Le mercredi 2 Mars 2005 21:32, Skip Montanaro a écrit :

def f():
    yield from (x for x in gen1(arg))

Skip

This suggestion had been made in a previous posting and it has my preference :

def f():
yield from gen1(arg)

Regards

Francis

 

[Nick Coghlan]

Wouldn't

yield *(x for x in gen1(arg))

be sufficient, and would already be supported by the proposal at
hand?

It would, but, as Steven pointed out, the * in func(*args) results in
tuple(args) being passed to the underlying function.

So I see no reason to expect "yield *iterable" to imply a for loop that yields
the iterators contents. IMO, it's even more of a stretch than the tuple
unpacking concept (at least that idea involves tuples!)

Whereas:

yield x for x in iterable if condition

Maps to:
for x in iterable:
if condition:
yield x

Just as:
[x for x in iterable if condition]

Maps to:
lc = []
for x in iterable:
if condition:
lc.append(x)

And:
(x for x in iterable if condition)

Maps to:
def g()
for x in iterable:
if condition:
yield x

And removing a couple of parentheses is at least as clear as adding an asterisk
to the front

Cheers,
Nick.

 

[Nick Coghlan]

At first I liked this, but the reason that is a syntax error is that it is
"supposed" to be

def f():
yield (x for x in gen1(arg))

which today on 2.4 returns a generator instance which will in turn
yield one generator instance from the genexp

And it would continue to do so in the future. On the other hand, removing the
parens makes it easy to write things like tree traversal algorithms:

def left_to_right_traverse(node):
yield x for x in node.left
yield node .value
yield x for x in node.right

In reality, I expect yielding each item of a sub-iterable to be more common than
building a generator that yields generators.

, and I am quite uncomfortable
with the difference between the proposed behaviors with and without the
parens.

Why? Adding parentheses can be expected to have significant effects when it
causes things to be parsed differently. Like the example I posted originally:

[x for x in iterable] # List comp (no parens == eval in place)
[(x for x in iterable)] # Parens - generator goes in list

Or, for some other cases where parentheses severely affect parsing:

print x, y
print (x, y)

assert x, y
assert (x, y)

If we want to pass an iterator into a function, we use a generator expression,
not extended call syntax. It makes sense to base a sub-iterable yield syntax on
the former, rather than the latter.

Moreover, since "yield" is supposed to be analogous to "return", what does

return x for x in gen1(arg)

do? Both "it returns a list" and "it returns a generator" have some
arguments in their favor.

No, it would translate to:

for x in gen1(arg):
return x

Which is nonsense, so you would never make it legal.

And I just now note that any * syntax, indeed, any syntax at all will
break this.

As you noted, this argument is specious because it applies to *any* change to
the yield syntax - yield and return are fundamentally different, since yield
allows resumption of processing on the next call to next().

> You know, given the marginal gains this gives anyway,

I'm not so sure the gains will be marginal. Given the penalties CPython imposes
on recursive calls, eliminating the nested "next()" invocations could
significantly benefit any code that uses nested iterators.

An interesting example where this could apply is:

def flatten(iterable):
for item in iterable:
if item is iterable:
# Do the right thing for self-iterative things
# like length 1 strings
yield iterable
raise StopIteration
try:
itr = iter(item):
except TypeError:
yield item
else:
yield x for x in flatten(item)

Cheers,
Nick.

P.S. Which looks more like executable pseudocode?

def traverse(node):
yield *node.left
yield node .value
yield *node.right

def traverse(node):
yield x for x in node.left
yield node .value
yield x for x in node.right

 

[Paul Moore]

Doug> def foogen(arg1):

Doug> def foogen1(arg2):
Doug> # Some code here

Doug> # Some code here
Doug> yield_all foogen1(arg3)
Doug> # Some code here
Doug> yield_all foogen1(arg4)
Doug> # Some code here
Doug> yield_all foogen1(arg5)
Doug> # Some code here
Doug> yield_all foogen1(arg6)

If this idea advances I'd rather see extra syntactic sugar introduced to
complement the current yield statement instead of adding a new keyword.

You can work around the need for something like yield_all, or
explicit loops, by defining an "iflatten" generator, which yields
every element of its (iterable) argument, unless the element is a
generator, in which case we recurse into it:
.... it = iter(it)
.... for val in it:
.... if isinstance(val, GeneratorType):
.... for v2 in iflatten(val):
.... yield v2
.... else:
.... yield val

To take this one step further, you can define an @iflattened
decorator (yes, it needs a better name...)
.... def wrapper(*args, **kw):
.... for val in iflatten(f(*args, **kw)):
.... yield val
.... return wrapper

Now, we can do things like:
.... def t():
.... def g1():
.... yield 'a'
.... yield 'b'
.... yield 'c'
.... def g2():
.... yield 'd'
.... yield 'e'
.... yield 'f'
.... yield g1()
.... yield 1
.... yield g2()
....['a', 'b', 'c', 1, 'd', 'e', 'f']

This can probably be tidied up and improved, but it may be a
reasonable workaround for something like the original example.

Paul.

 

[Jeremy Bowers]

This can probably be tidied up and improved, but it may be a
reasonable workaround for something like the original example.

This is why even though in some sense I'd love to see yield *expr, I can't
imagine it's going to get into the language itself; it's too easy to do it
yourself, or provide a library function to do it (which would A: Be a lot
easier if we had some sort of "iterable" interface support and B: Be a
great demonstration of something useful that really needs protocol support
to come off right, because isinstance(something, GeneratorType) isn't
sufficient in general).

Abstractly I like the star syntax, but concretely I'm not a big fan of
adding something to the language that can be done right now with a fairly
short function/generator and hardly even any extra keystrokes to invoke it
when done right, and that overrides my abstract appreciation.