Sending changed parameters into nested generators

[John O'Hagan]

I have a generator function which takes as arguments another generator and a
dictionary of other generators like this:

def modgen(gen, gendict):
for item in gen():
for k, v in gendict:
do_something_called_k(item, v.next())
yield item

I want to control the output of this generator by sending in new values to the
generators in gendict. I have done this using a list in a simple generator
like:

def loop(lis):
while True:
for item in lis:
yield item

and write:

lis[:] = [new_value]

to see the output of modgen() change.

However, I want to be able to use other modgen() instances as values in
gendict, each with a gendict of its own, which in turn may contain modgen()
instances, and so on recursively. The recursion stops wherever a simple
generator like loop() is encountered.

To this end I've made nested dictionaries like this example:

{'a':[1,2,3], 'b':{'a':[4,5,6]}, 'c':{'a':{'a':[7,8,9], 'b':{'c':[10]}}}}

and used a recursive function to convert them to the gendict I'm after like
this:

def iterized(dic):
itdic = {}
for k, v in dic.items():
if isinstance(v, dict):
itval = iterized(v)
itdic[k] = maingen(itval)
else:
itdic[k] = loop(v)
return itdic

Then I can write:

recdic = {nested:dictionary}
gendict = iterized(recdict)
mygen = modgen(gen, gendict)

The problem is that the gendict looks like this:

{'a': <generator object loop at 0x8841414>, 'b': <generator object modgen at
0x8841464>, 'c': <generator object modgen at 0x884148c>}

so I have no direct access from there to the sub-generators.

I have tried using a deep update function on the original nested dictionary,
but this only works if I change the contents of a list, not if I want to,
say, change a node from a list to a dictionary or vice versa, because in that
case I'm not operating on the same object the generator is using.

If anyone's still reading , how can I send new values to arbitrary sub-
generators?

Thanks,

john

 

[Ian]

I have tried using a deep update function on the original nested dictionary,
but this only works if I  change the contents of a list, not if I want to,
say, change a node from a list to a dictionary or vice versa, because in that
case I'm not operating on the same object the generator is using.

If anyone's still reading , how can I send new values to arbitrary sub-
generators?

I think you're focusing too heavily on generators. Just because a
nifty language construct exists doesn't mean that it's the right thing
to use in all situations. It sounds to me that what you really want
are objects with .next() methods and externally visible state -- i.e.
iterators.

Or it might be that coroutines are what you're really after, although
I think they would be difficult to manage in the way that you are
doing. You can view the docs on the pythonic approach to coroutines
at:

http://docs.python.org/whatsnew/2.5.html#pep-342-new-generator-features

Cheers,
Ian

 

[Arnaud Delobelle]

I have a generator function which takes as arguments another generator and a
dictionary of other generators like this:

def modgen(gen, gendict):
for item in gen():
for k, v in gendict:
do_something_called_k(item, v.next())
yield item

I want to control the output of this generator by sending in new values to the
generators in gendict. I have done this using a list in a simple generator
like:

def loop(lis):
while True:
for item in lis:
yield item

and write:

lis[:] = [new_value]

to see the output of modgen() change.

However, I want to be able to use other modgen() instances as values in
gendict, each with a gendict of its own, which in turn may contain modgen()
instances, and so on recursively. The recursion stops wherever a simple
generator like loop() is encountered.

To this end I've made nested dictionaries like this example:

{'a':[1,2,3], 'b':{'a':[4,5,6]}, 'c':{'a':{'a':[7,8,9], 'b':{'c':[10]}}}}

and used a recursive function to convert them to the gendict I'm after like
this:

def iterized(dic):
itdic = {}
for k, v in dic.items():
if isinstance(v, dict):
itval = iterized(v)
itdic[k] = maingen(itval)
else:
itdic[k] = loop(v)
return itdic

Then I can write:

recdic = {nested:dictionary}
gendict = iterized(recdict)
mygen = modgen(gen, gendict)

The problem is that the gendict looks like this:

{'a': <generator object loop at 0x8841414>, 'b': <generator object modgen at
0x8841464>, 'c': <generator object modgen at 0x884148c>}

so I have no direct access from there to the sub-generators.

I have tried using a deep update function on the original nested dictionary,
but this only works if I change the contents of a list, not if I want to,
say, change a node from a list to a dictionary or vice versa, because in that
case I'm not operating on the same object the generator is using.

If anyone's still reading , how can I send new values to arbitrary sub-
generators?

Thanks,

john

Try instead:

def loop(d, k):
while True:
for x in d[k]:
yield x

def iterized(dic):
itdic = {}
for k, v in dic.items():
if isinstance(v, dict):
itval = iterized(v)
itdic[k] = maingen(itval)
else:
itdic[k] = loop(dic, k)
return itdic

Untested.

 

[John O'Hagan]

I think you're focusing too heavily on generators. Just because a
nifty language construct exists doesn't mean that it's the right thing
to use in all situations. It sounds to me that what you really want
are objects with .next() methods and externally visible state -- i.e.
iterators.

Oddly enough I had just begun to replace some of the generator routines in the
real program with iterator classes, but it didn't occur to me that they could
solve the nesting problem. I guess I could access the parameters of nested
iterators like:

MyIter().param.iterator.param

and so on? I'll try it, thanks. Although if it can be done with built-in
functionality...

Or it might be that coroutines are what you're really after, although
I think they would be difficult to manage in the way that you are
doing.

From the rather slight docs on these features, it seems they are tailor-made
for what I'm doing (in fact my exact solution is described there as
"hackish"!), but so far I can't see how they could solve my nesting problem.
[...]

Thanks,

John

 

[Steven D'Aprano]

I have a generator function which takes as arguments another generator
and a dictionary of other generators like this:

def modgen(gen, gendict):
for item in gen():
for k, v in gendict:
do_something_called_k(item, v.next())
yield item [snip]
If anyone's still reading , how can I send new values to arbitrary
sub- generators?

I have a headache after reading your problem

I think it's a good time to point you at the Zen, particularly these five
maxims:

Beautiful is better than ugly.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
If the implementation is hard to explain, it's a bad idea.


I'm afraid that your nested generators inside another generator idea
fails all of those... it's not elegant (beautiful), it's complicated,
it's nested, and the implementation is hard to explain.

You could probably replace generators with full-blown iterators, but I
wonder what you're trying to accomplish that is so complicated that it
needs such complexity to solve it. What are you actually trying to
accomplish? Can you give a simple example of what practical task you hope
to perform? I suspect there's probably a more elegant way to solve the
problem.

 

[John O'Hagan]

I have a generator function which takes as arguments another generator
and a dictionary of other generators like this:

def modgen(gen, gendict):
for item in gen():
for k, v in gendict:
do_something_called_k(item, v.next())

yield item
[snip]

If anyone's still reading , how can I send new values to arbitrary
sub- generators?

I have a headache after reading your problem

I think it's a good time to point you at the Zen, particularly these five
maxims:

Beautiful is better than ugly.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
If the implementation is hard to explain, it's a bad idea.


I'm afraid that your nested generators inside another generator idea
fails all of those... it's not elegant (beautiful), it's complicated,
it's nested, and the implementation is hard to explain.

You could probably replace generators with full-blown iterators, but I
wonder what you're trying to accomplish that is so complicated that it
needs such complexity to solve it. What are you actually trying to
accomplish? Can you give a simple example of what practical task you hope
to perform? I suspect there's probably a more elegant way to solve the
problem.

I hope there is!

The project not practical but artistic; it's a real-time musical composition
program.

A (simplified) description: one module contains number-list generating
functions, others contain functions designed to filter and modify the number
lists produced, according to various parameters. Each such stream of number
lists is assigned a musical meaning (e.g. pitch, rhythm, volume, etc) and they
are combined to produce representations of musical phrases, which are sent to
a backend which plays the music as it is produced, and makes PDF scores. Each
such "instrument" runs as a separate thread, so several can play together in a
coordinated fashion.

All the compositional interest lies in the selection of number-list generators
and how their output is modified. For example, if I say "Play every third note
up an octave" it's not very interesting, compared to "Play every nth note up
an interval of m", where n and m vary according to some pattern. It gets even
more interesting when that pattern is a function of x and y, which also vary
according to another pattern, and so on.

To that end, I have each parameter of each modifier set by another generator,
such that the value may change with each iteration. This may continue
recursively, until at some level we give a parameter a simple value.

That's all working, but I also want it to be interactive. Each thread opens a
terminal where new options can be entered, but so far it only works, as I
mentioned, for changing the values in a top-level mutable object.

I agree that as a piece of code this is complicated, nested and hard to
explain, but I don't agree that it's ugly. In fact, it was my existing
interest in a "patterns within patterns" style of composition that got me
learning python in the first place, as the patterns got too hard to keep track
of manually.

I'm not attached to using nested generators, and would be happy with any
solution which enables me to do what I've described. It seems to me that the
concept is inherently nested, but I'd be glad to be proved wrong.

Regards,

John

 

[cbrown]

I have a generator function which takes as arguments another generator
and a dictionary of other generators like this:
def modgen(gen, gendict):
for item in gen():
for k, v in gendict:
do_something_called_k(item, v.next())
yield item

[snip]

If anyone's still reading , how can I send new values to arbitrary
sub- generators?

I have a headache after reading your problem

I think it's a good time to point you at the Zen, particularly these five
maxims:

Beautiful is better than ugly.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
If the implementation is hard to explain, it's a bad idea.

I'm afraid that your nested generators inside another generator idea
fails all of those... it's not elegant (beautiful), it's complicated,
it's nested, and the implementation is hard to explain.

You could probably replace generators with full-blown iterators, but I
wonder what you're trying to accomplish that is so complicated that it
needs such complexity to solve it. What are you actually trying to
accomplish? Can you give a simple example of what practical task you hope
to perform? I suspect there's probably a more elegant way to solve the
problem.

I hope there is!

The project not practical but artistic; it's a real-time musical composition
program.

A (simplified) description: one module contains number-list generating
functions, others contain functions designed to filter and modify the number
lists produced, according to various parameters. Each such stream of number
lists is assigned a musical meaning (e.g. pitch, rhythm, volume, etc) and they
are combined to produce representations of musical phrases, which are sent to
a backend which plays the music as it is produced, and makes PDF scores. Each
such "instrument" runs as a separate thread, so several can play together in a
coordinated fashion.

All the compositional interest lies in the selection of number-list generators
and how their output is modified. For example, if I say "Play every third note
up an octave" it's not very interesting, compared to "Play every nth note up
an interval of m", where n and m vary according to some pattern. It gets even
more interesting when that pattern is a function of x and y, which also vary
according to another pattern, and so on.

To that end, I have each parameter of each modifier set by another generator,
such that the value may change with each iteration. This may continue
recursively, until at some level we give a parameter a simple value.

That's all working, but I also want it to be interactive. Each thread opens a
terminal where new options can be entered, but so far it only works, as I
mentioned, for changing the values in a top-level mutable object.

I might first suggest this, although I have some caveats to add:

def genfilter(evaluator, **param_sources):
while True:
params = {}
for param, gen in param_sources.iteritems():
params[param] = gen.next()
yield evaluator(**params)

You can then do things like:
1|2
2|4
3}6
4|8

or, more relevant to your original question regarding modifying things
mid-stream:
3|9

Obviously more complex and useful examples could be made, but
hopefully you get the idea: don't focus on modifying the generators,
instead modify the action of the functions which take the generators
as inputs.

But a real problem with this whole strategy is that a generator's
next() function is called every time it is evaluated. If the
relationship between your various generators forms a rooted tree,
that's not a problem, but I would think the relationships form a
directed acyclic graph, and in that case, you end up 'double
incrementing' nodes in a way you don't want:
1|4
3|8
5|12
7|16

And that's not an unnatural situation for your application: you might
have a generator that produces a note, and then three other distinct
genfilters that you want to use to create three instruments which form
a chord based on that note. With the above pattern (and also your
original pattern) you would end up invoking the note generator three
times; and I think you don't really want that.

To solve that problem, you need a somewhat more complex solution: a
class that ensures that each previous stage is only invoked once per
'pass'. I've got an idea for that, if that is of interest.

Cheers - Chas