Generator Frustration

[TommyVee]

I'm using the SimPy package to run simulations. Anyone who's used this
package knows that the way it simulates process concurrency is through the
clever use of yield statements. Some of the code in my programs is very
complex and contains several repeating sequences of yield statements. I
want to combine these sequences into common functions. The problem of
course, is that once a yield gets put into a function, the function is now a
generator and its behavior changes. Is there any elegant way to do this? I
suppose I can do things like ping-pong yield statements, but that solutions
seems even uglier than having a very flat, single main routine with
repeating sequences.

Thanks in advance.

 

[Steven D'Aprano]

I'm using the SimPy package to run simulations. Anyone who's used this
package knows that the way it simulates process concurrency is through
the clever use of yield statements. Some of the code in my programs is
very complex and contains several repeating sequences of yield
statements. I want to combine these sequences into common functions.
The problem of course, is that once a yield gets put into a function,
the function is now a generator and its behavior changes. Is there any
elegant way to do this?

I don't quite understand the nature of your problem, but it seems to me
that it is easily solved by simply not using yield in the common
functions. Instead of:


def main():
for x in sequence:
if a:
y = a+b+c+d+e
yield y
elif b:
y = a+b+c+d+f
yield y
else:
y = a+b+c+d
yield y+1

(where each of the lines y = ... is meant to be a big block of mostly
common code), factor out the similar parts into one or more external
functions:

def f(a, b, c, d):
return a+b+c+d # big block of common code

def main():
for x in sequence:
if a:
y = f(a, b, c, d) # call the common part
y += e
yield y
elif b:
y = f(a, b, c, d)
y += f
yield y
else:
y = f(a, b, c, d)
y += 1
yield y


If this is not what you're talking about, an example may help.

I suppose I can do things like ping-pong yield statements,

I have no idea what you think that means, but another alternative is to
loop over the generator output and re-yield it:

for x in f():
yield x

A nice piece of syntax that has been proposed for Python is "yield from",
which will do the same thing, but you can't use that yet.

 

[Gregory Ewing]

A nice piece of syntax that has been proposed for Python is "yield from",
which will do the same thing, but you can't use that yet.

Unless you're impatient enough to compile your own
Python with my patch applied:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html

 

[Jack Diederich]

You can also patch the library to always return lists instead of generators.

def seriously_I_just_want_lists(func):
def replacement(*args, **kwargs):
return list(func(*args, **kwargs))
return replacement

import otherlib
otherlib.somefunc = seriously_I_just_want_lists(otherlib.somefunc)

But I'd avoid it where you can. While the idea of a list is simpler
than the idea of an iterable (and easier to get your head around)
iterables promise less and that makes them more flexible and easier to
composite.

-Jack

 

[Jan Decaluwe]

I'm using the SimPy package to run simulations. Anyone who's used
this package knows that the way it simulates process concurrency is
through the clever use of yield statements. Some of the code in my
programs is very complex and contains several repeating sequences of
yield statements. I want to combine these sequences into common
functions. The problem of course, is that once a yield gets put into
a function, the function is now a generator and its behavior changes.
Is there any elegant way to do this? I suppose I can do things like
ping-pong yield statements, but that solutions seems even uglier than
having a very flat, single main routine with repeating sequences.

In MyHDL, this is supported by the possibility to yield generators,
which are then handled by the simulation engine.

Jan


--
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
Python as a HDL: http://www.myhdl.org
VHDL development, the modern way: http://www.sigasi.com
Analog design automation: http://www.mephisto-da.com
World-class digital design: http://www.easics.com

 

[TommyVee]

"Gregory Ewing" wrote in message

A nice piece of syntax that has been proposed for Python is "yield from",
which will do the same thing, but you can't use that yet.

Unless you're impatient enough to compile your own
Python with my patch applied:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html

 

[Thomas Rachel]

Am 04.06.2011 20:27 schrieb TommyVee:

I'm using the SimPy package to run simulations. Anyone who's used this
package knows that the way it simulates process concurrency is through
the clever use of yield statements. Some of the code in my programs is
very complex and contains several repeating sequences of yield
statements. I want to combine these sequences into common functions.

Which are then generators.

The problem of course, is that once a yield gets put into a function,
the function is now a generator and its behavior changes.

Isn't your "main" function a generator as well?

Is there any elegant way to do this? I suppose I can do things like
ping-pong yield statements, but that solutions seems even uglier than
having a very flat, single main routine with repeating sequences.

I'm not sure if I got it right, but I think you could emulate this
"yield from" with a decorator:

def subgen1(): yield 1; yield 2;
def subgen2(): yield 1; yield 2;

Instead of doing now

def allgen():
for i in subgen1(): yield i
for i in subgen2(): yield i

you as well could do:

def yield_from(f):
def wrapper(*a, **k):
for sub in f(*a, **k):
for i in sub:
yield i
return wrapper

@yield_from
def allgen():
yield subgen1()
yield subgen2()

(Untested.)


Thomas

 

[TommyVee]

"Thomas Rachel" wrote in message
Am 04.06.2011 20:27 schrieb TommyVee:

I'm using the SimPy package to run simulations. Anyone who's used this
package knows that the way it simulates process concurrency is through
the clever use of yield statements. Some of the code in my programs is
very complex and contains several repeating sequences of yield
statements. I want to combine these sequences into common functions.

Which are then generators.

The problem of course, is that once a yield gets put into a function,
the function is now a generator and its behavior changes.

Isn't your "main" function a generator as well?

Is there any elegant way to do this? I suppose I can do things like
ping-pong yield statements, but that solutions seems even uglier than
having a very flat, single main routine with repeating sequences.

I'm not sure if I got it right, but I think you could emulate this
"yield from" with a decorator:

def subgen1(): yield 1; yield 2;
def subgen2(): yield 1; yield 2;

Instead of doing now

def allgen():
for i in subgen1(): yield i
for i in subgen2(): yield i

you as well could do:

def yield_from(f):
def wrapper(*a, **k):
for sub in f(*a, **k):
for i in sub:
yield i
return wrapper

@yield_from
def allgen():
yield subgen1()
yield subgen2()

(Untested.)


Thomas

Yes, the main function is a generator. Give me a day or two to absorb your
code. But in the meantime, perhaps a quick explanation of the SimPy package
is in order. Also, here's the link if you're interested:

http://simpy.sourceforge.net/simpy_overview.htm

The way the package works is that you instantiate what I'll call an "actor"
class, inheriting from one of the SimPy base classes. Once you instantiate
the actor class(es), it it "registered" in SimPy. Then when you start the
simulation, SimPy's main routine will begin to "dispatch" each of the actor
classes by driving what they call a "PEM" method in the actor class. Within
that method, you simulate the activity of the actor. The way that you tell
SimPy that you are "busy", or that you want to request or release a
resource, is with a yield statement. Here's a little example:

from SimPy.Simulation import *

svc1 = Resource(capacity=1)
svc2 = Resource(capacity=1)

class Customer(Process):
def PEM(self):

print now(), "I am starting..."

print now(), "I am requesting service1"
yield request, self, svc1

print now(), "I got service 1, now I'm going ask it to do 10 ticks
worth of work"
yield hold, self, 10

print now(), "Service 1's work is done, I am now releasing him"
yield release, self, svc1

print now(), "Now I am requesting service2"
yield request, self, svc2

print now(), "I got service 2, now I'm going ask him to do 5 ticks
worth of work"
yield hold, self, 5

print now(), "Service 2's work is done, I am now releasing him"
yield release, self, svc2

print now(), "I am ending..."

initialize()

# Create a customer and "register" (activate) him to SimPy
c = Customer()
activate(c, c.PEM())

# Pass control to the SimPy main dispatch routine, and run the simulation
for 100 ticks
simulate(until=100)

Note that when you do the yields, you actually send back information to the
SimPy main routine. For example, when you do a "yield hold, self, 10",
control will be yielded to SimPy, he will simulate the passage of 10 time
ticks, and redispatch you (do a "next" on your PEM). Similar deal with the
"yield request, self, svc1". When you yield to SimPy, he will attempt to
obtain the resource for you, and if it is already taken by another actor,
you will wait in a queue until released. At that time, SimPy will then
redispatch you. Obviously, this is a trivial example, since it only
involves the creation of a single actor. In a real simulation, there could
be hundreds of these things running "concurrently", all vying for resources,
holding them for varying amounts of time, etc. SimPy also uses yield
statements to simulate other things too, like waiting for a signal from
another actor.

In the example you see above, note that I am "repeating" a generic sequence
twice, e.g. "yield request, self, svcX", followed by "yield hold, self,
time", followed by "yield release, self, svcX". In a workflow example, you
may have literally dozens of services that you may hit in serial or
parallel. Because of the generator "restriction", you'd have to code these
three statements repeatedly in the PEM routine. It would be nice to just
have a class method which did:

def UseService(self, svcName, svcTime):
yield request, self, svcName
yield hold, self, svcTime
yield release, self, svcName

Then you can just call it in your main routine, passing the desired
parameters (e.g., "UseService(svc1, 10)").

Anyway, hopefully you'll see that there's a reason for my original question.
Let me absorb what you sent and see if it makes sense.

Thanks, Tom

 

[Joel]

I'm using the SimPy package to run simulations. Anyone who's used this
package knows that the way it simulates process concurrency is through the
clever use of yield statements. Some of the code in my programs is very
complex and contains several repeating sequences of yield statements.  I
want to combine these sequences into common functions.  The problem of
course, is that once a yield gets put into a function, the function is now a
generator and its behavior changes.  Is there any elegant way to do this?  I
suppose I can do things like ping-pong yield statements, but that solutions
seems even uglier than having a very flat, single main routine with
repeating sequences.

Thanks in advance.

I actually found a reasonable answer to this, I think. If one of the
called functions contains a yield, that function is by definition a
generator, and will test as such with 'if
type(result)==types.GeneratorType:'. This makes it possible for the
function that calls the subroutine to either do its own yield, or to
yield the result of the function's next 'yield' statement. I've got a
class the run method of which calls the 'cycle' method of its derived
class, as follows:

while True:
result= self.cycle(now()) # result may or may not be a
generator
if type(result)==types.GeneratorType:
# Next is a generator, meaning it includes a
'yield'.
# Otherwise, result should be None and cycle is a
simple
# function.
try:
yield result.next()
while True:
yield result.next()
except StopIteration:
pass

else:
# self.cycle() was a simple function, returning None-
it's
# done now.
pass
# It is time for this process to go back to sleep; all the
yields
# in self.cycle() have been processed.
yield hold,self,self.nextCycleTime-now()

 

[Terry Reedy]

....

I actually found a reasonable answer to this, I think. If one of the
called functions contains a yield, that function is by definition a
generator,

A nomenclature note: a function with yield is a 'generator function'. It
is an instance of the 'function' class, same as for any other def
statement (or lambda expression). It returns an instance of class
'generator, as you note here

and will test as such with 'if type(result)==types.GeneratorType:'.

It is the result, and not the function, that is the generator (a type of
iterator).

 

[Joel]

A nomenclature note: a function with yield is a 'generator function'. It
is an instance of the 'function' class, same as for any other def
statement (or lambda expression). It returns an instance of class
'generator, as you note here

 > and will test as such with 'if type(result)==types.GeneratorType:'.

It is the result, and not the function, that is the generator (a type of
iterator).

Yes, quite right. I stand corrected.

-Joel