Opposite of yield?

[achrist]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?


Al

 

[Paul Rubin]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?

Not much chance. Even Stackless can't do that any more, if I
understand correctly.

 

[Bob Gailer]

At 01:40 PM 9/10/2003, (e-mail address removed) wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Sounds like a task for a thread that sleeps a while then checks for the data.

Also reminiscent (a long time ago) someone proposed a COME FROM statement
for FORTRAN to be the inverse of GO TO.

Perhaps a new Python statement "suck"?

Bob Gailer
(e-mail address removed)
303 442 2625

 

[Erik Max Francis]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Yep:

generator.next()

 

[Peter Hansen]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?

Queue.Queue.get() ...

-Peter

 

[Dave Benjamin]

...

Perhaps a new Python statement "suck"?

Oh come on, now, just because Java does it...

 

[Christos TZOTZIOY Georgiou]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Almost. Use a Queue.Queue() and threads. That is, some other thread(s)
does/do .put() and you wait with a .get().
Without threads, I don't believe there is such a mechanism, not even in
Stackless, as Paul said.

If this is not implemented now, is there any chance that it's on the
list of things to come?

I believe such a chance would be a candidate for dietary products
advertising (that is, slim .

 

[Christos TZOTZIOY Georgiou]

Oh come on, now, just because Java does it...

QOTW +1

 

[achrist]

Queue.Queue.get() ...

That looks good. But that needs a thread to block, right?
A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?


Al

 

[achrist]

Yep:

generator.next()

This style supports only output-driven processes. The generator,
being on the input side, is always the server, and the output
side is always the client. It doesn't fit the situation where the
input is heterogeneous and input has to get spewed in several
directions, depending on its content.


Al

 

[Peter Hansen]

That looks good. But that needs a thread to block, right?
Yep.

A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?

The real question might be why would you want to?

If you don't want a thread, but you want something which takes
input from elsewhere and does some processing, then returns
control to some other place until more data is available (which
is what one might assume if threads aren't good for you), there
is already a convenient solution: the subroutine. ;-)

Seriously, what's the requirement driving the need for this?
(I suspect Erik Max's answer is still what you need, and you
misunderstand the nature of what he was suggesting, but if you'll
explain the specific case you have in mind we'll know for sure.)

-Peter

 

[Chris Liechti]

That looks good. But that needs a thread to block, right?
yes

A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

basicaly you poll every generator (or more precicly "iterable" as that is
what a generator returns), one after the other... enjoy the link below.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?

this one is for you
http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html

you wont get happy with hundrets of native threads, but with the light ones
from above you can have thousands...

generaly you will find interesting stuff here (one line):
http://www-
106.ibm.com/developerworks/views/linux/articles.jsp?sort_order=desc&expand=
&sort_by=Date&show_abstract=true&view_by=Search&search_by=charming+python%3
A

chris

 

[Bengt Richter]

This style supports only output-driven processes. The generator,
being on the input side, is always the server, and the output
side is always the client. It doesn't fit the situation where the
input is heterogeneous and input has to get spewed in several
directions, depending on its content.

Could you make some fantasy-code to illustrate what you mean?

Regards,
Bengt Richter

 

[Stuart D. Gathman]

This style supports only output-driven processes. The generator, being
on the input side, is always the server, and the output side is always
the client. It doesn't fit the situation where the input is
heterogeneous and input has to get spewed in several directions,
depending on its content.


It does if you add a dispatcher framework. There is a python project for
just such a framework, but I can't remember the name. The idea is dirt
simple, but the implementation gets knarly with exceptions and such.

Found it - Charming Python 
http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html

 

[Erik Max Francis]

This style supports only output-driven processes. The generator,
being on the input side, is always the server, and the output
side is always the client. It doesn't fit the situation where the
input is heterogeneous and input has to get spewed in several
directions, depending on its content.

The distinction you're making is push vs. pull processing. To get what
you want, you should use some sort of asynchronous mechanism with
synchronization when data is transferred. Threading with the use of the
Queue module comes to mind immediately. Other forms of asynchronous
processing are certainly possible, as well.

That being said, calling the next method of a generator _is_ the obvious
inverse process of a generator yielding a value.

 

[achrist]

The real question might be why would you want to?

If you don't want a thread, but you want something which takes
input from elsewhere and does some processing, then returns
control to some other place until more data is available (which
is what one might assume if threads aren't good for you), there
is already a convenient solution: the subroutine. ;-)

Seriously, what's the requirement driving the need for this?
(I suspect Erik Max's answer is still what you need, and you
misunderstand the nature of what he was suggesting, but if you'll
explain the specific case you have in mind we'll know for sure.)

No, I didn't say that I didn't want a thread. If there are no
serious drawbacks to relying heavily on threads, I'd be glad to
use them. This would be deluxe.

There really isn't any need for a generator, but it is a very
nice way to construct the input side of a program (the caller
doesn't have to worry about expicitly calling initialization and
finalization). Some kind of through-the-looking-glass version
of a generator would be a very nice way to construct the output
side.

I decided to test threads to see what happens:

-------------Warning, Code Follows---------------------

import Queue
import thread
import time

queues = [Queue.Queue()]
threads = []

appendLock = thread.allocate_lock()

def TransferData(queues, atPos):
while len(queues) < atPos + 1:
pass
inputQueue = queues[atPos]
outputQueue = Queue.Queue()
appendLock.acquire()
queues.append(outputQueue)
appendLock.release()
while 1:
data = inputQueue.get(True)
outputQueue.put(data, True)

startTime = time.time()
for i in range(1000):
print i,
threads.append(thread.start_new_thread(TransferData, (queues, i)))
queues[0].put(i, True)
appendLock.acquire()
print queues[-1].get(True)
appendLock.release()

print time.time() - startTime


--------------Warning Discussion Follows -------------------------

Running this on my Windows NT 200 NHz machine gave decent results.
With 1000 threads and 1000 Queues, memory used was 24MB. Python
takes almost 10 MB to start, so the overhead per thread + Queue
is only a little over 10 kb. Passing all the messages through
the chain took 2.5 minutes, so we are down around a quarter millisec
for each put or get.

That should be pretty good for many uses.


Al

 

[Oren Tirosh]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Here is a way of converting a generator function into a consumer using
a single item list as a mailbox:

from __future__ import generators

def consumer():
print "This code runs before any data is fed to the consumer"
mailbox = []; yield mailbox
while mailbox:
print "Here's what I got:", mailbox.pop()
yield mailbox
print "This code runs after end of data is signalled"

# How to feed a consumer:

i = 0
for mailbox in consumer():
if i < 5:
mailbox.append(i) # feed next item
i += 1


Output:
This code runs before any data is fed to the consumer
Here's what I got: 0
Here's what I got: 1
Here's what I got: 2
Here's what I got: 3
Here's what I got: 4
This code runs after end of data is signalled


For a more comprehensive solution try Christian Tismer's Stackless Python.

Oren

 

[Jan Decaluwe]

That looks good. But that needs a thread to block, right?
A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?

Al

You may want to check out how such things are done with my MyHDL
package. The target application is hardware design, but the idea
is more probably general.

http://jandecaluwe.com/Tools/MyHDL/manual/model-obj.html

Jan

 

[Greg Chapman]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?

You can use a generator as the body of a function. That is, the generator is
only ever accessed through the function, and the function and the generator have
some agreed on way to pass parameters. I.e.:

def getline(output):
'''getline accumulates text until it has a complete line (signaled
by either a newline (\\n) or a null (\\0)). It then passes the
line (without trailing \n) to output'''

genargs = [None]
def callgen(text):
genargs[0] = text
return gennext()
def gen():
line = ''
while True:
text, = genargs
if text:
lines = text.splitlines(True)
if line:
lines[0] = line+lines[0]
for line in lines[:-1]:
output(line[:-1])
line = lines[-1]
if line[-1] in '\n\0':
if line[-1] == '\n':
line = line[:-1]
output(line)
line = ''
yield None
gennext = gen().next
return callgen

def printer(text):
print text

lineprinter = getline(printer)

lineprinter('hello') #nothing output
lineprinter(' world\n') #prints 'hello world'

 

[Hung Jung Lu]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

This sounds like event-driven programming to me.

(1) Usage of threads in principle is optional, but at Python level
event-driven programming is probably always done in multi-threaded
environment.

(2) Also, at software level, interruptions are always emulated by
polling (i.e. event loop.) You will have a loop somewhere.

(3) I won't get into details of the event-handler implementation, as
it is well-known and you probably already know it or can find it
easily. But I'd like to mention a simple implementation using
exception handling. For instance:

while 1:
try:
poll_keyboard()
poll_mouse()
poll_microphone()
poll_temperature_sensor()
except IncomingSignal, e:
... handle event ...

You raise the IncomingSignal event inside the various polling
routines.

regards,

Hung Jung Lu