Stackless python and microthreads

[Matt Leslie]

Hi,

I'm trying to use microthreads under stackless python, since they sound
like exactly what I am after, but I am having very little success.
I've got a fresh install of python 2.3.3 from python.org, then
downloaded the binary python2.3 release of stackless python from
www.stackless.com. This contained three files:

python23.dll, python23.lib, python23.exp

I searched for the original versions of these files, and copied:

stackless python23.dll over c:\winnt\system32\python23.dll
stackless python23.lib over C:\Program Files\python2.3\libs\python23.lib

Since I believe .exp files are associated with .lib files. I put the
..exp file in the same place as the lib. There was no existing file with
this name on my file system.

I then downloaded the microthreads package from
http://willware.net:8080/uthread.py

I then ran python, type 'import uthread' and immediately got the
following error.

---------------------------------------
Python 2.3.3 Stackless 3.0 040407 (#51, Apr 7 2004, 19:28:46) [MSC
v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "uthread.py", line 35, in ?
import continuation, sys, traceback, bisect, time, StringIO
ImportError: No module named continuation

Has anyone run into this before? Can someone tell me where to find this
continuation module?

Thanks,
Matt

 

[Mike C. Fletcher]

Continuations (which are the coolest things since programming was
invented) were forced out of Stackless Python around the end of version
2.0 for political reasons IIRC. Current Stackless doesn't have
continuation support at all, so any version of micro-threads written to
use continuations is just dead-on-arrival for the current version.
Stackless 1.0 and 2.0 were only for pre-2.3 releases of Python AFAIK.

I would imagine that *someone* has written a Tasklets-based
micro-threading implementation for the new Stackless (after all,
micro-threading is AFAIK *the* major use-case for Stackless (save for
those of us who like to play around with new modes of programming)) and
you'll just need to poke around for that rewrite.

In a former life I got paid to work on extending the continuation-based
micro-thread implementation... continuations are the most awesome toys
you can imagine... we will all mourn their passing in time...

Good luck,
Mike

Matt Leslie wrote:
....

I'm trying to use microthreads under stackless python, since they
sound like exactly what I am after,
....

python23.dll, python23.lib, python23.exp
....

I then downloaded the microthreads package from
http://willware.net:8080/uthread.py
....

ImportError: No module named continuation

....
________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://members.rogers.com/mcfletch/
blog: http://zope.vex.net/~mcfletch/plumbing/

 

[Peter Hansen]

continuations are the most awesome toys
you can imagine... we will all mourn their passing in time...

Will that be _before_, or _after_ we understand what they are? ;-)

-Peter

 

[Donn Cave]

Quoth Peter Hansen <[email protected]>:
| Mike C. Fletcher wrote:
|
| > continuations are the most awesome toys
| > you can imagine... we will all mourn their passing in time...
|
| Will that be _before_, or _after_ we understand what they are? ;-)

I gave them a try, and I mourn them with a touch of relief, I guess.
I'm the guy who won't bother to learn list comprehensions and generally
despises cool gimmicks, but continuations were more than just cool,
they were terribly powerful.

I hooked them into a graphics toolkit wrapper with an event dispatching
model where each window, network connection etc., ran in its own native
thread. No blocking, always ready to respond to a new message. When a
couple of these threads have to interact, they would queue messages back
and forth, and the programming paradigm can get awkward - a lot of state
can pile up as everyone's trying to keep track of their progress through
some kind of procedure.

Well, suppose you send your message and save the whole function state
somewhere, and return to the dispatcher. When the response comes back,
you pick up the function state and continue it, feeding the response in.

Now instead of a tedious kind of state machine, you're writing an plain,
ordinary function that sends messages to its peer and looks at the response,
as though that were all synchronous, and it's so much simpler. Yet the
execution underneath that isn't synchronous at all, because your computation
is suspended in between your send and the response. It really does return
every time it sends a message, it just starts up next time where it left
off.

Of course it's hairy, too. That's where the relief comes in. I'll never
be able to use continuations ... but then, I'll never have to deal with
anyone else's code that uses them.

Donn Cave, (e-mail address removed)

 

[Michele Simionato]

Continuations (which are the coolest things since programming was
invented) were forced out of Stackless Python around the end of version
2.0 for political reasons IIRC. Current Stackless doesn't have
continuation support at all, so any version of micro-threads written to
use continuations is just dead-on-arrival for the current version.
Stackless 1.0 and 2.0 were only for pre-2.3 releases of Python AFAIK.

In a former life I got paid to work on extending the continuation-based
micro-thread implementation... continuations are the most awesome toys
you can imagine... we will all mourn their passing in time...

Good luck,
Mike

Uhm ...

My experience with continuations (in Scheme) is limited, but I really
wonder if continuations have a place in a high level programming language.
I mean, continuations are basic building blocks: you can build on top
of them exception systems, generators, coroutines, microthreads, etc,
and it is an interesting learning experience to understand how this is
done.

However, to pass from continuations to something useful takes a
LONG way. Continuations are a too low level concept. A high level language
should already provide the right high level tools. I mean, the language
designer should implement generators, exception systems, microthreads, etc.
not the application programmer. When you have the high level concepts written
down by others, you don't need continuations.

So I guess now Stackless has microthreads built inside, and the application
programmer is not forced to write them on top of continations.

Speaking in general, generators and the current exception system are more
than enough for my typical needs in Python: which kind of applications do
you have in mind where you would like to have continuations? (a part from
microthreads and things like changing the language introducing new
control structures, etc, all stuff than should not be left to the
application programmer, IMHO).

Michele

 

[Michael Walter]

My experience with continuations (in Scheme) is limited, but I really

wonder if continuations have a place in a high level programming language.
I mean, continuations are basic building blocks: you can build on top
of them exception systems, generators, coroutines, microthreads, etc,
and it is an interesting learning experience to understand how this is
done.

Yeah exactly, they are one reason which allows Scheme to be really
high-level.

They're pretty useful for web applications, too -- the difference to
above-mentioned examples would be that IMHO there is no adequate
alternative to them in current Python.

However, to pass from continuations to something useful takes a
LONG way. Continuations are a too low level concept.

That probably depends on the point of view. I don't think it matters
much whether a specific feature is built-in into the language as such or
part of the/a standard library.

> A high level language should already provide the right high level tools. I mean,
> the language designer should implement generators, exception systems, microthreads, etc.
not the application programmer. When you have the high level concepts written
down by others, you don't need continuations.

Would you have an idea on how a higher-level replacement for
continuations in web application development? I'm excited (and _not_
being any sarcastic or stuff, honestely!).

Speaking in general, generators and the current exception system are more
than enough for my typical needs in Python: which kind of applications do
you have in mind where you would like to have continuations? (a part from
microthreads and things like changing the language introducing new
control structures, etc, all stuff than should not be left to the
application programmer, IMHO).

Web application development involving Continuations (such as in Seaside,
the PST web server etc.).

Cheers,
Michael

 

[Just]

not the application programmer. When you have the high level concepts
written
down by others, you don't need continuations.

Would you have an idea on how a higher-level replacement for
continuations in web application development? I'm excited (and _not_
being any sarcastic or stuff, honestely!).[/QUOTE]

Co-routines.

Just

 

[Michael Walter]

Would you have an idea on how a higher-level replacement for
continuations in web application development? I'm excited (and _not_
being any sarcastic or stuff, honestely!).

Co-routines.[/QUOTE]
Would you care to elaborate? Can you store the state of a coroutine and
resume it multiple times?

Cheers,
Michael

 

[Just]

Would you care to elaborate? Can you store the state of a coroutine and
resume it multiple times?

You can resume a co-routine multiple times just like you can resume
generators mutliple times, but that's something completely different
from restarting a continuation multiple times.

When you mentioned continuations for web programming, I assumed you
meant their ability to turn flow control inside out (allowing to write
event-based networking code in a more natural way). Co-routines are a
much nicer abstraction for that than continuations.

Just

 

[Michael Walter]

You can resume a co-routine multiple times just like you can resume
generators mutliple times, but that's something completely different
from restarting a continuation multiple times.

Well, I think you would want to resume it multiple times based on a
given state (essentially like resuming a stored continuation).

When you mentioned continuations for web programming, I assumed you
meant their ability to turn flow control inside out (allowing to write
event-based networking code in a more natural way). Co-routines are a
much nicer abstraction for that than continuations.

Would you have some links/papers for me in regard to web programming and
coroutines? Some quick googling didn't bring anything up (but maybe I'm
just too tired).

What I meant was using continuations for "linear" code flow, such as
mentioned in
http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/msg02456.html,
described in http://www.double.co.nz/scheme/modal-web-server.html and
applied in Seaside, Borges, PLT Web Server etc.

Cheers,
Michael

 

[Mike C. Fletcher]

....

However, to pass from continuations to something useful takes a
LONG way. Continuations are a too low level concept. A high level language
should already provide the right high level tools. I mean, the language
designer should implement generators, exception systems, microthreads, etc.
not the application programmer. When you have the high level concepts written
down by others, you don't need continuations.

Except we did . When we sat down to start working with
micro-threads, they were (for our purposes) extremely heavy beasts when
they were sleeping and scheduling themselves. So, having a simple
Python implementation built out of continuations, we were able to say
"hey, we can fix that cheaply" and dedicate a few weeks of work to the
project to improve the implementation.

Guido may be omniscient , but I don't think he's achieved
omnipotence. He and the python-dev crowd can't, and *won't* implement
many things that people writing frameworks *need* (and really, in many
cases, they shouldn't be doing it anyway until the idea proves itself).
Having a building block such as continuations exposed allows framework
developers/meta-programmers the chance to experiment with new
flow-control structures and to do that experimentation *in a high-level
language* (does this sound familiar to all the PyPy peoples).

So I guess now Stackless has microthreads built inside, and the application
programmer is not forced to write them on top of continations.

Sure, but it was never the app programmer that did it anyway, so let's
leave them out of this. Using Python does not automatically mean you
are an applications programmer rather than a framework developer or
meta-programmer (btw, nothing wrong with being an apps programmer, it's
just a different type of work). Python isn't *just* a scripting
language either .

Speaking in general, generators and the current exception system are more
than enough for my typical needs in Python: which kind of applications do
you have in mind where you would like to have continuations? (a part from
microthreads and things like changing the language introducing new
control structures, etc, all stuff than should not be left to the
application programmer, IMHO).

Why make that kind of experimentation difficult just to erect a barrier
between user and developer? Python isn't a place to keep every powerful
tool from the plebeian masses just because they might hurt themselves.
Heck, if we're going to do that, what the heck are we exposing
metaclasses for (and really, would the two of us be happy if they took
away our metaclasses)? There are framework developers/meta-programmers
using Python who just aren't interested in working in C code any more.
Giving them tools is *not* a bad thing.

Regarding applications; restartable exceptions, micro-threads,
long-running-calculation restarts, migrating running code across
machines, and creating new methods of parsing all seem like experiments
that would benefit from having continuations available to ease
implementation. Any of those is a thing that an app programmer might
want/need, and be willing to have a meta-programmer create for them...
why force that work into C code needlessly?

However, continuations are dead, and though I may mourn them, I'm not
interested in trying to resurrect them. At the moment my company's
current contracts would only be interested in are micro-threads; and
even there, we aren't really interested enough to switch from our
already-built frameworks; just too much inertia. Too bad, because a
micro-threaded networking environment is the closest thing I've ever
seen to the promised land for networking... garn I miss working on that.

Peace all,
Mike

________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://members.rogers.com/mcfletch/
blog: http://zope.vex.net/~mcfletch/plumbing/

 

[Michele Simionato]

Regarding applications; restartable exceptions, micro-threads,
long-running-calculation restarts, migrating running code across
machines, and creating new methods of parsing all seem like experiments
that would benefit from having continuations available to ease
implementation. Any of those is a thing that an app programmer might
want/need, and be willing to have a meta-programmer create for them...
why force that work into C code needlessly?

Ok, I see your point, so let me restate my doubts in this form:
are you sure that you need to go to such a low level as continuations?
Wouldn't generators/coroutines be enough to fullfill your needs?

---

On a different note, I could notice that giving so much power to
metaprogrammers may have unfortunate results. For instance, we already
have three different kind of interfaces in three different frameworks
(Zope, Twisted, Peak) whereas there should be only one obvious way.
So,
I mantain that certain things should be left to the language designer
(i.e. Guido) and not to the metaprogrammer.

Otherwise you ends up as in Scheme, where essentially everyone can be
the designer of his own little language, with the disadvantages that
you
can see. I think metaprogrammers should be more humble, and think ten
times before using metaprogramming features to change the language.
But sometime the temptation is irrestible

Michele Simionato

 

[Andrew Bennetts]

]

On a different note, I could notice that giving so much power to
metaprogrammers may have unfortunate results. For instance, we already
have three different kind of interfaces in three different frameworks
(Zope, Twisted, Peak) whereas there should be only one obvious way.

FWIW, Twisted will be using Zope 3's interfaces framework soon, which will
leave just Zope and PEAK.

-Andrew.

 

[Michele Simionato]

FWIW, Twisted will be using Zope 3's interfaces framework soon

Aha! It seems clear that I have to find the time to start studying
Zope interfaces then ....

Michele

 

[Michele Simionato]

What I meant was using continuations for "linear" code flow, such as
mentioned in
http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/msg02456.html,
described in http://www.double.co.nz/scheme/modal-web-server.html and
applied in Seaside, Borges, PLT Web Server etc.

This is a most interesting reference, thanks! Incidentally, I am in the process
of learning Zope TAL/TALES/METAL, which I do not like that much; I would
rather prefer to generate HTML pages with Lisp macros and use real
s-expressions instead of XML/HTML, but I can't :-(
Lisp/Scheme is much more suitable for web applications (in principle)
than any other language I know, it is unfortunate that (in practice) it
is not that used ...


Michele Simionato

 

[Ville Vainio]

Michele> of learning Zope TAL/TALES/METAL, which I do not like
Michele> that much; I would rather prefer to generate HTML pages
Michele> with Lisp macros and use real s-expressions instead of
Michele> XML/HTML, but I can't :-(

What would you do with macros? The resulting s-exprs would still need
to be evaluated to get the html, so the performace wouldn't be
better...

And you can use s-exprs in python too:

page = [
[ head ,
[ title, "my page"] ],
[ body,
[ heading, "welcome to my page"],
[ para , "paragraph1 blah blah"]
[ para , "paragraph2 more blah blah "]]
]

Further elaboration on the idea at

http://www.students.tut.fi/~vainio24/pysexpr/

 

[Michael Walter]

Michele> of learning Zope TAL/TALES/METAL, which I do not like
Michele> that much; I would rather prefer to generate HTML pages
Michele> with Lisp macros and use real s-expressions instead of
Michele> XML/HTML, but I can't :-(
What would you do with macros? [..]

Syntactic abstraction. I would love to see macros in Python (actually
working on some ideas for adding them to Python).

Cheers,
Michael

 

[Valentino Volonghi aka Dialtone]

This is a most interesting reference, thanks! Incidentally, I am in the process
of learning Zope TAL/TALES/METAL, which I do not like that much; I would
rather prefer to generate HTML pages with Lisp macros and use real
s-expressions instead of XML/HTML, but I can't :-(
Lisp/Scheme is much more suitable for web applications (in principle)
than any other language I know, it is unfortunate that (in practice) it
is not that used ...

Hi Michele .

If all you need is s-expr you maybe can look at Nevow (pronounced
'nuevo'), which is the new web toolkit built on top of twisted.web.

It has a template syntax that is similar to ZPT.

And it has stan which is an s-expr-like syntax.

Here is a little helloworld example:

docFactory = rend.stan(
T.html[
T.head[
T.title['Hello'],
],
T.body[
T.p['Welcome to the wonderful world of Nevow!'],
],
]
)

More infos at www.nevow.com or in the nevow wiki:
http://divmod.org/users/wiki.twistd/nevow/moin.cgi/FrontPage

or in irc in #twisted.web

 

[Ville Vainio]

Valentino> And it has stan which is an s-expr-like syntax.

Valentino> Here is a little helloworld example:

Valentino> docFactory = rend.stan(
Valentino> T.html[
Valentino> T.head[
Valentino> T.title['Hello'],
Valentino> ],
Valentino> T.body[
Valentino> T.p['Welcome to the wonderful world of Nevow!'],
Valentino> ],
Valentino> ]
Valentino> )

Just a thought: it would be more s-expr like if it just had the
"function"ish part inside the list in the normal prefix notation:

[T.html,
[T.head,
[T.title, 'Hello']],
[T.body, [T.p, 'Welcome...]]]

In a way this seems slightly clearer (especially the , chars n/w ]
chars seem very suspicous to me.

 

[Valentino Volonghi aka Dialtone]

Just a thought: it would be more s-expr like if it just had the
"function"ish part inside the list in the normal prefix notation:

[T.html,
[T.head,
[T.title, 'Hello']],
[T.body, [T.p, 'Welcome...]]]

In a way this seems slightly clearer (especially the , chars n/w ]
chars seem very suspicous to me.

Sure, but I think this is less pythonic than the nevow version, and I
find that version also clearer, but I also think that this is a matter
of tastes .