Generator-based microthreads and games

Discussion in 'Python' started by neokosmos, Aug 11, 2006.

  1. neokosmos

    neokosmos Guest

    I've seen various generator-based microthread implementations online,
    but I've been wondering: has anyone used microthreads in this manner in
    a game environment? Note, I am emphatically *not* referring to
    Stackless, which I know has been used in a production game environment;
    this post is referring strictly to the standard Python interpreter, (v.
    2.4 or 2.5 as you wish).

    I intend to try a little cooperative microthreading implementation in
    my game server to avoid threading issues and possibly simplify the
    programming model. This seems particularly attractive given the
    enhancements Python 2.5 brings to generators.

    If anyone else has tried this, what I would like to know is this:

    * How many microthreads were you able to run concurrently with decent
    * How much computation is feasible to carry out in a microthread?
    (Certainly, this goes hand in hand with the previous question, of
    * What sort of scheduler did you use?
    * Was it worth it? By "worth it," I mean in terms of performance and
    ease of implementation.

    I'll no doubt have further questions later after I actually begin
    implementing this stuff, but I suppose that will do for now.

    neokosmos, Aug 11, 2006
    1. Advertisements

  2. neokosmos

    Michael Guest

    Several. I'll largely talk about our experience with Kamaelia below, but
    there's also LGT which is also pretty interesting.


    The reason I'll only talk about Kamaelia is because I'm the lead developer
    on that project, and because I think it's a fun way to write this sort of
    thing :)

    Naturally I'm biassed, bear that in mind :)
    Numbered in the thousands. It's been a while since we've done any benchmarks
    however. (Most of our use of generators has revolved around network
    systems, but we use Pygame fairly heavily)
    As much as you like, however the more you do, the less microthreads you can
    run (as you say, this goes hand in hand).

    For short examples of what we're using generators for, take a peek here:
    We use a simple round robin scheduler.
    Absolutely. There's a knock on of python style generators that people don't
    tend to realise: it forces a your generators to be simple and focussed.

    Given we augment generators with inboxes & outboxes for communications we
    find we get higher levels of reuse of many components than we would
    normally expect. We embed the generator in a class, giving far more
    flexible communications that python 2.5's newly added facilities (this
    approach works happily with python 2.2 onwards, meaning we've versions of
    Kamaelia that even run on nokia phones.)

    The only game we've written in Kamaelia at present is a simple bouncing cats
    game (aimed at 2-4 year olds :), which adds bouncing cats, and removes
    them to a noise. However the only slow down I've really seen with that is
    more due to the speed at which pygame & SDL can update the display AFAICT.
    Even if you're not interested in Kamaelia (then again you might - we find it
    incredibly useful), you may find our tutorial useful - it shows you how to
    go about building a simple scheduler and simple systems of communicating
    generators in easy simple steps. (Tested on over a dozen novice python
    programmers now).


    Systems we're using this for:
    * Collaborative whiteboarding (every client is also a server) including
    * Transcoding PVR for making a record of transmission of all BBC TV
    channels, including collection of EPG data.
    * Under development - video annotation/editing. (for allowing
    collaborative discussion of how to post process video)

    I suppose knocking up a simple game might be a nice thing to do as well :)

    For what it's worth, a focus in Kamaelia is on making it harder to do dumb
    things when doing things in parallel - as you are doing when using threads
    or generators. (That's the reason for the inbox/outbox metaphor - take it
    out an inbox, it's yours to do what you want, put it in an outbox, you're
    not allowed to do anything with it any more).

    As a result generator based components and thread based components can be
    mixed quite happily in Kamaelia, since you have explicit data handoff. This
    might not seem immediately important to you, but as a system grows, you
    need a way of handing blocking calls (since otherwise your generators and
    scheduler stall), and putting those in a thread is a common idea. Doing
    this safely is normally somewhat more complex than: (including imports)

    import Axon
    from Kamaelia.Chassis.Pipeline import pipeline
    from Kamaelia.Util.Console import ConsoleEchoer

    class ConsoleReader(Axon.ThreadedComponent.threadedcomponent):
    def main(self):
    while 1:
    data = raw_input(">>> ") # block waiting for user
    self.send(data, "outbox") # send on to next thing

    ConsoleReader(), # Threaded Component
    ConsoleEchoer(), # Generator Component

    Just as an indication of the sorts of things you can do with Kamaelia, we've
    had some Google Summer of Code students working on a small selection of
    Kamaelia related projects this summer. Perhaps the most relevant to you
    might be the 3D work, which makes it relatively simple to create 3D objects
    and then bounce them round the screen and interact with them.

    Two of the students have blogs which follow their progress which you can
    find here, which might also be useful when considering whether generators
    are a good approach :)

    Anyway, I've rambled on far too long, and now well off topic for the
    questions you were asking, so I'll be quiet :)

    Have fun, and IMO, the generator approach (using Kamaelia, LGT, or home
    rolled), is very much worth going down.

    Best Regards,

    Michael, Aug 11, 2006
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.