BBC R&D White Paper on Kamaelia Published (Essentially a framework using communicating python genera

Discussion in 'Python' started by Michael Sparks, Jun 11, 2005.

  1. Hi,


    I'm posting a link to this since I hope it's of interest to people here :)

    I've written up the talk I gave at ACCU Python UK on the Kamaelia Framework,
    and it's been published as a BBC R&D White Paper and is available here:

    * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

    Essentially it provides a means of a) communicating to python generators b)
    linking these communications between generators (the result is calling these
    components), c) building interesting systems using it. (Next release will
    include some interesting things like a shell, introspection tool, and
    pygame related components)

    Essentially the key tenet of the project is that of trying to make scalable
    concurrent and network systems easy to build and reuse.

    We're using it for network research purposes, but the system should be
    sufficiently general to be of use for all sorts of uses. (Though for
    production systems I'd probably recommend Twisted over Kamaelia for now,
    just for perspective :)

    Any comments, criticisms/etc very welcome!


    Michael.
    --
    http://kamaelia.sourceforge.net/
     
    Michael Sparks, Jun 11, 2005
    #1
    1. Advertising

  2. Michael Sparks

    Guest


    > I'm posting a link to this since I hope it's of interest to people here :)
    >
    > I've written up the talk I gave at ACCU Python UK on the Kamaelia Framework,
    > and it's been published as a BBC R&D White Paper and is available here:
    >
    > * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml


    I enjoyed the paper. I don't have time to play with the code right
    now, but it sounds like fun. I do have one question -- in practice,
    how difficult have you found it that Python generators don't maintain
    separate stacks?

    Regards,
    Pat
     
    , Jun 13, 2005
    #2
    1. Advertising

  3. wrote:

    >
    >> I've written up the talk I gave at ACCU Python UK on the Kamaelia
    >> Framework, and it's been published as a BBC R&D White Paper and is
    >> available here:
    >>
    >> * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

    > I enjoyed the paper.


    I'm pleased to hear that and I hope the ideas are useful at some point in
    time! (directly or indirectly :)

    > I don't have time to play with the code right now, but it sounds like fun.


    > I do have one question -- in practice, how difficult have you found it
    > that Python generators don't maintain separate stacks?


    I'm not sure I understand the question! (I think you're either coming from a
    false assumption or mean something different) I can see 3 possible
    answers...

    How difficult is it to build systems using them? Well, we managed to get a
    novice programmer up to speed and happy with python and using the core
    ideas in Kamaelia in a 2 week period. And over the 3 month period he was
    with us he implemented a small system essentially demoing previewing
    streamed content from a PVR on a Series 60 mobile (and on a desktop/laptop
    PC). (Of course that could be because he was a natural, *or* because it can
    be fairly easy - or both)

    Regarding the idea that python generators stacks "collide" aren't separate.
    If you mean an implementation detail I'm not aware of, it doesn't cause any
    problems at all. If you are coming from the assumption that they share the
    same state/stack frame...

    Consider the example generator function:

    >>> def fib():

    .... a,b = 0,1
    .... while 1:
    .... yield b
    .... a,b = b, a+b
    ....

    Then call this twice yielding two different generator objects:
    >>> G = fib()
    >>> G

    <generator object at 0x403b006c>
    >>> H = fib()
    >>> H

    <generator object at 0x403b03cc>

    If they shared the same stack (by which I assume you mean stack frame), then
    advancing one of these should advance both. Since in practice they run
    independently you get the following behaviour:
    >>> G.next()

    1
    >>> G.next()

    1
    >>> G.next()

    2
    >>> G.next()

    3
    >>> G.next()

    5
    >>> H.next()

    1
    >>> H.next()

    1
    >>> H.next()

    2
    >>> G.next()

    8

    ie both generators "run" independently with different local values for a,b
    since they *don't* share stack frames. (Also as I understand it the stack
    frame associated with each generator is effectively put aside between
    invocations)

    As a result from that interpretation of your question (that it's based on
    the mistaken assumption that state is shared between generators), we
    haven't found it a problem at all because it's a false assumption - the
    generators naturally don't share stacks.


    The alternative is that you're talking about the fact that if you put
    "yield" in a def statement that it forces that definition to define a
    generator function. (ie you can't nest a yield)

    For example in some TCP client code we have:
    def runClient(self,sock=None):
    try:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM); yield 1
    try:
    sock.setblocking(0); yield 2
    try:
    while not self.safeConnect(sock,(self.host, self.port)):
    yield 1
    .... snip ...

    It would be a lot more natural (especially with full co-routines) to have:

    def runClient(self,sock=None):
    try:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM); yield 1
    try:
    sock.setblocking(0); yield 2
    try:
    self.safeConnect(sock,(self.host, self.port)):
    .... snip ...

    And to still have runClient be the generator, but have safeConnect doing the
    yielding. (You can do this using Greenlets and/or Stackless for example)

    However in practice it seems that this limitation is in fact more of a
    strength - it encourages smaller simpler generators whose logic is more
    exposed. Furthermore we very rarely have to nest calls to generators because
    the way we deal with multiple generators is to set them running
    independently and wait for communications between them.

    These small generators also seem to lend themselves better to composition.
    For example:

    clientServerTestPort=1500
    pipeline(TCPClient("127.0.0.1",clientServerTestPort),
    VorbisDecode(),
    AOAudioPlaybackAdaptor()
    ).run()

    If you allow nested calls, it's perhaps natural to couple the last two
    together as one component - but that is ultimately limiting - because each
    component would then be trying to do too much.

    Finally, in practice, it's pretty easy to write new components - we tend to
    focus on the functionality we want, find it's main loop, add in yields in
    key locations to effectively time slice it to turn it into a generator.

    That then becomes the core of the component, and we then remove code that
    deals with direct input/output (eg from files) and recv/send to/from
    in/out-boxes.

    It leads to a fairly natural process for creating generator based
    components. (And can also lead to a fast assimilation process for
    existing python based code, when needed) There's a walk through the
    creation of multicast handling components here:
    http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi?rm=viewpost&postid=1113495151

    It's a bit long winded - mainly due to repeatedly copying code to highlight
    differences inplace - but should show that in practice it's a number of
    small simple incremental steps. (Showing how we can go from standalone test
    programs to generators to components) It also shows how the system can end
    up causing you to create more general code than originally intended, which
    is also more compact and obvious.

    I suppose the short answer to your question though is that the defined
    limitations of python's (simple) generators do in fact turn out to be a
    strength IMO.

    Anyhow, I hope that's answered your original question ? (or hopefully
    explained why I found your question unclear!)

    Best Regards,


    Michael
    --
    http://kamaelia.sourceforge.net/
     
    Michael Sparks, Jun 13, 2005
    #3
  4. Michael Sparks

    Guest

    Yes, the question was about the lack of full coroutine support in
    Python, and whether in practice that was a problem or not. It's
    interesting that you think it may actually be a strength -- I'll have
    to mull that over.

    Thanks!
    Pat
     
    , Jun 13, 2005
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rob R. Ainscough

    Are ASP.NET user interfaces essentially dead now?

    Rob R. Ainscough, Dec 8, 2005, in forum: ASP .Net
    Replies:
    24
    Views:
    1,007
    Brendan Reynolds
    Dec 11, 2005
  2. rickman
    Replies:
    9
    Views:
    904
    Colin Paul Gloster
    Jul 21, 2009
  3. bahr

    Genera question about a SELECT dropdown list

    bahr, Jul 26, 2005, in forum: ASP .Net Web Controls
    Replies:
    2
    Views:
    127
    Guest
    Jul 28, 2005
  4. Replies:
    1
    Views:
    213
    Simon Cropper
    Jul 31, 2012
  5. Replies:
    0
    Views:
    216
Loading...

Share This Page