creating generators from function

Discussion in 'Python' started by Simon Wittber, Dec 8, 2004.

  1. I use a coroutine/generator framework for simulating concurrent processes.

    To do this, I write all my functions using the general form:

    while True:
    do stuff
    yield None

    To make these generator functions compatible with a standard thread
    interface, I attempted to write a decorator which converts a standard
    function into a generator function (code attached below).
    Unfortunately, I received an exception, before I could test if my idea
    would work:

    TypeError: cannot create 'generator' instances

    I guess I can go the other way, and convert a generator into a
    function, but that just doesn't seem right.

    Is there anyway solve the problem described?

    Sw.


    from opcode import opmap
    import types
    globals().update(opmap)

    def generatorize(f):
    co = f.func_code
    nc = [ord(x) for x in co.co_code]
    for op,i in enumerate(nc):
    if op == RETURN_VALUE:
    nc = YIELD_VALUE
    newcode = ''.join([chr(x) for x in nc])
    codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
    co.co_flags, newcode, co.co_consts, co.co_names,
    co.co_varnames, co.co_filename, co.co_name,
    co.co_firstlineno, co.co_lnotab, co.co_freevars,
    co.co_cellvars)

    print types.GeneratorType(f)(codeobj, f.func_globals, f.func_name,
    f.func_defaults, f.func_closure)
    return f

    @generatorize
    def f():
    while True:
    return 1

    def g():
    while True:
    yield 1

    print g()
    print f()
     
    Simon Wittber, Dec 8, 2004
    #1
    1. Advertising

  2. Simon Wittber wrote:
    > I use a coroutine/generator framework for simulating concurrent processes.
    >
    > To do this, I write all my functions using the general form:
    >
    > while True:
    > do stuff
    > yield None
    >
    > To make these generator functions compatible with a standard thread
    > interface, I attempted to write a decorator which converts a standard
    > function into a generator function (code attached below).

    [snip]
    > @generatorize
    > def f():
    > while True:
    > return 1
    >
    > def g():
    > while True:
    > yield 1
    >
    > print g()
    > print f()


    I'm a little confused as to what you're trying to do here. The f()
    function, in particular, doesn't make much sense -- the 'while True'
    doesn't do anything since the 'return 1' is executed on the first time
    through the loop. If you really do want to turn your functions into
    generators of the form:

    while True:
    do stuff
    yield None

    why can't you just do:

    >>> def generatorize(f):

    .... def new_f(*args, **kwds):
    .... while True:
    .... f(*args, **kwds)
    .... yield None
    .... return new_f
    ....
    >>> @generatorize

    .... def f():
    .... return 1
    ....
    >>> i = f()
    >>> print i.next()

    None
    >>> print i.next()

    None

    Basically, I've assumed that 'do stuff' is the function that's being
    wrapped.

    Steve
     
    Steven Bethard, Dec 8, 2004
    #2
    1. Advertising

  3. Simon Wittber

    Peter Otten Guest

    Simon Wittber wrote:

    > I use a coroutine/generator framework for simulating concurrent processes.
    >
    > To do this, I write all my functions using the general form:
    >
    > while True:
    > do stuff
    > yield None
    >
    > To make these generator functions compatible with a standard thread
    > interface, I attempted to write a decorator which converts a standard
    > function into a generator function (code attached below).
    > Unfortunately, I received an exception, before I could test if my idea
    > would work:
    >
    > TypeError: cannot create 'generator' instances
    >
    > I guess I can go the other way, and convert a generator into a
    > function, but that just doesn't seem right.
    >
    > Is there anyway solve the problem described?


    Generators are just functions with an extra flag set. I tried the following
    modification of your code:

    <code>
    from opcode import opmap
    globals().update(opmap)

    def _f(): pass
    function = type(_f)
    del _f

    def generatorize(f):
    co = f.func_code
    nc = [ord(x) for x in co.co_code]
    for i, op in enumerate(nc[:-1]):
    if op == RETURN_VALUE:
    nc = YIELD_VALUE
    newcode = ''.join([chr(x) for x in nc])

    codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
    co.co_flags + 32, newcode, co.co_consts,
    co.co_names,
    co.co_varnames, co.co_filename, co.co_name,
    co.co_firstlineno, co.co_lnotab, co.co_freevars,
    co.co_cellvars)

    return function(codeobj, f.func_globals, f.func_name,
    f.func_defaults, f.func_closure)

    @generatorize
    def f():
    while True:
    return 1

    def g():
    while True:
    yield 1

    print g()
    print f()

    import itertools
    print list(itertools.islice(f(), 5))
    </code>

    The hardest part was to find the enumerate() bug. I agree with Steven that
    converting between functions and generators by replacing return with yield
    and vice versa is hardly ever useful, though.

    Peter
     
    Peter Otten, Dec 8, 2004
    #3
  4. > I'm a little confused as to what you're trying to do here. The f()
    > function, in particular, doesn't make much sense


    I see I am misunderstood. The example was contrived, and it seems, incorrect.

    I simulate threads, using generator functions and a scheduler.

    My implementation lives here: http://metaplay.com.au/svn/LGT/LGT/nanothreads.py

    If the same functions were to run inside a normal thread, they would
    need to loop forever, rather than yield None on each iteration. I see
    now that replacing return with yield won't work, a yield instruction
    needs to be inserted somewhere.

    I guess this changes my question to: Can I convert a function
    containing an infinite loop, into a generator which will yield on each
    iteration?

    Sw.
     
    Simon Wittber, Dec 8, 2004
    #4
  5. Simon Wittber wrote:
    > I guess this changes my question to: Can I convert a function
    > containing an infinite loop, into a generator which will yield on each
    > iteration?


    Why don't you just use threads? It would probably be easier in the long run...

    --
    Timo Virkkala
     
    Timo Virkkala, Dec 8, 2004
    #5
  6. Simon Wittber

    Terry Reedy Guest

    "Simon Wittber" <> wrote in message
    news:...
    > I guess this changes my question to: Can I convert a function
    > containing an infinite loop, into a generator which will yield on each
    > iteration?


    A function containing an infinite loop will never return, so it is bugged
    and useless unless it does has an external effect with something like
    'print xxx' within the loop. Even then, it depends on 'while True:'
    actually meaning 'while <external stop signal (control-C, reset, power off)
    not present>'. The obvious change for a buggy function is to insert a
    yield statement. For the implicit conditional with print, change 'print'
    to 'yield'.

    Note 1. Many algorithm books, especially when using pseudocode, use 'print
    sequence_item' to avoid language specific details of list construction.
    One of the beauties of Python2.2+ is that one can replace simply 'print'
    with 'yield' and have a working algorithm.

    Note 2. Delving deep into CPython internals, as you are, is CPython hacking
    rather than Python programming per se. Fun, maybe, but I wonder if this is
    really to best way to do what you want to do.

    Terry J. Reedy
     
    Terry Reedy, Dec 8, 2004
    #6
  7. > A function containing an infinite loop will never return, so it is bugged
    > and useless unless it does has an external effect with something like
    > 'print xxx' within the loop.


    I thought the idiom:

    while True:
    #code which iterates my simulation
    if condition: break

    wat quite normal. This is the style loop I am using.

    > Note 2. Delving deep into CPython internals, as you are, is CPython hacking
    > rather than Python programming per se. Fun, maybe, but I wonder if this is
    > really to best way to do what you want to do.


    I am trying to allow two modes of concurrency, (threading, and
    pseudo-threading using generators) without having to change my
    application code. If either CPython or Python hacking can provide a
    solution, I'm not sure, but I would be happy with either.

    Sw.
     
    Simon Wittber, Dec 9, 2004
    #7
  8. Simon Wittber

    Terry Reedy Guest

    "Simon Wittber" <> wrote in message
    news:...
    > I thought the idiom:
    >
    > while True:
    > #code which iterates my simulation
    > if condition: break
    >
    > wat quite normal. This is the style loop I am using.


    Yes, quite normal. This is Python's version of do...until, synthesized
    from while and break. As most people use the term, this is *not*
    structurally an infinite loop (meaning no break) and also not functionally
    so unless the condition is such that it never breaks either for some or all
    inputs to the function.

    > I am trying to allow two modes of concurrency, (threading, and
    > pseudo-threading using generators) without having to change my
    > application code. If either CPython or Python hacking can provide a
    > solution, I'm not sure, but I would be happy with either.


    One Python level approach would be to quote the text of the function defs
    that need to switch and then either programmatically edit them or not
    before exec-ing them. Also messy, but more robust against version and
    implementation changes.

    Terry J. Reedy
     
    Terry Reedy, Dec 9, 2004
    #8
  9. Simon Wittber

    Mike Meyer Guest

    Simon Wittber <> writes:

    >> A function containing an infinite loop will never return, so it is bugged
    >> and useless unless it does has an external effect with something like
    >> 'print xxx' within the loop.

    >
    > I thought the idiom:
    >
    > while True:
    > #code which iterates my simulation
    > if condition: break
    >
    > wat quite normal. This is the style loop I am using.


    I think it's a bit abnormal, because you have to scan the loop body
    for breaks. I tend to write:

    condition = True
    while not condition:
    #code which iterates my simulation

    But that's just me.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Dec 9, 2004
    #9
  10. Simon Wittber

    Isaac To Guest

    >>>>> "Mike" == Mike Meyer <> writes:

    Mike> I think it's a bit abnormal, because you have to scan the
    Mike> loop body for breaks. I tend to write:

    Mike> condition = True
    Mike> while condition: # corrected
    Mike> #code which iterates my simulation

    Then you'd have to scan the loop body to find the location where
    condition is set, which is more difficult than locating breaks
    normally. If you get a break, you really breaks. If you set
    condition to False, you still might be modifying it to True later in
    your code. And of course, most editors will highlight the "break" for
    you, while no editor will highlight for you the "condition" variable
    that you are staring at.

    Regards,
    Isaac.
     
    Isaac To, Dec 9, 2004
    #10
  11. Simon Wittber

    Mike Meyer Guest

    Isaac To <> writes:

    >>>>>> "Mike" == Mike Meyer <> writes:

    >
    > Mike> I think it's a bit abnormal, because you have to scan the
    > Mike> loop body for breaks. I tend to write:
    >
    > Mike> condition = True
    > Mike> while condition: # corrected
    > Mike> #code which iterates my simulation
    >
    > Then you'd have to scan the loop body to find the location where
    > condition is set, which is more difficult than locating breaks
    > normally. If you get a break, you really breaks. If you set
    > condition to False, you still might be modifying it to True later in
    > your code. And of course, most editors will highlight the "break" for
    > you, while no editor will highlight for you the "condition" variable
    > that you are staring at.


    Checking the condition and using a break at the bottom of the loop
    doesn't change any of these things, so my version is less work. In
    normal use, the "condition" is something that falls out of the
    operation - checking for an empty string on file in put, for example
    - and you can set the variable that is being tested to a value that
    causes the condition to be true.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Dec 9, 2004
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rhino
    Replies:
    4
    Views:
    5,785
    Roedy Green
    Jan 13, 2006
  2. Pavel

    Code Generators

    Pavel, May 14, 2006, in forum: Java
    Replies:
    7
    Views:
    727
    dimitar
    May 19, 2006
  3. Mark

    Memoizing Generators

    Mark, Jul 8, 2003, in forum: Python
    Replies:
    2
    Views:
    352
  4. Duncan Booth

    generators improvement

    Duncan Booth, Aug 19, 2003, in forum: Python
    Replies:
    4
    Views:
    295
    Oleg Leschov
    Aug 20, 2003
  5. J. A. Aczel
    Replies:
    1
    Views:
    613
    Lawrence D'Oliveiro
    Sep 1, 2007
Loading...

Share This Page