delay and force in Python

[Will Stuyvesant]

Here is a question for people who are more comfortable than
I am with new Python stuff like generators.

I am having fun implementing things from the Wizard book
(Abelson, Sussman, "Structure and Interpretation of Computer
Programs") in Python. In chapter 3.5 it is about streams as
delayed lists.

Streams are interesting because they are to lists like
xrange is to range. Could save a lot on memory and
computations.

Below is my straight translation from Scheme code in the
Wizard book to Python. It works, but now I want to rewrite
the delay and force functions using Python's new stuff,
generators or iterators or what-have-you. I have the
feeling that that is possible, can you do it?

The program below creates a stream with the numbers 1..995
and then filters the stream, keeping only the even numbers,
and then prints the second number in the stream (implemented
as the first number of the tail, just like in the 3.5
Section in the Wizard book).


.. # file: teststreams.py
..
.. def delay(exp): return lambda: exp
..
.. def force(o): return o()
..
.. def cons_stream(a,b): return [a, delay(b)]
..
.. def stream_hd(s): return s[0]
..
.. # we use tl for cdr
.. def tl(x):
.. if len(x) == 2: return x[1]
.. else: return x[1:]
..
.. def stream_tl(s): return force(tl(s))
..
.. def stream_enumerate_interval(low, high):
.. if low > high:
.. return None
.. else:
.. return cons_stream(
.. low,
.. stream_enumerate_interval(low+1, high))
..
.. def stream_filter(pred, s):
.. if s is None:
.. return None
.. elif pred(stream_hd(s)):
.. return cons_stream(
.. stream_hd(s),
.. stream_filter(pred, stream_tl(s)))
.. else:
.. return stream_filter(pred, stream_tl(s))
..
.. def isEven(n): return n % 2 == 0
..
.. print stream_hd(stream_tl(stream_filter(
.. isEven,
.. stream_enumerate_interval(1,995))))
.. # 4



Something else: this crashes with a "maximum recursion reached"
.. print stream_enumerate_interval(1,998)

while this does not crash
.. print stream_enumerate_interval(1,900)
this means Python has a maximum of something like 900
recursions?

 

[floydophone]

Check this out:

def mygen():
for i in xrange(0, 10):
print "about to yield",i
yield i

g = mygen()
print "calling next()", g.next()
print "calling next()", g.next()
print "calling next()", g.next()

 

[Benji York]

Streams are interesting because they are to lists like
xrange is to range. Could save a lot on memory and
computations.

I think you're looking for generators.

Below is my straight translation from Scheme code in the
Wizard book to Python.

Something else: this crashes with a "maximum recursion reached"
. print stream_enumerate_interval(1,998)

Unlike Scheme, Python isn't designed for heavily recursive algorithms.
Here's a more Pythonic equivalent of your code:

def count(start, stop):
i = start
while i < stop:
yield i
i += 1

def even(gen):
for x in gen:
if x % 2 == 0:
yield x

numbers = even(count(1, 999))
first = numbers.next()
second = numbers.next()

print second

 

[Nick Coghlan]

The program below creates a stream with the numbers 1..995
and then filters the stream, keeping only the even numbers,
and then prints the second number in the stream (implemented
as the first number of the tail, just like in the 3.5
Section in the Wizard book).

How's this:

Py> from itertools import islice
Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Breaking it into pieces:

Py> from itertools import islice
Py> stream = (x for x in xrange(1, 996) if x % 2 == 0)
Py> second_item = islice(stream, 1, 2).next()
Py> print second_item
4

And most of the stream hasn't been consumed yet:
Py> print stream.next()
6
Py> unconsumed = list(stream)
Py> len(unconsumed)
494

And this version has no problem with recursion limits:
Py> print islice((x for x in xrange(1, sys.maxint) if x % 2 == 0), 1, 2).next()
4

(xrange can't handle Python longs, unfortunately, so we *are* constrained by
sys.maxint. However, since my machine only has half a gig of RAM, the above is
still a damn sight quicker than the equivalent list comprehension would be!)

Something else: this crashes with a "maximum recursion reached"
. print stream_enumerate_interval(1,998)

while this does not crash
. print stream_enumerate_interval(1,900)
this means Python has a maximum of something like 900
recursions?

The CPython implementation is limited by the stack size allocated by the C
runtime library. The exact recursion limit is platform dependent, but something
around 1000 sounds fairly normal.

Cheers,
Nick.

 

[Dave Benjamin]

. def delay(exp): return lambda: exp

If you look at the definition of "delay" in SICP, you'll notice that
it's defined as "syntax sugar", in other words, a macro. Since Python
does not have macros, you'll have to just use "lambda", because by
defining "delay" as a function, you're forcing the expression "exp" to
evaluate immediately. In other words, theres a crucial difference between::

delay(sys.stdout.write('evaluated\n'))

and::

lambda: sys.stdout.write('evaluated\n')

If you type in those two snippets, you'll see what I mean.

Coincidentally, I attempted a similar experiment just a couple of days
ago, and here's what I came up with::

# Stream.py - Stream class inspired by SICP

class Stream(object):
pass

class EndOfStream(Exception):
pass

class Item(Stream):
def __init__(self, current, nextf):
self.current = current
self.nextf = nextf

next = property(lambda self: self.nextf())

def fold(self, f, init):
return f(self.current, self.next.fold(f, init))

def __getitem__(self, n):
if n == 0:
return self.current
else:
return self.next[n - 1]

def __str__(self):
return '<Stream.Item: %s, ...>' % self.current
__repr__ = __str__

class End(Stream):
def fold(self, f, init):
return init

def __getitem__(self, n):
raise EndOfStream()

def _fail(self):
raise EndOfStream()

current = property(_fail)
next = property(_fail)

def __str__(self):
return '<Stream.End>'
__repr__ = __str__

Here's how it works::

Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "Stream.py", line 37, in _fail
raise EndOfStream()
Stream.EndOfStream ... return Stream.Item(n, lambda: evens(n + 2))
... 4

I don't know if this is useful or not, but it was an interesting
exercise nonetheless.

Dave

 

[Dave Benjamin]

Something else: this crashes with a "maximum recursion reached"
. print stream_enumerate_interval(1,998)

while this does not crash
. print stream_enumerate_interval(1,900)
this means Python has a maximum of something like 900
recursions?

See "sys.getrecursionlimit()" and "sys.setrecursionlimit()". The lack of
tail-call optimization means you probably won't get very far with this
approach (nor with mine).

Dave

 

[Will Stuyvesant]

Yes you are right, if you just want to carry an expression
around then lambda does it; but delay was not intended as a
top-level function. Perhaps you think that my silly stream
implementation in the original post builds the whole list,
but it does not:

o = stream_enumerate_interval(11,121)
print o

f = o[1]
r = f()
print r

[12, <function <lambda> at 0x00BA86B0>]

instead it builds a list of lambda's, and that is so
desirable <wink>


Others suggested generators indeed, and I can imagine now a
usage pattern like

where the idea is to pass a generator to a function all the
time. What I don't like though is that I then, as I
understand it, have to code some for-loop inside that
function, I try to avoid for-loops. But a functional form
defined with a for-loop does not seem all that bad. Anyway
I have some reading-up to do, about things some posters use
and I forgot to keep up with since I have been coding Python
1.5.2 style for ages now: properties, itertools, ...printed
some stuff already, thanks!

 

[infidel]

It took me a while to figure out what the "translated" code was trying
to do. Here's a quick example that I think accomplishes the same
thing:
.... if i == 1:
.... print n
....
2

 

[Bengt Richter]

The CPython implementation is limited by the stack size allocated by the C
runtime library. The exact recursion limit is platform dependent, but something
around 1000 sounds fairly normal.

ISTM you forgot something ;-)
(I.e., that 1000 is only a default value (don't know if same on all platforms)).
Help on built-in function setrecursionlimit in module sys:

setrecursionlimit(...)
setrecursionlimit(n)

Set the maximum depth of the Python interpreter stack to n. This
limit prevents infinite recursion from causing an overflow of the C
stack and crashing Python. The highest possible limit is platform-
dependent.

Regards,
Bengt Richter

 

[infidel]

Of course I meant to put a break out of the loop after the print
statement. Duh on me.

 

[Will Stuyvesant]

Just for the record, an implementation without using generators,
somewhat like in Sect. 3.5 of the Wizard book, and without recursion
limit problems.
..
.. def stream_hd(s): # the head of the stream
.. return s[0]
..
.. def stream_tl(s): # the tail of the stream
.. return s[1]()
..
.. ##
.. # The low argument is required: the first element of the stream.
.. # You either use high= or stop_f= to define the end of the
.. # stream. Omit both for an infinite stream. stop_f is a function
.. # that takes low as argument and returns True or False.
.. # The next_f function takes one argument (an element of the stream,
.. # same type as the low argument has)
.. # The stop_f function takes one argument (same type as low)
.. # @param low object
.. # @param high object
.. # @param next_f function
.. # @param stop_f function
.. # @return list, a stream
.. def make_stream(low, high=None, next_f=None, stop_f=None):
.. if next_f is None: next_f = lambda x: x + 1
.. if high is not None: # using high
.. if low > high:
.. return None
.. else:
.. return [low, lambda: make_stream(
.. next_f(low), high=high, next_f=next_f)]
.. elif stop_f is not None: # using stop_f
.. if low > stop_f(low):
.. return None
.. else:
.. return [low, lambda: make_stream(
.. next_f(low), next_f=next_f, stop_f=stop_f)]
.. else: # infinite stream
.. return [low, lambda: make_stream(
.. next_f(low), next_f=next_f)]
..
.. ##
.. # iterative version of filter
.. # @param pred function, (stream-element) -> bool
.. # @param s list, a stream
.. # @return list, a stream
.. def stream_filter(pred, s):
.. if s is None: return []
.. while True:
.. elem = stream_hd(s)
.. if pred(elem):
.. return [elem, lambda: stream_filter(pred, stream_tl(s))]
.. else:
.. s = stream_tl(s)
.. if s is None: return []
..
..
.. if __name__ == '__main__':
..
.. def is100some(x): return x % 100 == 0
.. assert(stream_hd(stream_tl(stream_tl(stream_filter(
.. is100some,
.. make_stream(1, 11111111111111))))) == 300)
..
.. def add_33(x): return x + 33
.. assert(stream_hd(stream_tl(stream_filter(
.. is100some,
.. # stream 1,34,67,100,...
.. make_stream(1,999999999, next_f = add_33)))) == 3400)
..
.. assert(stream_hd(stream_tl(stream_filter(
.. is100some,
.. # infinite stream 1,2,...
.. make_stream(1)))) == 200)
..
.. def mod_20000(x): return x % 20000 == 0
.. # this will need more evaluations than the recursion limit count
.. infinite_filter = stream_filter(mod_20000, make_stream(1))
.. assert( stream_hd(stream_tl(infinite_filter)) == 40000 )
..

 

[Nick Coghlan]

How's this:

Py> from itertools import islice
Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

Cheers,
Nick.

 

[Fredrik Lundh]

How's this:

Py> from itertools import islice
Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

as I've always said, the sooner we can all use the itertools goodies without
even noticing, the better.

</F>

 

[Peter Otten]

Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

I like it. Of course you always have to bear in mind that one giant leap for
a list could be _many_ small steps for an iterator.

Peter

 

[Nick Coghlan]

Nick Coghlan wrote:

Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

I like it. Of course you always have to bear in mind that one giant leap for
a list could be _many_ small steps for an iterator.

Indeed. The main cases I am thinking of involve picking off the first few items
of an iterator (either to use them, or to throw them away before using the rest).

And if an app actually *needs* random access, there's a reason lists still exist

Cheers,
Nick.

 

[John J. Lee]

(xrange can't handle Python longs, unfortunately, so we *are*
constrained by sys.maxint. However, since my machine only has half a
gig of RAM, the above is still a damn sight quicker than the
equivalent list comprehension would be!)

[...]

Other way 'round: if you had even more RAM, the listcomp would be even
slower for this job!


John

 

[Bengt Richter]

Nick Coghlan wrote:

Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

I like it. Of course you always have to bear in mind that one giant leap for
a list could be _many_ small steps for an iterator.

Indeed. The main cases I am thinking of involve picking off the first few items
of an iterator (either to use them, or to throw them away before using the rest).

And if an app actually *needs* random access, there's a reason lists still exist

You can bail out of a generator expression with a raise-StopIteration expression spelled iter([]).next() ;-)

>>> def show(x): print x,; return x ...
>>> list(show(x) for x in xrange(20) if x<8 or iter([]).next())

0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]

Well, since
0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]

this change due to adding iter([]).next() might be more convincing:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

>>> list(show(x) for x in xrange(20) if x<8 or iter([]).next() or True)

0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]


Applied to example,

>>> print list(x for i,x in enumerate(x for x in xrange(1, 996) if x % 2 ==0) if i<3 or iter([]).next())[2]

6

or in case you just want one item as result,

>>> print list(x for i,x in enumerate(x for x in xrange(1, 996) if x % 2 ==0) if i==2 or i==3 and iter([]).next())[0]

6

Regards,
Bengt Richter

 

[Nick Coghlan]

You can bail out of a generator expression with a raise-StopIteration expression spelled iter([]).next() ;-)

list(show(x) for x in xrange(20) if x<8 or iter([]).next() or True)

0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]

This is both neat and incredibly arcane at the same time

Cheers,
Nick.