request help with Pipe class in iterwrap.py

[Steve R. Hastings]

While studying iterators and generator expressions, I started wishing I
had some tools for processing the values. I wanted to be able to chain
together a set of functions, sort of like the "pipelines" you can make
with command-line programs.

So, I wrote a module called iterwrap.py. You can download it from here:

http://home.blarg.net/~steveha/iterwrap.tar.gz



iterwrap has functions that "wrap" an iterator; when you call the
..next() method on a wrapped iterator, it will get the .next() value
from the original iterator, apply a function to it, and return the new
value. Of course, a wrapped iterator is itself an iterator, so you can
wrap it again: you can build up a "chain" of wrappers that will do the
processing you want.

As an example, here's a command-line pipeline:

cat mylist | sort | uniq > newlist


Here's the analogous example from iterwrap:

newlist = list(iterwrap.uniq(iterwrap.sort(mylist)))


You need to call list() because all the wrapper functions in iterwrap
always return an iterator. That final list() forces the iterator
returned by uniq() to be expanded out to a list.



iterwrap.py defines functions based on many common command-line tools:
sort, uniq, tr, grep, cat, head, tail, and tee. Plus it defines some
other functions that seemed like they would be useful.

Well, it doesn't take very many nested function calls before the call gets
visually confusing, with lots of parentheses at the end. To avoid this,
you can arrange the calls in a vertical chain, like this:

temp = iterwrap.sort(mylist)
temp = iterwrap.uniq(temp)
newlist = list(temp)


But I wanted to provide a convenience class to allow "dot-chaining". I
wanted something like this to work:

from iterwrap import *
newlist = Pipe(mylist).sort.uniq.list()


I have actually coded up two classes. One, Pipe, works as shown above.
The other, which I unimaginatively called "IW" (for "iterwrap"), works in
a right-to-left order:

from iterwrap import *
iw = IW()
newlist = iw.list.uniq.sort(mylist)


Hear now my cry for help:

Both IW() and Pipe() have annoying problems. I'd like to have one class
that just works.

The problem with Pipe() is that it will act very differently depending on
whether the user remembers to put the "()" on the end. For all the
dot-chained functions in the middle of the chain, you don't need to put
parentheses; it will just work. However, for the function at the end of
the dot-chain, you really ought to put the parentheses.

In the given example, if the user remembers to put the parentheses, mylist
will be set to a list; otherwise, mylist will be set to an instance of
class Pipe.

An instance of class Pipe works as an iterator, so in this example:

itr = Pipe(mylist).sort.uniq

....then the user really need not care whether there are parentheses after
uniq() or not. Which of course will make it all the more confusing when
the list() case breaks.


In comparison with Pipe, IW is clean and elegant. The user cannot forget
the parenthetical expression on the end, since that's where the initial
sequence (list or iterator) is provided! The annoying thing about IW is
that the dot-chained functions cannot have extra arguments passed in.

This example works correctly:

newlist = Pipe(mylist).grep("larch").grep("parrot", "v").list()

newlist will be set to a list of all strings from mylist that contain the
string "larch" but do not contain the string "parrot". There is no way to
do this example with IW, because IW expects just one call to its
__call__() function. The best you could do with IW is:

temp = iw.grep(mylist, "larch")
newlist = iw.list.grep(temp, "parrot", "v")

Since it *is* legal to pass extra arguments to the one permitted
__call__(), this works, but it's really not very much of an advantage over
the vertical chain:

temp = grep(mylist, "larch")
temp = grep(temp, "parrot", "v")
newlist = list(temp)



The key point here is that, when processing a dot-chain, my code doesn't
actually know whether it's looking at the end of the dot-chain. If you
had

newlist = Pipe(mylist).foo.bar.baz

and if my code could somehow know that baz is the last thing in the chain,
it could treat baz specially (and do the right thing whether there are
parentheses on it, or not). I wish there were a special method __set__
called when an expression is being assigned somewhere; that would make
this trivial.



What is the friendliest and most Pythonic way to write a Pipe class for
iterwrap?


P.S. I have experimented with overloading the | operator to allow this
syntax:

newlist = Pipe(mylist) | sort | uniq | list()

Personally, I much prefer the dot-chaining syntax. The above is just too
tricky.

 

[Marc 'BlackJack' Rintsch]

What is the friendliest and most Pythonic way to write a Pipe class for
iterwrap?

Maybe one with less "magic" syntax. What about using a function that
takes the iterators and an iterable and returns an iterator of the chained
iterators::

new_list = pipe(grep('larch'), grep('parrot', 'v')), list)(my_list)


Ciao,
Marc 'BlackJack' Rintsch

 

[Steve R. Hastings]

Maybe one with less "magic" syntax. What about using a function that
takes the iterators and an iterable and returns an iterator of the chained
iterators::

new_list = pipe(grep('larch'), grep('parrot', 'v')), list)(my_list)

This is a good idea. But not quite magic enough, I'm afraid.

One of the features of Pipe() is that it automatically pastes in the first
argument of each function call (namely, the iterator returned by the
previous function call). It is able to do this because of a special
__getattr__ that grabs the function reference but returns the "self"
instance of the Pipe class, to allow the dot-chain to continue.

Any supplied options will then be pasted in after that first argument.

In your example, "grep('larch')" is going to be evaluated by Python, and
immediately called. And it will then complain because its first argument
is not an iterator. I cannot see any way to modify this call before it
happens.


If we take your basic idea, and apply just a little bit of magic, we could
do this:

new_list = Pipe(my_list, grep, 'larch', grep, ('parrot', 'v'), list)


The rules would be:

* the first argument to Pipe is always the initial iterable sequence.

* each argument after that is tested to see if it is callable. If it is,
it's remembered; if not, it is presumed to be an argument for the
remembered callable. Multiple arguments must be packaged up into a tuple
or list. Once Pipe() has a callable and an argument or sequence of
arguments, Pipe() can paste in all arguments and make the call;
alternatively, once Pipe() sees another callable, it can safely assume
that there aren't going to be any extra arguments for the remembered
callable, and paste in that one iterator argument and make the call.

Now Pipe always knows when it has reached the last callable, because it
will have reached the end of the supplied arguments! Then it can safely
assume there aren't going to be any extra arguments, and make the call to
the last remembered callable.



However, I remain fond of the dot-chaining syntax. For interactively
playing around with data, I think the dot-chaining syntax is more natural
for most people.

newlist = Pipe(mylist).sort.uniq.list()

newlist = Pipe(mylist, sort, uniq, list)


Hmmm. The second one really isn't bad... Also, the second one doesn't
require my tricky e_eval() to work; it just lets Python figure out all the
function references.



Thinking about it, I realize that "list" is a very common thing with
which to end a dot-chain. I think perhaps if my code would just notice
that the last function reference is "list", which takes exactly one
argument and thus cannot be waiting for additional arguments, it could
just call list() right away.

If there were a general way to know that a function reference only expects
a single argument, I could generalize this idea. But it may be enough to
just do the special case for list().

I think I'll keep Pipe(), hackish as it is, but I will also add a new one
based on your idea. Maybe I'll call it "Chain()".

newlist = Chain(mylist, sort, uniq, list)


I did kind of want a way to make a "reusable pipe". If you come up with a
useful chain, it might be nice if you could use it again with convenient
syntax. Maybe like so:

sort_u = [sort, uniq, list]
newlist = Chain(mylist, sort_u)



Thank you very much for making a helpful suggestion!