Do you have real-world use cases for map's None fill-in feature?

Discussion in 'Python' started by Raymond Hettinger, Jan 8, 2006.

  1. I am evaluating a request for an alternate version of itertools.izip()
    that has a None fill-in feature like the built-in map function:

    >>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature

    [('a', '1'), ('b', '2'), ('c', '3'), (None, '4'), (None, '5')]

    The movitation is to provide a means for looping over all data elements
    when the input lengths are unequal. The question of the day is whether
    that is both a common need and a good approach to real-world problems.
    The answer to the question can likely be found in results from other
    programming languages or from real-world Python code that has used
    map's None fill-in feature.

    I scanned the docs for Haskell, SML, and Perl and found that the norm
    for map() and zip() is to truncate to the shortest input or raise an
    exception for unequal input lengths. I scanned the standard library
    and found no instances where map's fill-in feature was used. Likewise,
    I found no examples in all of the code I've ever written.

    The history of Python's current zip() function serves as another
    indicator that the proposal is weak. PEP 201 contemplated and rejected
    the idea as one that likely had unintended consequences. In the years
    since zip() was introduced in Py2.0, SourceForge has shown no requests
    for a fill-in version of zip().

    My request for readers of comp.lang.python is to search your own code
    to see if map's None fill-in feature was ever used in real-world code
    (not toy examples). I'm curious about the context, how it was used,
    and what alternatives were rejected (i.e. did the fill-in feature
    improve the code).

    Also, I'm curious as to whether someone has seen a zip fill-in feature
    employed to good effect in some other programming language, perhaps
    LISP or somesuch?

    Maybe a few real-word code examples and experiences from other
    languages will shed light on the question of whether lock-step
    iteration has meaning beyond the length of the shortest matching
    elements. If ordinal position were considered as a record key, then
    the proposal equates to a database-style outer join operation (where
    data elements with unmatched keys are included) and order is
    significant. Does an outer-join have anything to do with lock-step
    iteration? Is this a fundamental looping construct or just a
    theoretical wish-list item? IOW, does Python really need
    itertools.izip_longest() or would that just become a distracting piece
    of cruft?


    Raymond Hettinger


    P.S. FWIW, the OP's use case involved printing files in multiple
    columns:

    for f, g in itertools.izip_longest(file1, file2, fillin_value=''):
    print '%-20s\t|\t%-20s' % (f.rstrip(), g.rstrip())

    The alternative was straight-forward but not as terse:

    while 1:
    f = file1.readline()
    g = file2.readline()
    if not f and not g:
    break
    print '%-20s\t|\t%-20s' % (f.rstrip(), g.rstrip())
     
    Raymond Hettinger, Jan 8, 2006
    #1
    1. Advertising

  2. Raymond Hettinger

    Paul Rubin Guest

    "Raymond Hettinger" <> writes:
    > I am evaluating a request for an alternate version of itertools.izip()
    > that has a None fill-in feature like the built-in map function:
    >
    > >>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature


    I think finding different ways to write it was an entertaining
    exercise but it's too limited in usefulness to become a standard
    feature.

    I do think some idiom ought to develop to allow checking whether an
    iterator is empty, without consuming an item. Here's an idea:
    introduce something like

    iterator = check_empty(iterator)

    where check_empty would work roughly like (untested):

    def check_empty(iterator):
    iclass = iterator.__class__
    class buffered(iclass):
    def __init__(self):
    n = iter((self.next(),)) # might raise StopIteration
    self.__save = chain(n, self)
    def next(self):
    return self.__save.next()
    # all other operations are inherited from iclass

    return buffered(iterator)

    The idea is you get back a new iterator which yields the same stream
    and supports the same operations as the old one, if the old one is
    non-empty. Otherwise it raises StopIteration.

    There are some obvious problems with the above:

    1) the new iterator should support all of the old one's attributes,
    not just inherit its operations
    2) In the case where the old iterator is already buffered, the
    constructor should just peek at the lookahead instead of making
    a new object. That means that checking an iterator multiple times
    won't burn more and more memory.

    Maybe there is some way of doing the above with metaclasses but I've
    never been able to wrap my head around those.
     
    Paul Rubin, Jan 9, 2006
    #2
    1. Advertising

  3. On 7 Jan 2006 23:19:41 -0800, "Raymond Hettinger" <> wrote:

    >I am evaluating a request for an alternate version of itertools.izip()
    >that has a None fill-in feature like the built-in map function:
    >
    >>>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature

    >[('a', '1'), ('b', '2'), ('c', '3'), (None, '4'), (None, '5')]
    >

    I don't like not being able to supply my own sentinel. None is too common
    a value. Hm, ... <bf warning> unless maybe it can also be a type that we can instantiate with
    really-mean-it context level like None(5) ;-)

    >>> map(None(5), 'abc', '12345') # demonstrate map's None fill-in feature

    [('a', '1'), ('b', '2'), ('c', '3'), (None(5), '4'), (None(5), '5')]

    But seriously, standard sentinels for "missing data" and "end of data" might be nice to have,
    and to have produced in appropriate standard contexts. Singleton string subclass
    instances "NOD" and "EOD"? Doesn't fit with EOF=='' though.
    </bf warning>

    >The movitation is to provide a means for looping over all data elements
    >when the input lengths are unequal. The question of the day is whether
    >that is both a common need and a good approach to real-world problems.
    >The answer to the question can likely be found in results from other
    >programming languages or from real-world Python code that has used
    >map's None fill-in feature.
    >

    What about some semantics like my izip2 in
    http://groups.google.com/group/comp.lang.python/msg/3e9eb63a1ddb1f46?hl=en

    (which doesn't even need a separate name, since it would be backwards compatible)

    Also, what about factoring sequence-related stuff into being methods or attributes
    of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
    which could be a substitute for izip and then some? Methods could be called via a returned
    iterator before or after the first .next() call, to control various features, such as
    sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
    steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
    The point being to have a place to implement universal sequence stuff.

    >I scanned the docs for Haskell, SML, and Perl and found that the norm
    >for map() and zip() is to truncate to the shortest input or raise an
    >exception for unequal input lengths. I scanned the standard library
    >and found no instances where map's fill-in feature was used. Likewise,
    >I found no examples in all of the code I've ever written.
    >
    >The history of Python's current zip() function serves as another
    >indicator that the proposal is weak. PEP 201 contemplated and rejected
    >the idea as one that likely had unintended consequences. In the years
    >since zip() was introduced in Py2.0, SourceForge has shown no requests
    >for a fill-in version of zip().
    >
    >My request for readers of comp.lang.python is to search your own code
    >to see if map's None fill-in feature was ever used in real-world code
    >(not toy examples). I'm curious about the context, how it was used,
    >and what alternatives were rejected (i.e. did the fill-in feature
    >improve the code).
    >
    >Also, I'm curious as to whether someone has seen a zip fill-in feature
    >employed to good effect in some other programming language, perhaps
    >LISP or somesuch?

    ISTM in general there is a chicken-egg problem where workarounds are easy.
    I.e., the real question is how many workaround situations there are
    that would have been handled conveniently with a builtin feature,
    and _then_ to see whether the convenience would be worth enough.
    >
    >Maybe a few real-word code examples and experiences from other
    >languages will shed light on the question of whether lock-step
    >iteration has meaning beyond the length of the shortest matching
    >elements. If ordinal position were considered as a record key, then
    >the proposal equates to a database-style outer join operation (where
    >data elements with unmatched keys are included) and order is
    >significant. Does an outer-join have anything to do with lock-step
    >iteration? Is this a fundamental looping construct or just a
    >theoretical wish-list item? IOW, does Python really need
    >itertools.izip_longest() or would that just become a distracting piece
    >of cruft?

    Even if there is little use for continuing in correct code, IWT getting
    at the state of the iterator in an erroroneous situation would be a benefit.
    Being able to see the result of the last attempt at gathering tuple elements
    could help. (I can see reasons for wanting variations of trying all streams
    vs shortcutting on the first to exhaust though).

    Regards,
    Bengt Richter
     
    Bengt Richter, Jan 10, 2006
    #3
  4. [Bengt Richter]
    > What about some semantics like my izip2 in
    > http://groups.google.com/group/comp.lang.python/msg/3e9eb63a1ddb1f46?hl=en
    >
    > (which doesn't even need a separate name, since it would be backwards compatible)
    >
    > Also, what about factoring sequence-related stuff into being methods or attributes
    > of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
    > which could be a substitute for izip and then some? Methods could be called via a returned
    > iterator before or after the first .next() call, to control various features, such as
    > sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
    > steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
    > The point being to have a place to implement universal sequence stuff.


    ISTM, these cures are worse than the disease ;-)


    > Even if there is little use for continuing in correct code, IWT getting
    > at the state of the iterator in an erroroneous situation would be a benefit.
    > Being able to see the result of the last attempt at gathering tuple elements
    > could help. (I can see reasons for wanting variations of trying all streams
    > vs shortcutting on the first to exhaust though).


    On the one hand, that seems reasonable. On the other hand, I can't see
    how to use it without snarling the surrounding code in which case it is
    probably better to explicitly manage individual iterators within a
    while loop.


    Raymond
     
    Raymond Hettinger, Jan 10, 2006
    #4
  5. [Raymond Hettinger]
    > > I am evaluating a request for an alternate version of itertools.izip()
    > > that has a None fill-in feature like the built-in map function:
    > >
    > > >>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature


    [Paul Rubin]
    > I think finding different ways to write it was an entertaining
    > exercise but it's too limited in usefulness to become a standard
    > feature.


    Makes sense.

    > I do think some idiom ought to develop to allow checking whether an
    > iterator is empty, without consuming an item. Here's an idea:
    > introduce something like
    >
    > iterator = check_empty(iterator)


    There are so many varieties of iterator that it's probably not workable
    to alter the iterator API for all of the them. In any case, a broad
    API change like this would need its own PEP.


    > There are some obvious problems with the above:
    >
    > 1) the new iterator should support all of the old one's attributes,
    > not just inherit its operations
    > 2) In the case where the old iterator is already buffered, the
    > constructor should just peek at the lookahead instead of making
    > a new object. That means that checking an iterator multiple times
    > won't burn more and more memory.
    >
    > Maybe there is some way of doing the above with metaclasses but I've
    > never been able to wrap my head around those.


    Metaclasses are unlikely to be of help because there are so many,
    unrelated kinds of iterator -- most do not inherit from a common
    parent.


    Raymond
     
    Raymond Hettinger, Jan 10, 2006
    #5
  6. Raymond Hettinger

    Paul Rubin Guest

    "Raymond Hettinger" <> writes:
    > > iterator = check_empty(iterator)

    >
    > There are so many varieties of iterator that it's probably not workable
    > to alter the iterator API for all of the them. In any case, a broad
    > API change like this would need its own PEP.


    The hope was that it wouldn't be an API change, but rather just a new
    function dropped into the existing library, that could wrap any
    existing iterator without having to change or break anything that's
    already been written. Maybe the resulting iterator couldn't support
    every operation, or maybe it could have a __getattr__ that delegates
    everything except "next" to the wrapped iterator, or something. The
    obvious implementation methods that I can see are very kludgy but
    maybe something better is feasible. I defer to your knowledge about
    this.
     
    Paul Rubin, Jan 10, 2006
    #6
  7. Raymond Hettinger

    Tim Peters Guest

    [Raymond Hettinger]
    > ...
    > I scanned the docs for Haskell, SML, and Perl and found that the norm
    > for map() and zip() is to truncate to the shortest input or raise an
    > exception for unequal input lengths.
    > ...
    > Also, I'm curious as to whether someone has seen a zip fill-in feature
    > employed to good effect in some other programming language, perhaps
    > LISP or somesuch?


    FYI, Common Lisp's `pairlis` function requires that its first two
    arguments be lists of the same length. It's a strain to compare to
    Python's zip() though, as the _intended_ use of `pairlis` is to add
    new pairs to a Lisp association list. For that reason, `pairlis`
    accepts an optional third argument; if present, this should be an
    association list, and pairs from zipping the first two arguments are
    prepended to it. Also for this reason, the _order_ in which pairs are
    taken from the first two arguments isn't defined(!).

    http://www.lispworks.com/documentation/HyperSpec/Body/f_pairli.htm#pairlis

    For its intended special-purpose use, it wouldn't make sense to allow
    arguments of different lengths.
     
    Tim Peters, Jan 10, 2006
    #7
  8. On 10 Jan 2006 00:47:36 -0800, "Raymond Hettinger" <> wrote:

    >[Bengt Richter]
    >> What about some semantics like my izip2 in
    >> http://groups.google.com/group/comp.lang.python/msg/3e9eb63a1ddb1f46?hl=en
    >>
    >> (which doesn't even need a separate name, since it would be backwards compatible)
    >>
    >> Also, what about factoring sequence-related stuff into being methods or attributes
    >> of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
    >> which could be a substitute for izip and then some? Methods could be called via a returned
    >> iterator before or after the first .next() call, to control various features, such as
    >> sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
    >> steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
    >> The point being to have a place to implement universal sequence stuff.

    >
    >ISTM, these cures are worse than the disease ;-)

    Are you reacting to my turgidly rambling post, or to

    >>> from ut.izip2 import izip2 as izip
    >>> it = izip('abc','12','ABCD')
    >>> for t in it: print t

    ...
    ('a', '1', 'A')
    ('b', '2', 'B')

    Then after a backwards-compatible izip, if the iterator has
    been bound, it can be used to continue, with sentinel sustitution:

    >>> for t in it.rest('<sentinel>'): print t

    ...
    ('c', '<sentinel>', 'C')
    ('<sentinel>', '<sentinel>', 'D')

    or optionally in sentinel substitution mode from the start:

    >>> for t in izip('abc','12','ABCD').rest('<sentinel>'): print t

    ...
    ('a', '1', 'A')
    ('b', '2', 'B')
    ('c', '<sentinel>', 'C')
    ('<sentinel>', '<sentinel>', 'D')

    Usage-wise, this seems not too diseased to me, so I guess I want to make sure
    this is what you were reacting to ;-)

    (Implementation was just to hack together a working demo. I'm sure it can be improved upon ;-)

    >
    >
    >> Even if there is little use for continuing in correct code, IWT getting
    >> at the state of the iterator in an erroroneous situation would be a benefit.
    >> Being able to see the result of the last attempt at gathering tuple elements
    >> could help. (I can see reasons for wanting variations of trying all streams
    >> vs shortcutting on the first to exhaust though).

    >
    >On the one hand, that seems reasonable. On the other hand, I can't see
    >how to use it without snarling the surrounding code in which case it is
    >probably better to explicitly manage individual iterators within a
    >while loop.
    >

    The above would seem to allow separation of concerns. I.e., if you care why
    a normal iteration terminated, you can test after the fact. I.e., if all sequences
    were the same length, the .rest() iterator will be empty. And if you don't care at
    all about possible data, you can just try: it.rest().next() and catch StopIteration
    to check.

    BTW, is there any rule against passing information with StopIteration?

    Regards,
    Bengt Richter
     
    Bengt Richter, Jan 10, 2006
    #8
  9. > There are so many varieties of iterator that it's probably not workable
    > to alter the iterator API for all of the them.


    i always wondered if it can be implemented:

    there are iterators which has length:
    >>> i = iter([1,2,3])
    >>> len(i)

    3

    now isn't there a way to make this length inheritible?
    eg. generators could have length in this case:
    >>> g = (x for x in [1,2,3])
    >>> # len(g) == len([1,2,3]) == 3


    of course in special cases length would remain undefined:
    >>> f = (x for x in [1,2,3] if x>2)
    >>> # len(f) == ?


    IMHO there are special cases when this is useful:
    L=list(it)
    here if it has length, then list creation can be more effective
    (required memory is known in advance)

    nsz
     
    Szabolcs Nagy, Jan 10, 2006
    #9
  10. > There are so many varieties of iterator that it's probably not workable
    > to alter the iterator API for all of the them.


    i always wondered if it can be implemented:

    there are iterators which has length:
    >>> i = iter([1,2,3])
    >>> len(i)

    3

    now isn't there a way to make this length inheritible?
    eg. generators could have length in this case:
    >>> g = (x for x in [1,2,3])
    >>> # len(g) == len([1,2,3]) == 3


    of course in special cases length would remain undefined:
    >>> f = (x for x in [1,2,3] if x>2)
    >>> # len(f) == ?


    IMHO there are special cases when this is useful:
    L=list(it)
    here if it has length, then list creation can be more effective
    (required memory is known in advance)

    nsz
     
    Szabolcs Nagy, Jan 10, 2006
    #10
  11. Raymond Hettinger

    Paul Rubin Guest

    "Szabolcs Nagy" <> writes:
    > there are iterators which has length:
    > >>> i = iter([1,2,3])
    > >>> len(i)

    > 3
    >
    > now isn't there a way to make this length inheritible?


    I expect that's a __len__ method, which can be inherited.

    > eg. generators could have length in this case:
    > >>> g = (x for x in [1,2,3])
    > >>> # len(g) == len([1,2,3]) == 3


    I dunno what happens with this now.
     
    Paul Rubin, Jan 10, 2006
    #11
  12. Szabolcs Nagy wrote:

    > there are iterators which has length:
    > >>> i = iter([1,2,3])
    > >>> len(i)

    > 3


    that's a bug, which has been fixed in 2.5:

    Python 2.5a0 (#5, Dec 14 2005, 22:28:52)
    Type "help", "copyright", "credits" or "license" for more information.
    >>> i = iter([1, 2, 3])
    >>> len(i)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    TypeError: len() of unsized object

    </F>
     
    Fredrik Lundh, Jan 10, 2006
    #12
  13. [Raymond]
    > >ISTM, these cures are worse than the disease ;-)


    [Bengt]
    > Are you reacting to my turgidly rambling post, or to
    >
    > >>> from ut.izip2 import izip2 as izip
    > >>> it = izip('abc','12','ABCD')
    > >>> for t in it: print t

    > ...
    > ('a', '1', 'A')
    > ('b', '2', 'B')
    >
    > Then after a backwards-compatible izip, if the iterator has
    > been bound, it can be used to continue, with sentinel sustitution:
    >
    > >>> for t in it.rest('<sentinel>'): print t

    > ...
    > ('c', '<sentinel>', 'C')
    > ('<sentinel>', '<sentinel>', 'D')
    >
    > or optionally in sentinel substitution mode from the start:
    >
    > >>> for t in izip('abc','12','ABCD').rest('<sentinel>'): print t

    > ...
    > ('a', '1', 'A')
    > ('b', '2', 'B')
    > ('c', '<sentinel>', 'C')
    > ('<sentinel>', '<sentinel>', 'D')
    >
    > Usage-wise, this seems not too diseased to me, so I guess I want to make sure
    > this is what you were reacting to ;-)


    There is an elegance to the approach; however, if some sort of fill-in
    were needed, I'm more inclined to add a separate function than to
    append a method to the izip object. The latter API presents a bit of a
    learning/memory challenge, not because it is hard, but because it is
    atypical.

    A unique advantage for your API is that the loop can be run in two
    phases, matched and unmatched. The question then turns to whether there
    is a need for that option. So far, the three threads on the subject
    have shown us to be starved for use cases for a single phase
    izip_longest function, much less a two-pass version of the same.



    Raymond
     
    Raymond Hettinger, Jan 10, 2006
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Raymond Hettinger
    Replies:
    30
    Views:
    795
    Steven D'Aprano
    Jan 25, 2006
  2. length power
    Replies:
    2
    Views:
    115
    Rustom Mody
    Apr 10, 2014
  3. Skip Montanaro
    Replies:
    0
    Views:
    82
    Skip Montanaro
    Apr 10, 2014
  4. Johannes Schneider

    Re: why i have the output of [None, None, None]

    Johannes Schneider, Apr 10, 2014, in forum: Python
    Replies:
    0
    Views:
    72
    Johannes Schneider
    Apr 10, 2014
  5. Terry Reedy
    Replies:
    0
    Views:
    83
    Terry Reedy
    Apr 10, 2014
Loading...

Share This Page