empty lists vs empty generators

Discussion in 'Python' started by Brian Roberts, May 3, 2005.

  1. I'm using using generators and iterators more and more intead of
    passing lists around, and prefer them. However, I'm not clear on the
    best way to detect an empty generator (one that will return no items)
    when some sort of special case handling is required.

    Typical code for handling an empty list:
    if somelist:
    for x in somelist:
    something(x)
    else:
    empty_list_special_case

    But this doesn't work with iterators -- a generator is "true"
    regardless of whether its going to return any items. (I understand
    why).

    The closest equivalent I know of is:
    n = 0
    for n, x in enumerate(somegenerator()):
    something(x)
    if n == 0:
    empty_list_special_case

    Which seems rather awkward -- doesn't read as easily for me, and
    introduces another variable.

    Q1: Is there a better or alternate way to handle this?
    Q2: Is there a way that handles both lists and generators, so I don't
    have to worry about which one I've got?

    Thanks,
    Brian.
    Brian Roberts, May 3, 2005
    #1
    1. Advertising

  2. Brian Roberts

    jfj Guest

    Brian Roberts wrote:

    > I'm using using generators and iterators more and more intead of
    > passing lists around, and prefer them. However, I'm not clear on the
    > best way to detect an empty generator (one that will return no items)
    > when some sort of special case handling is required.
    >


    Usually it will be the job of the generator to signal something like
    this. I think a possible way might be:

    class GeneratorEmpty: pass

    def generator():
    if not X:
    raise GeneratorEmpty
    for i in X:
    yield i

    try:
    for x in generator
    something (x)
    except GeneratorEmpty:
    generator_special_case

    The trick is that when generators raise exceptions they terminate.
    Although this is probably not what you want. The thing is that you
    cannot know if a generator will return any elements until you call
    its next() method.


    > Q2: Is there a way that handles both lists and generators, so I don't
    > have to worry about which one I've got?


    I don't think this is possible. A generator must be called (with
    next()) in order for its code to take over and see if it is empty or
    not. Unlike the list.


    jfj
    jfj, May 3, 2005
    #2
    1. Advertising

  3. Brian Roberts

    Roy Smith Guest

    In article <>,
    (Brian Roberts) wrote:

    > I'm using using generators and iterators more and more intead of
    > passing lists around, and prefer them. However, I'm not clear on the
    > best way to detect an empty generator (one that will return no items)
    > when some sort of special case handling is required.


    The best I can come up with is to depend on the fact that

    for item in foo:
    pass

    only defines item if foo yields any items. Assuming item is not defined
    before you execute the for loop, you can check to see if it's defined after
    the loop, and use that to tell if foo was an empty list or generator.
    Here's a demo. Unfortunately, I'm not sure if it's really any cleaner than
    your way (but at least it doesn't add any extraneous variables)


    # Creates an iterator which yields n items.
    class gen:
    def __init__(self, n):
    self.n = n

    def __iter__(self):
    for i in range(self.n):
    yield None

    def checkEmpty(genOrList):
    for item in genOrList:
    pass

    try:
    item
    print "%s had items" % genOrList
    except NameError:
    print "%s was empty" % genOrList

    checkEmpty(gen(0))
    checkEmpty(gen(1))
    checkEmpty([])
    checkEmpty([1])

    --------------

    Roy-Smiths-Computer:play$ ./gen.py
    <__main__.gen instance at 0x36c620> was empty
    <__main__.gen instance at 0x36c620> had items
    [] was empty
    [1] had items
    Roy Smith, May 3, 2005
    #3
  4. On Mon, 02 May 2005 16:14:57 -0700, Brian Roberts wrote:
    > Q1: Is there a better or alternate way to handle this? Q2: Is there a way
    > that handles both lists and generators, so I don't have to worry about
    > which one I've got?


    Are you in control of your generators? You could put a method on them that
    tells if there is anything in them by manually implementing the .next()
    call.

    The other thing you could do is a generator wrapper that can tell for you,
    but you'll lose some performance:

    class EmptyGeneratorDetector(object):
    """Provides a method you can call to detect an empty
    generator. You should probably name this class something
    shorter.

    Check if the generator is empty after construction by looking at
    the isEmpty property."""

    def __init__(self, generator):
    self.generator = generator

    self.isEmpty = False
    self.givenFirst = False
    try:
    self.firstItem = generator.next()
    except StopIteration:
    self.isEmpty = True

    def next(self):
    if self.isEmpty:
    raise StopIteration

    if not self.givenFirst:
    self.givenFirst = True
    return self.firstItem
    else:
    return self.generator.next()

    def __iter__(self):
    return self

    In action:

    Python 2.3.5 (#1, Mar 3 2005, 17:32:12)
    [GCC 3.4.3 (Gentoo Linux 3.4.3, ssp-3.4.3-0, pie-8.7.6.6)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from genwrap import *
    >>> def emptyGenerator():

    .... raise StopIteration
    .... yield None
    ....
    >>> def nonEmptyGenerator():

    .... yield 1
    .... yield 2
    .... yield 3
    ....
    >>> e = emptyGenerator()
    >>> n = nonEmptyGenerator()
    >>> E = EmptyGeneratorDetector(e)
    >>> N = EmptyGeneratorDetector(n)
    >>> E.isEmpty

    True
    >>> N.isEmpty

    False
    >>> for i in E:

    .... print i
    ....
    >>> for i in N:

    .... print i
    ....
    1
    2
    3
    >>>


    It is tested as much as you see it above :)

    (I recall a lengthy discussion of the best way to create an empty iterator
    a while back, and that was not the winner. But it will do for now.)
    Jeremy Bowers, May 3, 2005
    #4
  5. On 2 May 2005 16:14:57 -0700, (Brian Roberts) wrote:

    >I'm using using generators and iterators more and more intead of
    >passing lists around, and prefer them. However, I'm not clear on the
    >best way to detect an empty generator (one that will return no items)
    >when some sort of special case handling is required.
    >
    >Typical code for handling an empty list:
    > if somelist:
    > for x in somelist:
    > something(x)
    > else:
    > empty_list_special_case
    >
    >But this doesn't work with iterators -- a generator is "true"
    >regardless of whether its going to return any items. (I understand
    >why).
    >
    >The closest equivalent I know of is:
    > n = 0
    > for n, x in enumerate(somegenerator()):
    > something(x)
    > if n == 0:
    > empty_list_special_case
    >
    >Which seems rather awkward -- doesn't read as easily for me, and
    >introduces another variable.

    And, if I understood the intent, doesn't work ;-)

    >>> n = 0
    >>> for n, x in enumerate(c for c in 'a'):

    ... print 'something', x
    ...
    something a
    >>> if n == 0:

    ... print 'empty list special case ??'
    ...
    empty list special case ??

    You could have used n = -1 as a sentinel that enumerate would not set,
    but using a guaranteed-unique sentinel, you don't need enumerate, e.g.,

    >>> x = sentinel = object()
    >>> for x in (c for c in 'a'):

    ... print 'something', x
    ...
    something a
    >>> if x is sentinel:

    ... print 'empty list special case ??'
    ...

    (nothing printed there)
    and for the actually empty sequence

    >>> x = sentinel = object()
    >>> for x in (c for c in ''):

    ... print 'something', x
    ...
    >>> if x is sentinel:

    ... print 'empty list special case ??'
    ...
    empty list special case ??

    >
    >Q1: Is there a better or alternate way to handle this?
    >Q2: Is there a way that handles both lists and generators, so I don't
    >have to worry about which one I've got?
    >

    UIAM this should work for any iterable. You don't have to manufacture
    a locally bound sentinel as above. You could pick anything to preset
    the for-target that you know is not going to be produced by the iterable,
    though you might need to use '==' instead of 'is' depending on your choice.
    But e.g., I don't think I'd write

    x = Exception # weird sentinel choice
    for x in mystring:
    print x, ord(x)
    if x is Exception:
    print 'null sequence'

    None probably works well a lot of the time, but not always.
    Similarly ''. Seems like a builtin sentinel binding like sentinel = object()
    might be handy to standardize usage.

    Regards,
    Bengt Richter
    Bengt Richter, May 3, 2005
    #5
  6. Starting from Python 2.4 we have tee in the itertools
    module, so you can define the following:

    from itertools import tee

    def is_empty(it):
    it_copy = tee(it)[1]
    try:
    it_copy.next()
    except StopIteration:
    return True
    else:
    return False

    It works with generic iterables too.

    Michele Simionato
    Michele Simionato, May 3, 2005
    #6
  7. Brian Roberts

    Terry Reedy Guest

    "Brian Roberts" <> wrote in message
    news:...
    > I'm using using generators and iterators more and more intead of
    > passing lists around, and prefer them. However, I'm not clear on the
    > best way to detect an empty generator (one that will return no items)
    > when some sort of special case handling is required.


    If you write an iterator class instead of the abbreviated generator form,
    and you can tell from the initialization parameters whether there will be
    any data, then you can give the class a __nonzero__ method. You can also
    have an initially nonempty iterator flag when it becomes empty.

    My point is that writing an iterator as a generator is a convenience, not a
    necessity, and that one gives up the full flexibility of an iterator class
    when one does so, but that one is not required to do so.

    I quite understanding wanting to have your cake and eat it too. The
    convenience is sometimes major.

    Terry J. Reedy
    Terry Reedy, May 3, 2005
    #7
  8. On 2 May 2005 21:49:33 -0700, "Michele Simionato"
    <> wrote:

    >Starting from Python 2.4 we have tee in the itertools
    >module, so you can define the following:
    >
    >from itertools import tee
    >
    >def is_empty(it):
    > it_copy = tee(it)[1]
    > try:
    > it_copy.next()
    > except StopIteration:
    > return True
    > else:
    > return False
    >
    >It works with generic iterables too.


    Are you sure this is going to do the right thing ?
    seems to me it would drop the first element of
    "it"... (the yielded element entered the tee twins,
    but already got out of "it").
    I would say that unless you use the second twin
    after calling is_empty that code wouldn't work...

    Am I correct or instead "tee" uses black magic to
    just peek at the yielded value without starting a
    continuation ?

    Andrea
    Andrea Griffini, May 3, 2005
    #8
  9. Andrea Griffini:

    > Are you sure this is going to do the right thing ?


    Argh! I missed these two lines from the documentation:

    """Note, once tee() has made a split, the original iterable should not
    be used anywhere else; otherwise, the iterable could get advanced
    without the tee objects being informed."""

    Since the original iterator cannot be reused, we need an alternative
    approach. Here is a possibility:

    #<check.py>

    import itertools

    def check(it):
    it_copy1, it_copy2 = itertools.tee(it)
    try:
    it_copy2.next()
    except StopIteration:
    return None
    else:
    return it_copy1

    #</check.py>

    Here a few examples of usage:

    >>> from check import check


    >>> it0 = iter([])
    >>> print check(it0) # empty iterator, returns None

    None

    >>> it1 = iter([1])
    >>> it1 = check(it1) # non-empty iterator, returns a copy of the

    original one
    >>> it1.next()

    1

    In general you can use the idiom

    it = check(it) # check for emptiness
    if it:
    # do something

    This time I have checked the examples here with my doctester
    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/410052 ;)

    Michele Simionato
    Michele Simionato, May 4, 2005
    #9
  10. Jeremy Bowers wrote:
    > def __init__(self, generator):
    > self.generator = generator


    You'll want to use iter(generator) there in order to handle reiterables.
    Leif K-Brooks, May 4, 2005
    #10
  11. On Wed, 04 May 2005 13:45:00 +0000, Leif K-Brooks wrote:

    > Jeremy Bowers wrote:
    >> def __init__(self, generator):
    >> self.generator = generator

    >
    > You'll want to use iter(generator) there in order to handle reiterables.


    Can you expand that explanation a bit? I'm not certain what you mean. I'm
    just trusting what the user passes in; maybe the user should pass it
    iter(generator) when it's a "reiterable"? (Honest question.)

    What definition of "re-iterable" are you using? (A quick google for
    "Python reiterabile" just turns up some Python dev list entries from 2003.)
    Jeremy Bowers, May 4, 2005
    #11
  12. Jeremy Bowers wrote:
    > On Wed, 04 May 2005 13:45:00 +0000, Leif K-Brooks wrote:
    >
    >
    >>Jeremy Bowers wrote:
    >>
    >>> def __init__(self, generator):
    >>> self.generator = generator

    >>
    >>You'll want to use iter(generator) there in order to handle reiterables.

    >
    >
    > Can you expand that explanation a bit? I'm not certain what you mean. I'm
    > just trusting what the user passes in; maybe the user should pass it
    > iter(generator) when it's a "reiterable"? (Honest question.)
    >
    > What definition of "re-iterable" are you using? (A quick google for
    > "Python reiterabile" just turns up some Python dev list entries from 2003.)


    Reiterable is generally defined as an object which can be iterated over
    multiple times (i.e. is iterable but isn't an iterator). The simplest
    example is a list, but a few other built-in types (set and dict, for
    instance) also qualify.

    With the EmptyGeneratorDetector class as you defined it, lists will fail:

    >>> EmptyGeneratorDetector([])

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 15, in __init__
    AttributeError: 'list' object has no attribute 'next'

    Of course, the class is labeled as an empty generator detector, not an
    empty iterable detector, so it's doing what it says it will, but a
    little bit of extra generalism can't hurt.
    Leif K-Brooks, May 4, 2005
    #12
  13. On Wed, 04 May 2005 20:33:31 +0000, Leif K-Brooks wrote:
    > With the EmptyGeneratorDetector class as you defined it, lists will fail:
    >
    > >>> EmptyGeneratorDetector([])

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in ?
    > File "<stdin>", line 15, in __init__
    > AttributeError: 'list' object has no attribute 'next'
    >
    > Of course, the class is labeled as an empty generator detector, not an
    > empty iterable detector, so it's doing what it says it will, but a little
    > bit of extra generalism can't hurt.


    OK, thanks, now I see what you mean. I was worried that you might be
    referring to an iterator type that returned something other than itself
    when you called iter on it, which I thought wasn't legal.
    Jeremy Bowers, May 4, 2005
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?ISO-8859-1?Q?Lasse_V=E5gs=E6ther_Karlsen?=

    Merging sorted lists/iterators/generators into one stream of values...

    =?ISO-8859-1?Q?Lasse_V=E5gs=E6ther_Karlsen?=, Oct 6, 2005, in forum: Python
    Replies:
    24
    Views:
    724
    =?ISO-8859-1?Q?Lasse_V=E5gs=E6ther_Karlsen?=
    Oct 11, 2005
  2. =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==

    List of lists of lists of lists...

    =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==, May 8, 2006, in forum: Python
    Replies:
    5
    Views:
    404
    =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==
    May 15, 2006
  3. Josh Dukes
    Replies:
    0
    Views:
    293
    Josh Dukes
    Feb 10, 2009
  4. Josh Dukes
    Replies:
    3
    Views:
    321
    Josh Dukes
    Feb 10, 2009
  5. Veli-Pekka Tätilä

    Emulating Generators: Iterators for Lists (Newbie)

    Veli-Pekka Tätilä, Mar 4, 2006, in forum: Perl Misc
    Replies:
    9
    Views:
    127
    Anno Siegel
    Mar 6, 2006
Loading...

Share This Page