subtle side effect of generator/generator expression

Discussion in 'Python' started by bonono@gmail.com, Oct 16, 2005.

  1. Guest

    Hi,

    I initially thought that generator/generator expression is cool(sort of
    like the lazy evaluation in Haskell) until I notice this side effect.

    >>>a=(x for x in range(2))
    >>>list(a)

    [1,2]
    >>>list(a)

    []

    Would this make generator/generator expression's usage pretty limited ?
    As when the program/system goes beyond a single module, this behaviour
    can cause subtle bugs ?
     
    , Oct 16, 2005
    #1
    1. Advertisements

  2. wrote:

    > I initially thought that generator/generator expression is cool (sort of
    > like the lazy evaluation in Haskell) until I notice this side effect.
    >
    > >>>a=(x for x in range(2))
    > >>>list(a)

    > [1,2]
    > >>>list(a)

    > []
    >
    > Would this make generator/generator expression's usage pretty limited ?


    nope.

    > As when the program/system goes beyond a single module, this behaviour
    > can cause subtle bugs ?


    sure, in the same way as

    >>> f = open(filename)
    >>> f.read()

    'hello world\n'
    >>> f.read() # oops!

    ''

    causes subtle bugs (that is, almost never)

    </F>
     
    Fredrik Lundh, Oct 16, 2005
    #2
    1. Advertisements

  3. Guest

    That is exactly what I meant, in fact. These IO thing are expected to
    have side effects so they are not subtle. Generator on the other hand,
    is sort of "clever iteratables".

    Now that I notice that, Of course I can be sure I would be careful. But
    what about the following situation :

    I import some function from some third party module which said, "oh, my
    function returns a iteratable". Then I can only list(o) if I want to
    pass it around or I consume it in only one and only one place. turning
    it into a list completely negate what generator is intended
    for(generate as much as you need) and using it in only one and only one
    place(like your fread example) would IMO means its usage is pretty
    limited.

    Beside, I can do a fseek to rewind a file handle but not a generator
    object.

    Fredrik Lundh wrote:
    >
    > > As when the program/system goes beyond a single module, this behaviour
    > > can cause subtle bugs ?

    >
    > sure, in the same way as
    >
    > >>> f = open(filename)
    > >>> f.read()

    > 'hello world\n'
    > >>> f.read() # oops!

    > ''
    >
    > causes subtle bugs (that is, almost never)
    >
    > </F>
     
    , Oct 16, 2005
    #3
  4. wrote:
    > That is exactly what I meant, in fact. These IO thing are expected to
    > have side effects so they are not subtle. Generator on the other hand,
    > is sort of "clever iteratables".
    >
    > Now that I notice that, Of course I can be sure I would be careful. But
    > what about the following situation :
    >
    > I import some function from some third party module which said, "oh, my
    > function returns a iteratable". Then I can only list(o) if I want to
    > pass it around or I consume it in only one and only one place. turning
    > it into a list completely negate what generator is intended
    > for(generate as much as you need) and using it in only one and only one
    > place(like your fread example) would IMO means its usage is pretty
    > limited.
    >
    > Beside, I can do a fseek to rewind a file handle but not a generator
    > object.


    Its a question of protocol. A iterator only guarantes that you can fetch
    items from it until it is possibly exhausted. In the same way, a stream
    may only guarantee you that you can read from it. If you need more, you
    have tgo use a different protocol. Lists are one that allows you to
    randomly access it. Files allow to seek, in addition to stream semantics.

    And given that python is an imperative language with side-effects, you
    don't know what sideeffects a generator can cause - which makes a
    "rewind" or copy-semantics to make it work several time impossible (in
    the general case, that is). So it has to focus on what it _can_ guarantee.

    Haskell and other FP languages canwork differently here, as sideeffects
    are made excplicit using e.g. monads or other means - which allows for
    re-use of a generator-like construct. But there is no way to make this
    work in pythons paradigm.

    Diez
     
    Diez B. Roggisch, Oct 16, 2005
    #4
  5. Diez B. Roggisch wrote:

    > Files allow to seek, in addition to stream semantics.


    Some files. Not all files support seek operations. Some only support
    forward seek.

    </F>
     
    Fredrik Lundh, Oct 16, 2005
    #5
  6. If you find that you want to iterate over an iterable multiple times,
    have a look at the solution that the tee() function in the itertools
    module provides (http://docs.python.org/lib/itertools-functions.html).
    (Have a look at the rest of the itertools module as well, for that
    matter.)
     
    Simon Percivall, Oct 16, 2005
    #6
  7. Guest

    True. That is why I have now reverted back to use list whenever
    possible. As while list can also be modified(say in a multi-thread
    situation), at least if I don't do the update(coding policy, practice
    or whatever), they are sort of "guaranteed".

    I would only use generator as IO monad in Haskell, i.e. don't use it
    unless I absolutely need to.

    Diez B. Roggisch wrote:
    >
    > Its a question of protocol. A iterator only guarantes that you can fetch
    > items from it until it is possibly exhausted. In the same way, a stream
    > may only guarantee you that you can read from it. If you need more, you
    > have tgo use a different protocol. Lists are one that allows you to
    > randomly access it. Files allow to seek, in addition to stream semantics.
    >
    > And given that python is an imperative language with side-effects, you
    > don't know what sideeffects a generator can cause - which makes a
    > "rewind" or copy-semantics to make it work several time impossible (in
    > the general case, that is). So it has to focus on what it _can_ guarantee.
    >
    > Haskell and other FP languages canwork differently here, as sideeffects
    > are made excplicit using e.g. monads or other means - which allows for
    > re-use of a generator-like construct. But there is no way to make this
    > work in pythons paradigm.
    >
    > Diez
     
    , Oct 16, 2005
    #7
  8. Guest

    thanks. I was looking for scanl in itertools but can't find it so I
    implement my own then run into some subtle bugs which first made me
    think my scanl is the problem. Then notice my wrong perception about
    generator(and iterable in general, though the built-in iterables like
    list, dict don't seem to have this characteristic).

    Simon Percivall wrote:
    > If you find that you want to iterate over an iterable multiple times,
    > have a look at the solution that the tee() function in the itertools
    > module provides (http://docs.python.org/lib/itertools-functions.html).
    > (Have a look at the rest of the itertools module as well, for that
    > matter.)
     
    , Oct 16, 2005
    #8
  9. On Sun, 16 Oct 2005 15:52:54 +0200, Fredrik Lundh wrote:

    > wrote:
    >
    >> I initially thought that generator/generator expression is cool (sort of
    >> like the lazy evaluation in Haskell) until I notice this side effect.
    >>
    >> >>>a=(x for x in range(2))
    >> >>>list(a)

    >> [1,2]
    >> >>>list(a)

    >> []
    >>
    >> Would this make generator/generator expression's usage pretty limited ?

    >
    > nope.


    In fairness, it is a pretty big Gotcha.



    >> As when the program/system goes beyond a single module, this behaviour
    >> can cause subtle bugs ?

    >
    > sure, in the same way as
    >
    > >>> f = open(filename)
    > >>> f.read()

    > 'hello world\n'
    > >>> f.read() # oops!

    > ''
    >
    > causes subtle bugs (that is, almost never)


    Are you saying that the bugs it causes aren't subtle? *wink*


    --
    Steven
     
    Steven D'Aprano, Oct 16, 2005
    #9
  10. Guest

    > Are you saying that the bugs it causes aren't subtle? *wink*

    Exactly. Destructive generator problems are caught almost immediately.
     
    , Oct 16, 2005
    #10
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John E. Jardine

    s/// has apparent side effect on grep()

    John E. Jardine, Apr 12, 2004, in forum: Perl
    Replies:
    2
    Views:
    627
    John Jardine
    Apr 13, 2004
  2. =?Utf-8?B?R2lhbmNhcmxv?=

    Strange side effect with events

    =?Utf-8?B?R2lhbmNhcmxv?=, Jan 13, 2005, in forum: ASP .Net
    Replies:
    2
    Views:
    463
    =?Utf-8?B?R2lhbmNhcmxv?=
    Jan 13, 2005
  3. Roedy Green
    Replies:
    18
    Views:
    1,170
    Raymond DeCampo
    Sep 25, 2005
  4. DaKoadMunky
    Replies:
    1
    Views:
    516
    Ron Natalie
    Aug 18, 2004
  5. Sensorflo
    Replies:
    3
    Views:
    863
    Ron Natalie
    Aug 19, 2004
  6. Keith Doyle

    Undefined side-effect of setvbuf?

    Keith Doyle, Aug 23, 2003, in forum: C Programming
    Replies:
    2
    Views:
    524
    Alan Balmer
    Aug 25, 2003
  7. snacktime
    Replies:
    13
    Views:
    737
    Dan Sommers
    Feb 22, 2005
  8. Replies:
    7
    Views:
    355
Loading...