Slicing iterables in sub-generators without loosing elements

Discussion in 'Python' started by Thomas Bach, Sep 29, 2012.

  1. Thomas Bach

    Thomas Bach Guest

    Hi,

    say we have the following:

    >>> data = [('foo', 1), ('foo', 2), ('bar', 3), ('bar', 2)]


    is there a way to code a function iter_in_blocks such that

    >>> result = [ list(block) for block in iter_in_blocks(data) ]


    evaluates to

    >>> result = [ [('foo', 1), ('foo', 2)], [('bar', 3), ('bar', 2)] ]


    by _only_ _iterating_ over the list (caching all the elements sharing
    the same first element doesn't count)?

    I came up with the following

    def iter_in_blocks(iterable):
    my_iter = iter(iterable)
    while True:
    first = next(my_iter)
    pred = lambda entry: entry[0] == first[0]
    def block_iter():
    yield first
    for entry in itertools.takewhile(pred, my_iter):
    yield entry
    yield block_iter()

    which does not work as itertools.takewhile consumes the first entry
    not fulfilling the pred.

    I currently have the intuition that the problem is not solvable
    without using e.g. a global to pass something back to iter_in_blocks
    from block_iter. Any other suggestions?

    Regards,
    Thomas Bach.
    Thomas Bach, Sep 29, 2012
    #1
    1. Advertising

  2. Thomas Bach

    Paul Rubin Guest

    Thomas Bach <-mainz.de> writes:
    >>>> result = [ [('foo', 1), ('foo', 2)], [('bar', 3), ('bar', 2)] ]

    > by _only_ _iterating_ over the list (caching all the elements sharing
    > the same first element doesn't count)?


    itertools.groupby(data, lambda (x,y) : x)

    is basically what you want.
    Paul Rubin, Sep 29, 2012
    #2
    1. Advertising

  3. Thomas Bach

    Thomas Bach Guest

    On Sat, Sep 29, 2012 at 09:26:00AM -0700, Paul Rubin wrote:
    > Thomas Bach <-mainz.de> writes:
    >
    > itertools.groupby(data, lambda (x,y) : x)
    >
    > is basically what you want.


    True!

    Thanks,
    Thomas Bach
    Thomas Bach, Sep 29, 2012
    #3
  4. On Sunday, September 30, 2012 12:15:57 AM UTC+8, Thomas Bach wrote:
    > Hi,
    >
    >
    >
    > say we have the following:
    >
    >
    >
    > >>> data = [('foo', 1), ('foo', 2), ('bar', 3), ('bar', 2)]

    >
    >
    >
    > is there a way to code a function iter_in_blocks such that
    >
    >
    >
    > >>> result = [ list(block) for block in iter_in_blocks(data) ]

    >
    >
    >
    > evaluates to
    >
    >
    >
    > >>> result = [ [('foo', 1), ('foo', 2)], [('bar', 3), ('bar', 2)] ]

    >
    >
    >
    > by _only_ _iterating_ over the list (caching all the elements sharing
    >
    > the same first element doesn't count)?
    >
    >
    >
    > I came up with the following
    >
    >
    >
    > def iter_in_blocks(iterable):
    >
    > my_iter = iter(iterable)
    >
    > while True:
    >
    > first = next(my_iter)
    >
    > pred = lambda entry: entry[0] == first[0]
    >
    > def block_iter():
    >
    > yield first
    >
    > for entry in itertools.takewhile(pred, my_iter):
    >
    > yield entry
    >
    > yield block_iter()
    >
    >
    >
    > which does not work as itertools.takewhile consumes the first entry
    >
    > not fulfilling the pred.
    >
    >
    >
    > I currently have the intuition that the problem is not solvable
    >
    > without using e.g. a global to pass something back to iter_in_blocks
    >
    > from block_iter. Any other suggestions?
    >
    >
    >
    > Regards,
    >
    > Thomas Bach.


    Your question seems vague to me. If you know you are storing
    only immutable tuples in a list, then the way to iterate is simple.

    For example:

    data = [('foo', 1), ('foo', 2), ('bar', 3), ('bar', 2)]
    # all tuples

    for item in data:
    x1=item[0] # first entry in each tuple
    x2=item[1]
    print x1, x2 # or use yield in a function to iterate
    88888 Dihedral, Oct 1, 2012
    #4
  5. On Sunday, September 30, 2012 12:15:57 AM UTC+8, Thomas Bach wrote:
    > Hi,
    >
    >
    >
    > say we have the following:
    >
    >
    >
    > >>> data = [('foo', 1), ('foo', 2), ('bar', 3), ('bar', 2)]

    >
    >
    >
    > is there a way to code a function iter_in_blocks such that
    >
    >
    >
    > >>> result = [ list(block) for block in iter_in_blocks(data) ]

    >
    >
    >
    > evaluates to
    >
    >
    >
    > >>> result = [ [('foo', 1), ('foo', 2)], [('bar', 3), ('bar', 2)] ]

    >
    >
    >
    > by _only_ _iterating_ over the list (caching all the elements sharing
    >
    > the same first element doesn't count)?
    >
    >
    >
    > I came up with the following
    >
    >
    >
    > def iter_in_blocks(iterable):
    >
    > my_iter = iter(iterable)
    >
    > while True:
    >
    > first = next(my_iter)
    >
    > pred = lambda entry: entry[0] == first[0]
    >
    > def block_iter():
    >
    > yield first
    >
    > for entry in itertools.takewhile(pred, my_iter):
    >
    > yield entry
    >
    > yield block_iter()
    >
    >
    >
    > which does not work as itertools.takewhile consumes the first entry
    >
    > not fulfilling the pred.
    >
    >
    >
    > I currently have the intuition that the problem is not solvable
    >
    > without using e.g. a global to pass something back to iter_in_blocks
    >
    > from block_iter. Any other suggestions?
    >
    >
    >
    > Regards,
    >
    > Thomas Bach.


    Your question seems vague to me. If you know you are storing
    only immutable tuples in a list, then the way to iterate is simple.

    For example:

    data = [('foo', 1), ('foo', 2), ('bar', 3), ('bar', 2)]
    # all tuples

    for item in data:
    x1=item[0] # first entry in each tuple
    x2=item[1]
    print x1, x2 # or use yield in a function to iterate
    88888 Dihedral, Oct 1, 2012
    #5
  6. On 01/10/2012 01:58, 88888 Dihedral wrote:
    >
    > Your question seems vague to me. If you know you are storing
    > only immutable tuples in a list, then the way to iterate is simple.
    >


    Does Python have a magic method that let's me use mutable tuples? I'd
    also like immutable lists. Is it worth raising a feature request on the
    bug tracker?

    --
    Cheers.

    Mark Lawrence.
    Mark Lawrence, Oct 1, 2012
    #6
  7. On Monday, 1 October 2012 13:47:50 UTC+5:30, Mark Lawrence wrote:
    > On 01/10/2012 01:58, 88888 Dihedral wrote:
    >
    > >

    >
    > > Your question seems vague to me. If you know you are storing

    >
    > > only immutable tuples in a list, then the way to iterate is simple.

    >
    > >

    >
    >
    >
    > Does Python have a magic method that let's me use mutable tuples? I'd
    >
    > also like immutable lists. Is it worth raising a feature request on the
    >
    > bug tracker?
    >
    >
    >
    > --
    >
    > Cheers.
    >
    >
    >
    > Mark Lawrence.


    Mark, you are talking to a bot.
    Ramchandra Apte, Oct 2, 2012
    #7
  8. On Monday, 1 October 2012 13:47:50 UTC+5:30, Mark Lawrence wrote:
    > On 01/10/2012 01:58, 88888 Dihedral wrote:
    >
    > >

    >
    > > Your question seems vague to me. If you know you are storing

    >
    > > only immutable tuples in a list, then the way to iterate is simple.

    >
    > >

    >
    >
    >
    > Does Python have a magic method that let's me use mutable tuples? I'd
    >
    > also like immutable lists. Is it worth raising a feature request on the
    >
    > bug tracker?
    >
    >
    >
    > --
    >
    > Cheers.
    >
    >
    >
    > Mark Lawrence.


    Mark, you are talking to a bot.
    Ramchandra Apte, Oct 2, 2012
    #8
  9. On 02/10/2012 17:12, Ramchandra Apte wrote:
    > On Monday, 1 October 2012 13:47:50 UTC+5:30, Mark Lawrence wrote:
    >> On 01/10/2012 01:58, 88888 Dihedral wrote:
    >>
    >>>

    >>
    >>> Your question seems vague to me. If you know you are storing

    >>
    >>> only immutable tuples in a list, then the way to iterate is simple.

    >>
    >>>

    >>
    >>
    >>
    >> Does Python have a magic method that let's me use mutable tuples? I'd
    >>
    >> also like immutable lists. Is it worth raising a feature request on the
    >>
    >> bug tracker?
    >>
    >>
    >>
    >> --
    >>
    >> Cheers.
    >>
    >>
    >>
    >> Mark Lawrence.

    >
    > Mark, you are talking to a bot.
    >


    What happened to freedom of speech? If I want to talk to a bot, I'll
    talk to a bot. Besides I'm not convinced it/he/she is a bot. Plus if
    you read my post carefully, add in several years experience of Python
    the language and Python the comedy, you might come to the conclusion
    that a certain amount of urine extraction was going on :)

    --
    Cheers.

    Mark Lawrence.
    Mark Lawrence, Oct 2, 2012
    #9
  10. On Wed, Oct 3, 2012 at 2:44 AM, Mark Lawrence <> wrote:
    > What happened to freedom of speech? If I want to talk to a bot, I'll talk
    > to a bot. Besides I'm not convinced it/he/she is a bot. Plus if you read
    > my post carefully, add in several years experience of Python the language
    > and Python the comedy, you might come to the conclusion that a certain
    > amount of urine extraction was going on :)


    Coupled with a bit of bot-seeding, which is always fun.

    One of these days I'm going to mail Dihedral a whole pile of Gilbert
    and Sullivan operetta and see if any of it comes back in his posts...

    Dihedral might be a bot and might not. I've come to the conclusion
    that it's not worth trying to find out, given that a good bot can
    outdo a lot of humans in useful conversation.

    ChrisA
    Chris Angelico, Oct 2, 2012
    #10
  11. On 02/10/2012 18:58, Chris Angelico wrote:
    >
    > Dihedral might be a bot and might not. I've come to the conclusion
    > that it's not worth trying to find out, given that a good bot can
    > outdo a lot of humans in useful conversation.
    >
    > ChrisA
    >


    Try telling that to the newbies on the Python tutor mailing list and
    they simply won't believe ya :)

    --
    Cheers.

    Mark Lawrence.
    Mark Lawrence, Oct 2, 2012
    #11
  12. Thomas Bach

    Terry Reedy Guest

    On 10/2/2012 1:58 PM, Chris Angelico wrote:
    > On Wed, Oct 3, 2012 at 2:44 AM, Mark Lawrence <> wrote:
    >> What happened to freedom of speech? If I want to talk to a bot, I'll talk
    >> to a bot. Besides I'm not convinced it/he/she is a bot. Plus if you read
    >> my post carefully, add in several years experience of Python the language
    >> and Python the comedy, you might come to the conclusion that a certain
    >> amount of urine extraction was going on :)

    >
    > Coupled with a bit of bot-seeding, which is always fun.
    >
    > One of these days I'm going to mail Dihedral a whole pile of Gilbert
    > and Sullivan operetta and see if any of it comes back in his posts...
    >
    > Dihedral might be a bot and might not. I've come to the conclusion
    > that it's not worth trying to find out, given that a good bot can
    > outdo a lot of humans in useful conversation.


    I just read that bots playing Unreal Tournament with an 'act human'
    module seem more human to at least some humans than humans. Perhaps that
    is because humans have to become bot-like to play UT well.

    --
    Terry Jan Reedy
    Terry Reedy, Oct 2, 2012
    #12
  13. On Monday, October 1, 2012 4:17:50 PM UTC+8, Mark Lawrence wrote:
    > On 01/10/2012 01:58, 88888 Dihedral wrote:
    >
    > >

    >
    > > Your question seems vague to me. If you know you are storing

    >
    > > only immutable tuples in a list, then the way to iterate is simple.

    >
    > >

    >
    >
    >
    > Does Python have a magic method that let's me use mutable tuples? I'd
    >
    > also like immutable lists. Is it worth raising a feature request on the
    >
    > bug tracker?
    >
    >
    >
    > --
    >
    > Cheers.
    >
    >
    >
    > Mark Lawrence.


    Python resolves objects by names and dynamical types.
    This will force the programmer to write structured programs.
    88888 Dihedral, Oct 2, 2012
    #13
  14. On Monday, October 1, 2012 4:17:50 PM UTC+8, Mark Lawrence wrote:
    > On 01/10/2012 01:58, 88888 Dihedral wrote:
    >
    > >

    >
    > > Your question seems vague to me. If you know you are storing

    >
    > > only immutable tuples in a list, then the way to iterate is simple.

    >
    > >

    >
    >
    >
    > Does Python have a magic method that let's me use mutable tuples? I'd
    >
    > also like immutable lists. Is it worth raising a feature request on the
    >
    > bug tracker?
    >
    >
    >
    > --
    >
    > Cheers.
    >
    >
    >
    > Mark Lawrence.


    Python resolves objects by names and dynamical types.
    This will force the programmer to write structured programs.
    88888 Dihedral, Oct 2, 2012
    #14
  15. On Wed, 03 Oct 2012 03:58:02 +1000, Chris Angelico wrote:

    > Dihedral might be a bot and might not. I've come to the conclusion that
    > it's not worth trying to find out, given that a good bot can outdo a lot
    > of humans in useful conversation.


    Oh, I'm convinced that it's a bot.

    The fact that Dihedral never responds to conversations about him/it is a
    give away: nearly all people are far to egotistical to let accusations of
    bot-hood go unchallenged. And even though its responses are grammatically
    correct, they often are semantically meaningless. Even the best AIs
    (Evie, CleverBot, Alice) don't come close to passing the Turing test. It
    helps that Dihedral only talks about computer programming, but even so,
    it has well and truly failed my personal Turing test.

    Mind you, some human beings fail the Turing test too:

    http://northernplanets.blogspot.com.au/2006/06/failing-turing-test.html


    Of course, Dihedral's creator wanted to really confound us, the
    occasional human response would probably muddy the waters sufficiently.



    --
    Steven
    Steven D'Aprano, Oct 3, 2012
    #15
  16. Steven D'Apranoæ–¼ 2012å¹´10月3日星期三UTC+8上åˆ8時57分20秒寫é“:
    > On Wed, 03 Oct 2012 03:58:02 +1000, Chris Angelico wrote:
    >
    >
    >
    > > Dihedral might be a bot and might not. I've come to the conclusion that

    >
    > > it's not worth trying to find out, given that a good bot can outdo a lot

    >
    > > of humans in useful conversation.

    >
    >
    >
    > Oh, I'm convinced that it's a bot.
    >
    >
    >
    > The fact that Dihedral never responds to conversations about him/it is a
    >
    > give away: nearly all people are far to egotistical to let accusations of
    >
    > bot-hood go unchallenged. And even though its responses are grammatically
    >
    > correct, they often are semantically meaningless. Even the best AIs
    >
    > (Evie, CleverBot, Alice) don't come close to passing the Turing test. It
    >
    > helps that Dihedral only talks about computer programming, but even so,
    >
    > it has well and truly failed my personal Turing test.
    >
    >
    >
    > Mind you, some human beings fail the Turing test too:
    >
    >
    >
    > http://northernplanets.blogspot.com.au/2006/06/failing-turing-test.html
    >
    >
    >
    >
    >
    > Of course, Dihedral's creator wanted to really confound us, the
    >
    > occasional human response would probably muddy the waters sufficiently.
    >
    >
    >
    >
    >
    >
    >
    > --
    >
    > Steven


    Are you still not getting the generator part in Python?
    Do you really test any generator before?
    88888 Dihedral, Oct 3, 2012
    #16
  17. On Tue, 02 Oct 2012 18:11:20 -0700, 88888 Dihedral wrote:

    > Steven D'Apranoæ–¼ 2012å¹´10月3日星期三UTC+8上åˆ8時57分20秒寫é“:


    >> Oh, I'm convinced that it's a bot.
    >> The fact that Dihedral never responds to conversations about him/it is
    >> a give away: nearly all people are far to egotistical to let
    >> accusations of bot-hood go unchallenged.

    [...]
    >> Of course, Dihedral's creator wanted to really confound us, the
    >> occasional human response would probably muddy the waters sufficiently.


    > Are you still not getting the generator part in Python? Do you really
    > test any generator before?


    I rest my case :)

    I have many problems with generators. Can you tell me more about them? I
    really need your help. Sometimes, when it is very cold, the diesel
    freezes and the generator raises StopIteration at the wrong place.



    --
    Steven
    Steven D'Aprano, Oct 3, 2012
    #17
  18. Steven D'Apranoæ–¼ 2012å¹´10月3日星期三UTC+8上åˆ9時24分13秒寫é“:
    > On Tue, 02 Oct 2012 18:11:20 -0700, 88888 Dihedral wrote:
    >
    >
    >
    > > Steven D'Apranoæ–¼ 2012å¹´10月3日星期三UTC+8上åˆ8時57分20秒寫é“:

    >
    >
    >
    > >> Oh, I'm convinced that it's a bot.

    >
    > >> The fact that Dihedral never responds to conversations about him/it is

    >
    > >> a give away: nearly all people are far to egotistical to let

    >
    > >> accusations of bot-hood go unchallenged.

    >
    > [...]
    >
    > >> Of course, Dihedral's creator wanted to really confound us, the

    >
    > >> occasional human response would probably muddy the waters sufficiently..

    >
    >
    >
    > > Are you still not getting the generator part in Python? Do you really

    >
    > > test any generator before?

    >
    >
    >
    > I rest my case :)
    >
    >
    >
    > I have many problems with generators. Can you tell me more about them? I
    >
    > really need your help. Sometimes, when it is very cold, the diesel
    >
    > freezes and the generator raises StopIteration at the wrong place.
    >
    >
    >
    >
    >
    >
    >
    > --
    >
    > Steven

    Please work out your own templates as I showed in the decorator part.
    88888 Dihedral, Oct 3, 2012
    #18
  19. On Tuesday, 2 October 2012 22:13:20 UTC+5:30, Mark Lawrence wrote:
    > On 02/10/2012 17:12, Ramchandra Apte wrote:
    >
    > > On Monday, 1 October 2012 13:47:50 UTC+5:30, Mark Lawrence wrote:

    >
    > >> On 01/10/2012 01:58, 88888 Dihedral wrote:

    >
    > >>

    >
    > >>>

    >
    > >>

    >
    > >>> Your question seems vague to me. If you know you are storing

    >
    > >>

    >
    > >>> only immutable tuples in a list, then the way to iterate is simple.

    >
    > >>

    >
    > >>>

    >
    > >>

    >
    > >>

    >
    > >>

    >
    > >> Does Python have a magic method that let's me use mutable tuples? I'd

    >
    > >>

    >
    > >> also like immutable lists. Is it worth raising a feature request on the

    >
    > >>

    >
    > >> bug tracker?

    >
    > >>

    >
    > >>

    >
    > >>

    >
    > >> --

    >
    > >>

    >
    > >> Cheers.

    >
    > >>

    >
    > >>

    >
    > >>

    >
    > >> Mark Lawrence.

    >
    > >

    >
    > > Mark, you are talking to a bot.

    >
    > >

    >
    >
    >
    > What happened to freedom of speech? If I want to talk to a bot, I'll
    >
    > talk to a bot. Besides I'm not convinced it/he/she is a bot. Plus if
    >
    > you read my post carefully, add in several years experience of Python
    >
    > the language and Python the comedy, you might come to the conclusion
    >
    > that a certain amount of urine extraction was going on :)
    >
    >
    >
    > --
    >
    > Cheers.
    >
    >
    >
    > Mark Lawrence.


    Never said you can't.
    Ramchandra Apte, Oct 3, 2012
    #19
  20. On Tuesday, 2 October 2012 22:13:20 UTC+5:30, Mark Lawrence wrote:
    > On 02/10/2012 17:12, Ramchandra Apte wrote:
    >
    > > On Monday, 1 October 2012 13:47:50 UTC+5:30, Mark Lawrence wrote:

    >
    > >> On 01/10/2012 01:58, 88888 Dihedral wrote:

    >
    > >>

    >
    > >>>

    >
    > >>

    >
    > >>> Your question seems vague to me. If you know you are storing

    >
    > >>

    >
    > >>> only immutable tuples in a list, then the way to iterate is simple.

    >
    > >>

    >
    > >>>

    >
    > >>

    >
    > >>

    >
    > >>

    >
    > >> Does Python have a magic method that let's me use mutable tuples? I'd

    >
    > >>

    >
    > >> also like immutable lists. Is it worth raising a feature request on the

    >
    > >>

    >
    > >> bug tracker?

    >
    > >>

    >
    > >>

    >
    > >>

    >
    > >> --

    >
    > >>

    >
    > >> Cheers.

    >
    > >>

    >
    > >>

    >
    > >>

    >
    > >> Mark Lawrence.

    >
    > >

    >
    > > Mark, you are talking to a bot.

    >
    > >

    >
    >
    >
    > What happened to freedom of speech? If I want to talk to a bot, I'll
    >
    > talk to a bot. Besides I'm not convinced it/he/she is a bot. Plus if
    >
    > you read my post carefully, add in several years experience of Python
    >
    > the language and Python the comedy, you might come to the conclusion
    >
    > that a certain amount of urine extraction was going on :)
    >
    >
    >
    > --
    >
    > Cheers.
    >
    >
    >
    > Mark Lawrence.


    Never said you can't.
    Ramchandra Apte, Oct 3, 2012
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steven Bethard

    *expression and iterables

    Steven Bethard, Aug 20, 2004, in forum: Python
    Replies:
    0
    Views:
    397
    Steven Bethard
    Aug 20, 2004
  2. Replies:
    7
    Views:
    278
  3. Ben
    Replies:
    2
    Views:
    883
  4. Lawrence D'Oliveiro

    Death To Sub-Sub-Sub-Directories!

    Lawrence D'Oliveiro, May 5, 2011, in forum: Java
    Replies:
    92
    Views:
    2,019
    Lawrence D'Oliveiro
    May 20, 2011
  5. Ian Kelly
    Replies:
    0
    Views:
    190
    Ian Kelly
    Sep 29, 2012
Loading...

Share This Page