breaking a list into smaller lists

Discussion in 'Python' started by Bart Nessux, Jun 17, 2004.

  1. Bart Nessux

    Bart Nessux Guest

    I understand how a Python list can be broken into smaller lists by using
    slice like this:

    new_small_list_1 = BIG_LIST[0:1001] #0-1000
    new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
    new_small_list_3 = BIG_LIST[2001:3001] #2001-3000
    ....
    ....
    ....

    However, I was wondering if there's an easier way to do this. For
    example, say I have a list that contains the names of 100,000 files that
    I wish to open and read. Because of file handle limits, I can only open
    roughly 1000 files at a time while the OS is doing other things.

    Is there a way to turn this big list into 100 small lists (each
    containing 1000 files) with one simple line of code?

    Thanks,
    Bart
     
    Bart Nessux, Jun 17, 2004
    #1
    1. Advertising

  2. Bart Nessux wrote:

    > Is there a way to turn this big list into 100 small lists (each
    > containing 1000 files) with one simple line of code?
    >
    > Thanks,
    > Bart


    big = range(87)

    print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
    10)>0) ) ]

    replace 10 with any value you need, but it is better not to keep all 100 000
    filenames in memory if it isn't necessary

    --
    Maciej "Fiedzia" Dziardziel (fiedzia (at) fiedzia (dot) prv (dot) pl)
    www.fiedzia.prv.pl
     
    Maciej Dziardziel, Jun 17, 2004
    #2
    1. Advertising

  3. Bart Nessux

    Bart Nessux Guest

    Maciej Dziardziel wrote:
    > Bart Nessux wrote:
    >
    >
    >>Is there a way to turn this big list into 100 small lists (each
    >>containing 1000 files) with one simple line of code?
    >>
    >>Thanks,
    >>Bart

    >
    >
    > big = range(87)
    >
    > print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
    > 10)>0) ) ]
    >
    > replace 10 with any value you need, but it is better not to keep all 100 000
    > filenames in memory if it isn't necessary
    >


    Thank you for this example. It works very well.
     
    Bart Nessux, Jun 17, 2004
    #3
  4. Bart Nessux

    Phil Frost Guest

    One could easilly break it into a list of small lists:

    small = [ BIG_LIST[i:i+1000] for i in xrange(0,len(BIG_LIST),1000) ]

    On Wed, Jun 16, 2004 at 07:04:10PM -0400, Bart Nessux wrote:
    > I understand how a Python list can be broken into smaller lists by using
    > slice like this:
    >
    > new_small_list_1 = BIG_LIST[0:1001] #0-1000
    > new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
    > new_small_list_3 = BIG_LIST[2001:3001] #2001-3000
    > ...
    > ...
    > ...
    >
    > However, I was wondering if there's an easier way to do this. For
    > example, say I have a list that contains the names of 100,000 files that
    > I wish to open and read. Because of file handle limits, I can only open
    > roughly 1000 files at a time while the OS is doing other things.
    >
    > Is there a way to turn this big list into 100 small lists (each
    > containing 1000 files) with one simple line of code?
    >
    > Thanks,
    > Bart
     
    Phil Frost, Jun 17, 2004
    #4
  5. Bart Nessux

    Terry Reedy Guest

    "Bart Nessux" <> wrote in message
    news:caqjke$mh8$...
    > I understand how a Python list can be broken into smaller lists by using
    > slice like this:
    >
    > new_small_list_1 = BIG_LIST[0:1001] #0-1000


    this is 1001 items: you probably want 0:1000

    > new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
    > new_small_list_3 = BIG_LIST[2001:3001] #2001-3000


    these are 1000 items, but with off-by-one correction, 1000:2000, etc

    >
    > However, I was wondering if there's an easier way to do this. For
    > example, say I have a list that contains the names of 100,000 files that
    > I wish to open and read. Because of file handle limits, I can only open
    > roughly 1000 files at a time while the OS is doing other things.
    >
    > Is there a way to turn this big list into 100 small lists (each
    > containing 1000 files) with one simple line of code?


    using *your* pattern, corrected, but untested and written while slightly
    tired:
    [biglist[i:i+1000] for i in range(len(biglist)/1000)]

    Terry J. Reedy
     
    Terry Reedy, Jun 17, 2004
    #5
  6. Maciej Dziardziel wrote:
    > Bart Nessux wrote:
    >
    >> Is there a way to turn this big list into 100 small lists (each
    >> containing 1000 files) with one simple line of code?
    >>
    >> Thanks,
    >> Bart

    >
    > big = range(87)
    >
    > print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
    > 10)>0) ) ]


    I would add that you should definitely use the // operator in this case
    as this code will break in 2.4.

    Also, I would leave the modulo part out and write the thing as follows:

    print [ big[i*10:(i+1)*10] for i in xrange((len(big)-1) // 10 + 1) ]

    Reinhold

    --
    Wenn eine Linuxdistribution so wenig brauchbare Software wie Windows
    mitbrächte, wäre das bedauerlich. Was bei Windows der Umfang eines
    "kompletten Betriebssystems" ist, nennt man bei Linux eine Rescuedisk.
    -- David Kastrup in de.comp.os.unix.linux.misc
     
    Reinhold Birkenfeld, Jun 17, 2004
    #6

  7. > I would add that you should definitely use the // operator in this case
    > as this code will break in 2.4.

    2.4? It seems that in PEP: 238, it will break only in 3.0
    Does anyone can confirm ?

    " - Classic division will remain the default in the Python 2.x
    series; true division will be standard in Python 3.0. "

    Xavier
     
    Xavier Combelle, Jun 17, 2004
    #7
  8. Xavier Combelle wrote:
    >> I would add that you should definitely use the // operator in this case
    >> as this code will break in 2.4.

    > 2.4? It seems that in PEP: 238, it will break only in 3.0
    > Does anyone can confirm ?
    >
    > " - Classic division will remain the default in the Python 2.x
    > series; true division will be standard in Python 3.0. "


    Sorry, it's my fault, I didn't recall it properly.

    Reinhold

    --
    Wenn eine Linuxdistribution so wenig brauchbare Software wie Windows
    mitbrächte, wäre das bedauerlich. Was bei Windows der Umfang eines
    "kompletten Betriebssystems" ist, nennt man bei Linux eine Rescuedisk.
    -- David Kastrup in de.comp.os.unix.linux.misc
     
    Reinhold Birkenfeld, Jun 17, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amy
    Replies:
    0
    Views:
    513
  2. flamesrock
    Replies:
    3
    Views:
    620
    gene tani
    Jun 7, 2005
  3. =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==

    List of lists of lists of lists...

    =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==, May 8, 2006, in forum: Python
    Replies:
    5
    Views:
    410
    =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==
    May 15, 2006
  4. Jason
    Replies:
    7
    Views:
    4,470
  5. Armin
    Replies:
    12
    Views:
    1,060
    Steve Holden
    Mar 23, 2009
Loading...

Share This Page