breaking a list into smaller lists

[Bart Nessux]

I understand how a Python list can be broken into smaller lists by using
slice like this:

new_small_list_1 = BIG_LIST[0:1001] #0-1000
new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
new_small_list_3 = BIG_LIST[2001:3001] #2001-3000
....
....
....

However, I was wondering if there's an easier way to do this. For
example, say I have a list that contains the names of 100,000 files that
I wish to open and read. Because of file handle limits, I can only open
roughly 1000 files at a time while the OS is doing other things.

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart

 

[Maciej Dziardziel]

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart

big = range(87)

print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
10)>0) ) ]

replace 10 with any value you need, but it is better not to keep all 100 000
filenames in memory if it isn't necessary

 

[Bart Nessux]

Bart Nessux wrote:

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart

big = range(87)

print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
10)>0) ) ]

replace 10 with any value you need, but it is better not to keep all 100 000
filenames in memory if it isn't necessary

Thank you for this example. It works very well.

 

[Phil Frost]

One could easilly break it into a list of small lists:

small = [ BIG_LIST[i:i+1000] for i in xrange(0,len(BIG_LIST),1000) ]

 

[Terry Reedy]

I understand how a Python list can be broken into smaller lists by using
slice like this:

new_small_list_1 = BIG_LIST[0:1001] #0-1000

this is 1001 items: you probably want 0:1000

new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
new_small_list_3 = BIG_LIST[2001:3001] #2001-3000

these are 1000 items, but with off-by-one correction, 1000:2000, etc

However, I was wondering if there's an easier way to do this. For
example, say I have a list that contains the names of 100,000 files that
I wish to open and read. Because of file handle limits, I can only open
roughly 1000 files at a time while the OS is doing other things.

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

using *your* pattern, corrected, but untested and written while slightly
tired:
[biglist[i:i+1000] for i in range(len(biglist)/1000)]

Terry J. Reedy

 

[Reinhold Birkenfeld]

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart

big = range(87)

print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
10)>0) ) ]

I would add that you should definitely use the // operator in this case
as this code will break in 2.4.

Also, I would leave the modulo part out and write the thing as follows:

print [ big[i*10i+1)*10] for i in xrange((len(big)-1) // 10 + 1) ]

Reinhold

 

[Xavier Combelle]

I would add that you should definitely use the // operator in this case
as this code will break in 2.4.

2.4? It seems that in PEP: 238, it will break only in 3.0
Does anyone can confirm ?

" - Classic division will remain the default in the Python 2.x
series; true division will be standard in Python 3.0. "

Xavier

 

[Reinhold Birkenfeld]

2.4? It seems that in PEP: 238, it will break only in 3.0
Does anyone can confirm ?

" - Classic division will remain the default in the Python 2.x
series; true division will be standard in Python 3.0. "

Sorry, it's my fault, I didn't recall it properly.

Reinhold