Best way to add a "position" value to each item in a list

[Sean]

I have a huge list, 10,000,000+ items. Each item is a dictionary with
fields used to sort the list. When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x:x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time. So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

Any way to do that with list comprehension? Any other good way to do
it besides iterating over the list?

Thanks

 

[Sean]

I have a huge list, 10,000,000+ items.  Each item is a dictionary with
fields used to sort the list.  When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x:x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time.  So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

Any way to do that with list comprehension?  Any other good way to do
it besides iterating over the list?

Thanks

I was able to do this by doing the following:
page_data = [[start_position + 1 + n, x] for n, x in enumerate
(page_data)]

However, this creates a list of lists, each containing an int and a
dictionary. I wanted to add it directly to the dictionary. I can add
the key position to the dictionary when the data is created, but how
would I assign the value in a list comprehension?

 

[Pablo Torres N.]

Howdy,

I have a huge list, 10,000,000+ items.  Each item is a dictionary with
fields used to sort the list.  When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x:x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time.  So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

I don't get it, what do you increment by one, the value of a given key
or the number of key/value pairs? Also, if the 'position' key is the
index of the item in the list, then I don't understand what you mean
by 'page'. Could you tell us about the structure of these
dictionaries?

 

[Jerry Hill]

I have a huge list, 10,000,000+ items.  Each item is a dictionary with
fields used to sort the list.  When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x:x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time.  So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

Any way to do that with list comprehension?  Any other good way to do
it besides iterating over the list?

Normally you wouldn't mutate the items in a list with a list
comprehension. Instead, you would use a for loop, like this:

for idx,item in enumerate(my_list_of_dicts):
item['position'] = idx

Is there a particular reason you want to do this with a list
comprehension? Using a list comp means you're going to create an
extra copy of your 10 million item list, just so you can add a key to
each member, then (probably) throw away the original list. It doesn't
seem like the right tool for the job here.

 

[Sean]

I have a huge list, 10,000,000+ items.  Each item is a dictionary with
fields used to sort the list.  When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x:x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time.  So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

Any way to do that with list comprehension?  Any other good way to do
it besides iterating over the list?

Thanks

I was able to do this by doing the following:
page_data = [[start_position + 1 + n, x] for n, x in enumerate
(page_data)]

However, this creates a list of lists, each containing an int and a
dictionary. I wanted to add it directly to the dictionary. I can add
the key position to the dictionary when the data is created, but how
would I assign the value in a list comprehension?

 

[Carl Banks]

I have a huge list, 10,000,000+ items.  Each item is a dictionary with
fields used to sort the list.  When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x:x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time.  So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

Any way to do that with list comprehension?  Any other good way to do
it besides iterating over the list?

Not really, but if you want to avoid the delay in setting the
dictionary elements (I'm guessing that is why you don't want to
iterate over the list--and parenthetically it's not iterating over the
list but adding dictionary items that is driving the time, and that
cost is unavoidable), you should consider adding the position elements
on demand. That is, instead of running a big loop once to add
position to all ten million elements, just run the loop on individual
pages.

def get_page(start,end):
page = list_data[start:end]
for i,item in enumerate(page):
page['position'] = start+i
return page


That might not work depending on what you are doing but you should
consider it.


Carl Banks