Best way to add a "position" value to each item in a list

Discussion in 'Python' started by Sean, Jul 9, 2009.

  1. Sean

    Sean Guest

    I have a huge list, 10,000,000+ items. Each item is a dictionary with
    fields used to sort the list. When I have completed sorting I want to
    grab a page of items, say 1,000 of them which I do easily by using
    list_data[x:x+1000]

    Now I want to add an additional key/value pair to each dictionary in
    the list, incrementing them by 1 each time. So, if I grabbed page 2
    of the list I would get:

    [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
    1002}, ...]

    Any way to do that with list comprehension? Any other good way to do
    it besides iterating over the list?

    Thanks
     
    Sean, Jul 9, 2009
    #1
    1. Advertising

  2. Sean

    Sean Guest

    On Jul 9, 1:16 pm, Sean <> wrote:
    > I have a huge list, 10,000,000+ items.  Each item is a dictionary with
    > fields used to sort the list.  When I have completed sorting I want to
    > grab a page of items, say 1,000 of them which I do easily by using
    > list_data[x:x+1000]
    >
    > Now I want to add an additional key/value pair to each dictionary in
    > the list, incrementing them by 1 each time.  So, if I grabbed page 2
    > of the list I would get:
    >
    > [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
    > 1002}, ...]
    >
    > Any way to do that with list comprehension?  Any other good way to do
    > it besides iterating over the list?
    >
    > Thanks


    I was able to do this by doing the following:
    page_data = [[start_position + 1 + n, x] for n, x in enumerate
    (page_data)]

    However, this creates a list of lists, each containing an int and a
    dictionary. I wanted to add it directly to the dictionary. I can add
    the key position to the dictionary when the data is created, but how
    would I assign the value in a list comprehension?
     
    Sean, Jul 9, 2009
    #2
    1. Advertising

  3. Howdy,

    On Thu, Jul 9, 2009 at 15:16, Sean<> wrote:
    > I have a huge list, 10,000,000+ items.  Each item is a dictionary with
    > fields used to sort the list.  When I have completed sorting I want to
    > grab a page of items, say 1,000 of them which I do easily by using
    > list_data[x:x+1000]
    >
    > Now I want to add an additional key/value pair to each dictionary in
    > the list, incrementing them by 1 each time.  So, if I grabbed page 2
    > of the list I would get:
    >
    > [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
    > 1002}, ...]
    >


    I don't get it, what do you increment by one, the value of a given key
    or the number of key/value pairs? Also, if the 'position' key is the
    index of the item in the list, then I don't understand what you mean
    by 'page'. Could you tell us about the structure of these
    dictionaries?

    > Any way to do that with list comprehension?  Any other good way to do
    > it besides iterating over the list?
    >
    > Thanks
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >




    --
    Pablo Torres N.
     
    Pablo Torres N., Jul 9, 2009
    #3
  4. Sean

    Jerry Hill Guest

    On Thu, Jul 9, 2009 at 4:16 PM, Sean<> wrote:
    > I have a huge list, 10,000,000+ items.  Each item is a dictionary with
    > fields used to sort the list.  When I have completed sorting I want to
    > grab a page of items, say 1,000 of them which I do easily by using
    > list_data[x:x+1000]
    >
    > Now I want to add an additional key/value pair to each dictionary in
    > the list, incrementing them by 1 each time.  So, if I grabbed page 2
    > of the list I would get:
    >
    > [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
    > 1002}, ...]
    >
    > Any way to do that with list comprehension?  Any other good way to do
    > it besides iterating over the list?


    Normally you wouldn't mutate the items in a list with a list
    comprehension. Instead, you would use a for loop, like this:

    for idx,item in enumerate(my_list_of_dicts):
    item['position'] = idx

    Is there a particular reason you want to do this with a list
    comprehension? Using a list comp means you're going to create an
    extra copy of your 10 million item list, just so you can add a key to
    each member, then (probably) throw away the original list. It doesn't
    seem like the right tool for the job here.

    --
    Jerry
     
    Jerry Hill, Jul 9, 2009
    #4
  5. Sean

    Sean Guest

    On Jul 9, 1:16 pm, Sean <> wrote:
    > I have a huge list, 10,000,000+ items.  Each item is a dictionary with
    > fields used to sort the list.  When I have completed sorting I want to
    > grab a page of items, say 1,000 of them which I do easily by using
    > list_data[x:x+1000]
    >
    > Now I want to add an additional key/value pair to each dictionary in
    > the list, incrementing them by 1 each time.  So, if I grabbed page 2
    > of the list I would get:
    >
    > [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
    > 1002}, ...]
    >
    > Any way to do that with list comprehension?  Any other good way to do
    > it besides iterating over the list?
    >
    > Thanks


    I was able to do this by doing the following:
    page_data = [[start_position + 1 + n, x] for n, x in enumerate
    (page_data)]

    However, this creates a list of lists, each containing an int and a
    dictionary. I wanted to add it directly to the dictionary. I can add
    the key position to the dictionary when the data is created, but how
    would I assign the value in a list comprehension?
     
    Sean, Jul 9, 2009
    #5
  6. Sean

    Carl Banks Guest

    On Jul 9, 1:16 pm, Sean <> wrote:
    > I have a huge list, 10,000,000+ items.  Each item is a dictionary with
    > fields used to sort the list.  When I have completed sorting I want to
    > grab a page of items, say 1,000 of them which I do easily by using
    > list_data[x:x+1000]
    >
    > Now I want to add an additional key/value pair to each dictionary in
    > the list, incrementing them by 1 each time.  So, if I grabbed page 2
    > of the list I would get:
    >
    > [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
    > 1002}, ...]
    >
    > Any way to do that with list comprehension?  Any other good way to do
    > it besides iterating over the list?



    Not really, but if you want to avoid the delay in setting the
    dictionary elements (I'm guessing that is why you don't want to
    iterate over the list--and parenthetically it's not iterating over the
    list but adding dictionary items that is driving the time, and that
    cost is unavoidable), you should consider adding the position elements
    on demand. That is, instead of running a big loop once to add
    position to all ten million elements, just run the loop on individual
    pages.

    def get_page(start,end):
    page = list_data[start:end]
    for i,item in enumerate(page):
    page['position'] = start+i
    return page


    That might not work depending on what you are doing but you should
    consider it.


    Carl Banks
     
    Carl Banks, Jul 9, 2009
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ssoss

    Display Related Item or Add Item

    ssoss, Sep 17, 2003, in forum: ASP .Net
    Replies:
    2
    Views:
    489
    ssoss
    Sep 19, 2003
  2. Replies:
    5
    Views:
    473
  3. Replies:
    0
    Views:
    322
  4. QUASAR
    Replies:
    6
    Views:
    424
    QUASAR
    Jan 17, 2004
  5. Namor
    Replies:
    0
    Views:
    277
    Namor
    Jan 19, 2006
Loading...

Share This Page