search multiple dictionaries efficiently?

Discussion in 'Python' started by Livin, Jan 18, 2006.

  1. Livin

    Livin Guest

    I'm a noobie so please be easy on me. I have searched a ton and did not find
    anything I could understand.

    I'm using py2.3

    I've been using Try/Except but this gets long with multiple dictionaries.

    I found some code on web pages and such but cannot get them to work. Any
    help is appreciated as I need to search multiple dictionaries for keys.



    here's the code I found but cannot get to work...

    dict_set = (self.dictHSdevices, self.dictHSevents, self.dictHSbtnDevices)
    <-- /creates set of dictionaries/

    /code to search the set/-->

    val = [ d for d in dict_set if d[value] in dict_set ]
    OR
    for key, value in dict.iteritems(): pass
     
    Livin, Jan 18, 2006
    #1
    1. Advertising

  2. Livin

    Paul McGuire Guest

    "Livin" <> wrote in message news:A3izf.418$MJ.192@fed1read07...
    > I'm a noobie so please be easy on me. I have searched a ton and did not

    find
    > anything I could understand.
    >
    > I'm using py2.3
    >
    > I've been using Try/Except but this gets long with multiple dictionaries.
    >
    > I found some code on web pages and such but cannot get them to work. Any
    > help is appreciated as I need to search multiple dictionaries for keys.
    >
    >
    >
    > here's the code I found but cannot get to work...
    >
    > dict_set = (self.dictHSdevices, self.dictHSevents, self.dictHSbtnDevices)
    > <-- /creates set of dictionaries/
    >
    > /code to search the set/-->
    >
    > val = [ d for d in dict_set if d[value] in dict_set ]
    > OR
    > for key, value in dict.iteritems(): pass
    >



    >>> dict_set = [ {1:'a'}, {2:'b',3:'c'} ]
    >>> def lookup(value):

    .... for d in dict_set:
    .... try:
    .... return d[value]
    .... except KeyError:
    .... pass
    .... raise KeyError(value)
    ....
    >>> lookup(1)

    'a'
    >>> lookup(2)

    'b'
    >>> lookup(3)

    'c'
    >>> lookup(4)

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 7, in lookup
    KeyError: 4
    >>>
     
    Paul McGuire, Jan 18, 2006
    #2
    1. Advertising

  3. Livin wrote:

    > I'm a noobie so please be easy on me. I have searched a ton and did not find
    > anything I could understand.
    >
    > I'm using py2.3
    >
    > I've been using Try/Except but this gets long with multiple dictionaries.
    >
    > I found some code on web pages and such but cannot get them to work. Any
    > help is appreciated as I need to search multiple dictionaries for keys.
    >
    >
    >
    > here's the code I found but cannot get to work...
    >
    > dict_set = (self.dictHSdevices, self.dictHSevents, self.dictHSbtnDevices)
    > <-- /creates set of dictionaries/
    >
    > /code to search the set/-->
    >
    > val = [ d for d in dict_set if d[value] in dict_set ]
    > OR
    > for key, value in dict.iteritems(): pass



    If I understood correctly what you want to do, here's a one-liner that
    does it:

    def lookup(value, *dicts):
    return [d.get(value) for d in dicts if value in d]

    >>> a = {'x':1, 'y':2}
    >>> b = {'y':4, 'z':7}
    >>> lookup('y', a, b)

    [2, 4]

    It's not the *most* efficient way because value is looked up twice if
    it is contained in the dictionary; if you absolutely need it to be as
    efficient as possible and can't figure it out on your own, ask again
    and someone will help you out.

    George
     
    George Sakkis, Jan 18, 2006
    #3
  4. Livin

    gene tani Guest

    Livin wrote:
    > I'm a noobie so please be easy on me. I have searched a ton and did not find
    > anything I could understand.
    >
    > I'm using py2.3
    >
    > I've been using Try/Except but this gets long with multiple dictionaries.
    >
    > I found some code on web pages and such but cannot get them to work. Any
    > help is appreciated as I need to search multiple dictionaries for keys.
    >
    >
    >
    > here's the code I found but cannot get to work...
    >
    > dict_set = (self.dictHSdevices, self.dictHSevents, self.dictHSbtnDevices)
    > <-- /creates set of dictionaries/
    >
    > /code to search the set/-->
    >
    > val = [ d for d in dict_set if d[value] in dict_set ]
    > OR
    > for key, value in dict.iteritems(): pass


    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/305268
    http://mail.python.org/pipermail/python-list/2005-August/294519.html

    (I think these may be a little different from what you're asking above,
    since they search dicts sequentially and return 1st hit)
     
    gene tani, Jan 18, 2006
    #4
  5. Livin

    Duncan Booth Guest

    George Sakkis wrote:

    > It's not the *most* efficient way because value is looked up twice if
    > it is contained in the dictionary; if you absolutely need it to be as
    > efficient as possible and can't figure it out on your own, ask again
    > and someone will help you out.
    >

    How do you *know* it is not the *most* efficient way? Have you tried timing
    different ways of approaching this problem and found that looking up the
    value twice is slow?

    I've tried timing dictionary lookups in the past, and the three obvious
    solutions roughly come out as follows:

    try/except is fastest when the value is in the dictionary, but it is a
    *lot* slower if the exception gets thrown. If missing values are a very
    rare occurrence this might be a good way to do it, but usually the code
    doesn't read as well so its best to avoid. [0.26/4.11]

    Test with the 'in' operator and then retrieving the value is the fastest
    solution when the value isn't in the dictionary (it only does a single
    lookup then), and is still fast when it is. [0.36/0.2]

    Using the get method of the dictionary with a default value to be retrieved
    if the key is not present is slower than using the 'in' operator in all
    cases (it does beat try/except when an exception is thrown) [0.49/0.54]

    The numbers above are the times produced in each case for a key present/key
    missing using a simple test with timeit.py.

    Part of the reason, BTW, that calling d.get(key,default) is slow is that is
    also requires two dictionary lookups: one to find the get method and then
    another to access the key in the dictionary, plus it has other overheads (a
    method call) which test&get avoids.

    These figures could of course be invalidated if the actual use is too far
    from the simple string lookup I tried. For example if the key has a slow
    hash function saving the second lookup would be worthwhile.
     
    Duncan Booth, Jan 18, 2006
    #5
  6. Livin

    Ben Sizer Guest

    Duncan Booth wrote:
    > Test with the 'in' operator and then retrieving the value is the fastest
    > solution when the value isn't in the dictionary (it only does a single
    > lookup then), and is still fast when it is. [0.36/0.2]
    >
    > Using the get method of the dictionary with a default value to be retrieved
    > if the key is not present is slower than using the 'in' operator in all
    > cases (it does beat try/except when an exception is thrown) [0.49/0.54]


    Assuming those statistics are replicatable, it's quite unfortunate that
    the obvious and concise way to do things works out more slowly than the
    approach that you'd expect to take twice as long. Thankfully there
    doesn't seem to be too many of these problems in Python.

    --
    Ben Sizer
     
    Ben Sizer, Jan 18, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. darrel
    Replies:
    2
    Views:
    333
    darrel
    Aug 9, 2005
  2. lysdexia
    Replies:
    6
    Views:
    575
    John Machin
    Dec 2, 2007
  3. Ulf Meinhardt
    Replies:
    22
    Views:
    1,185
  4. sasuke
    Replies:
    5
    Views:
    2,201
    Tom Anderson
    Jul 6, 2008
  5. Brandon
    Replies:
    12
    Views:
    522
    Brandon
    Aug 15, 2008
Loading...

Share This Page