Safely add a key to a dict only if it does not already exist?

Discussion in 'Python' started by Steven D'Aprano, Jan 19, 2013.

  1. I wish to add a key to a dict only if it doesn't already exist, but do it
    in a thread-safe manner.

    The naive code is:

    if key not in dict:
    dict[key] = value


    but of course there is a race condition there: it is possible that
    another thread may have added the same key between the check and the
    store.

    How can I add a key in a thread-safe manner?

    Any solutions must work with built-in dicts, I cannot use a custom dict
    subclass.


    --
    Steven
    Steven D'Aprano, Jan 19, 2013
    #1
    1. Advertising

  2. Steven D'Aprano

    Chris Rebert Guest

    On Friday, January 18, 2013, Steven D'Aprano wrote:

    > I wish to add a key to a dict only if it doesn't already exist, but do it
    > in a thread-safe manner.
    >
    > The naive code is:
    >
    > if key not in dict:
    > dict[key] = value
    >
    >
    > but of course there is a race condition there: it is possible that


    another thread may have added the same key between the check and the
    > store.
    >
    > How can I add a key in a thread-safe manner?
    >


    I'm not entirely sure, but have you investigated dict.setdefault() ?


    --
    Cheers,
    Chris
    --
    http://rebertia.com
    Chris Rebert, Jan 19, 2013
    #2
    1. Advertising

  3. Steven D'Aprano

    Lie Ryan Guest

    On 19/01/13 15:15, Chris Rebert wrote:
    > On Friday, January 18, 2013, Steven D'Aprano wrote:
    >
    > I wish to add a key to a dict only if it doesn't already exist, but
    > do it
    > in a thread-safe manner.
    >
    > The naive code is:
    >
    > if key not in dict:
    > dict[key] = value
    >
    >
    > but of course there is a race condition there: it is possible that
    >
    > another thread may have added the same key between the check and the
    > store.
    >
    > How can I add a key in a thread-safe manner?
    >
    >
    > I'm not entirely sure, but have you investigated dict.setdefault() ?


    dict.setdefault() was not atomic on older python version, they were made
    atomic in Python 2.7.3 and Python 3.2.3.

    See bug13521 in the issue tracker http://bugs.python.org/issue13521

    PS: The bug tracker seems down at the moment, so pulled this from
    Google's cache:
    https://webcache.googleusercontent..../issue13521 &cd=1&hl=en&ct=clnk&client=ubuntu
    Lie Ryan, Jan 19, 2013
    #3
  4. Steven D'Aprano wrote:

    > I wish to add a key to a dict only if it doesn't already exist, but do it
    > in a thread-safe manner.
    >
    > The naive code is:
    >
    > if key not in dict:
    > dict[key] = value
    >
    >
    > but of course there is a race condition there: it is possible that
    > another thread may have added the same key between the check and the
    > store.
    >
    > How can I add a key in a thread-safe manner?


    using locks?

    import threading

    lock = threading.Lock()
    with lock:
    if key not in dict:
    dict[key] = value


    --
    ZeD
    Vito De Tullio, Jan 19, 2013
    #4
  5. Chris Rebert wrote:

    >> How can I add a key in a thread-safe manner?

    > I'm not entirely sure, but have you investigated dict.setdefault() ?


    but how setdefault makes sense in this context? It's used to set a default
    value when you try to retrieve an element from the dict, not when you try to
    set a new one ...

    --
    ZeD
    Vito De Tullio, Jan 19, 2013
    #5
  6. On 01/19/2013 02:27 AM, Vito De Tullio wrote:
    > Chris Rebert wrote:
    >
    >>> How can I add a key in a thread-safe manner?

    >> I'm not entirely sure, but have you investigated dict.setdefault() ?

    > but how setdefault makes sense in this context? It's used to set a default
    > value when you try to retrieve an element from the dict, not when you try to
    > set a new one ...
    >


    I guess setdefault with a sentinel default value, then set to your
    new value if d[k] is sentinel?

    - mitya


    --
    Lark's Tongue Guide to Python: http://lightbird.net/larks/
    Mitya Sirenef, Jan 19, 2013
    #6
  7. Steven D'Aprano

    Peter Otten Guest

    Vito De Tullio wrote:

    > Chris Rebert wrote:
    >
    >>> How can I add a key in a thread-safe manner?

    >> I'm not entirely sure, but have you investigated dict.setdefault() ?

    >
    > but how setdefault makes sense in this context? It's used to set a default
    > value when you try to retrieve an element from the dict, not when you try
    > to set a new one ...


    But it also sets the value if the key is not found:

    """
    setdefault(...)
    D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D
    """

    >>> d = {}
    >>> d.setdefault(1, 2)

    2
    >>> d

    {1: 2}
    >>> d.setdefault(1, 3)

    2
    >>> d

    {1: 2}


    It has been suggested that get_or_set() would have been a better name for
    that functionality...
    Peter Otten, Jan 19, 2013
    #7
  8. Peter Otten wrote:

    >>>> How can I add a key in a thread-safe manner?
    >>> I'm not entirely sure, but have you investigated dict.setdefault() ?

    >>
    >> but how setdefault makes sense in this context? It's used to set a
    >> default value when you try to retrieve an element from the dict, not when
    >> you try to set a new one ...

    >
    > But it also sets the value if the key is not found:


    yeah, sure, but with a fixed value :)

    I mean: if the value is not important, why bother at all trying not to
    override it with an if or a lock or other tecniques? doing

    d['key'] = 'fixed_value'

    multiple times in different threads is not a problem in my eyes...

    --
    ZeD
    Vito De Tullio, Jan 19, 2013
    #8
  9. On 01/19/2013 02:35 AM, Mitya Sirenef wrote:
    > On 01/19/2013 02:27 AM, Vito De Tullio wrote:
    >> Chris Rebert wrote:
    >>
    >>>> How can I add a key in a thread-safe manner?
    >>> I'm not entirely sure, but have you investigated dict.setdefault() ?

    >> but how setdefault makes sense in this context? It's used to set a
    >> default
    >> value when you try to retrieve an element from the dict, not when you
    >> try to
    >> set a new one ...
    >>

    >
    > I guess setdefault with a sentinel default value, then set to your
    > new value if d[k] is sentinel?
    >
    > - mitya
    >
    >


    Er, that makes no sense.. just setdefault to desired value. -m



    --
    Lark's Tongue Guide to Python: http://lightbird.net/larks/
    Mitya Sirenef, Jan 19, 2013
    #9
  10. Peter Otten wrote:

    uhhmm.. I think I misread the example

    >>>> d = {}
    >>>> d.setdefault(1, 2)

    > 2
    >>>> d

    > {1: 2}
    >>>> d.setdefault(1, 3)

    > 2
    >>>> d

    > {1: 2}


    yeah, sure it can be useful for the OP...

    --
    ZeD
    Vito De Tullio, Jan 19, 2013
    #10
  11. On Sat, Jan 19, 2013 at 7:16 PM, Vito De Tullio <> wrote:
    > yeah, sure, but with a fixed value :)
    >
    > I mean: if the value is not important, why bother at all trying not to
    > override it with an if or a lock or other tecniques? doing
    >
    > d['key'] = 'fixed_value'
    >
    > multiple times in different threads is not a problem in my eyes...


    How fixed is fixed?

    >>> d={}
    >>> d.setdefault("foo",1)

    1
    >>> d.setdefault("bar",2)

    2
    >>> d

    {'bar': 2, 'foo': 1}

    If list append is guaranteed atomic, and since setdefault is
    apparently atomic, then this would be a thread-safe way to maintain a
    dictionary of lists:

    >>> d={}
    >>> lst=d.setdefault("foo",[])
    >>> lst.append(1)


    Are those guarantees made?

    ChrisA
    Chris Angelico, Jan 19, 2013
    #11
  12. On Fri, 18 Jan 2013 20:15:26 -0800, Chris Rebert wrote:

    > On Friday, January 18, 2013, Steven D'Aprano wrote:
    >
    >> I wish to add a key to a dict only if it doesn't already exist, but do
    >> it in a thread-safe manner.

    [...]
    > I'm not entirely sure, but have you investigated dict.setdefault() ?


    Great! That's exactly what I need, thanks to everyone who responded.



    --
    Steven
    Steven D'Aprano, Jan 19, 2013
    #12
  13. Steven D'Aprano

    Roy Smith Guest

    In article <50fa1bf1$0$30003$c3e8da3$>,
    Steven D'Aprano <> wrote:

    > I wish to add a key to a dict only if it doesn't already exist, but do it
    > in a thread-safe manner.
    >
    > The naive code is:
    >
    > if key not in dict:
    > dict[key] = value
    >
    >
    > but of course there is a race condition there: it is possible that
    > another thread may have added the same key between the check and the
    > store.
    >
    > How can I add a key in a thread-safe manner?


    You want something along the lines of:

    from threading import Lock
    lock = Lock()
    [...]
    lock.acquire()
    if key not in dict:
    dict[key] = value
    lock.release()

    You probably want to wrap that up in a context manager to ensure the
    lock is released if you get an exception. You don't want your entire
    program to hang just because somebody handed you a key which wasn't
    hashable, for example.
    Roy Smith, Jan 19, 2013
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. LT
    Replies:
    7
    Views:
    2,078
    Phlip
    Jul 25, 2004
  2. David Zaret

    how to safely extract dict values

    David Zaret, Jul 31, 2006, in forum: Python
    Replies:
    10
    Views:
    557
    Amit Khemka
    Jul 31, 2006
  3. Noah Roberts

    Does this already exist?

    Noah Roberts, Nov 20, 2006, in forum: C++
    Replies:
    2
    Views:
    286
    Noah Roberts
    Nov 20, 2006
  4. Albert van der Horst

    dict's as dict's key.

    Albert van der Horst, Jan 13, 2010, in forum: Python
    Replies:
    5
    Views:
    247
    Lie Ryan
    Jan 17, 2010
  5. Christoph Groth

    dict: retrieve the original key by key

    Christoph Groth, May 15, 2011, in forum: Python
    Replies:
    0
    Views:
    184
    Christoph Groth
    May 15, 2011
Loading...

Share This Page