Safely add a key to a dict only if it does not already exist?

[Steven D'Aprano]

I wish to add a key to a dict only if it doesn't already exist, but do it
in a thread-safe manner.

The naive code is:

if key not in dict:
dict[key] = value


but of course there is a race condition there: it is possible that
another thread may have added the same key between the check and the
store.

How can I add a key in a thread-safe manner?

Any solutions must work with built-in dicts, I cannot use a custom dict
subclass.

 

[Chris Rebert]

I wish to add a key to a dict only if it doesn't already exist, but do it
in a thread-safe manner.

The naive code is:

if key not in dict:
dict[key] = value


but of course there is a race condition there: it is possible that

another thread may have added the same key between the check and the

store.

How can I add a key in a thread-safe manner?

I'm not entirely sure, but have you investigated dict.setdefault() ?

 

[Lie Ryan]

On Friday, January 18, 2013, Steven D'Aprano wrote:

I wish to add a key to a dict only if it doesn't already exist, but
do it
in a thread-safe manner.

The naive code is:

if key not in dict:
dict[key] = value


but of course there is a race condition there: it is possible that

another thread may have added the same key between the check and the
store.

How can I add a key in a thread-safe manner?


I'm not entirely sure, but have you investigated dict.setdefault() ?

dict.setdefault() was not atomic on older python version, they were made
atomic in Python 2.7.3 and Python 3.2.3.

See bug13521 in the issue tracker http://bugs.python.org/issue13521

PS: The bug tracker seems down at the moment, so pulled this from
Google's cache:
https://webcache.googleusercontent..../issue13521+&cd=1&hl=en&ct=clnk&client=ubuntu

 

[Vito De Tullio]

I wish to add a key to a dict only if it doesn't already exist, but do it
in a thread-safe manner.

The naive code is:

if key not in dict:
dict[key] = value


but of course there is a race condition there: it is possible that
another thread may have added the same key between the check and the
store.

How can I add a key in a thread-safe manner?

using locks?

import threading

lock = threading.Lock()
with lock:
if key not in dict:
dict[key] = value

 

[Vito De Tullio]

I'm not entirely sure, but have you investigated dict.setdefault() ?

but how setdefault makes sense in this context? It's used to set a default
value when you try to retrieve an element from the dict, not when you try to
set a new one ...

 

[Mitya Sirenef]

but how setdefault makes sense in this context? It's used to set a default
value when you try to retrieve an element from the dict, not when you try to
set a new one ...

I guess setdefault with a sentinel default value, then set to your
new value if d[k] is sentinel?

- mitya

 

[Peter Otten]

but how setdefault makes sense in this context? It's used to set a default
value when you try to retrieve an element from the dict, not when you try
to set a new one ...

But it also sets the value if the key is not found:

"""
setdefault(...)
D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D
"""
{1: 2}


It has been suggested that get_or_set() would have been a better name for
that functionality...

 

[Vito De Tullio]

But it also sets the value if the key is not found:

yeah, sure, but with a fixed value

I mean: if the value is not important, why bother at all trying not to
override it with an if or a lock or other tecniques? doing

d['key'] = 'fixed_value'

multiple times in different threads is not a problem in my eyes...

 

[Mitya Sirenef]

but how setdefault makes sense in this context? It's used to set a
default
value when you try to retrieve an element from the dict, not when you
try to
set a new one ...

I guess setdefault with a sentinel default value, then set to your
new value if d[k] is sentinel?

- mitya

Er, that makes no sense.. just setdefault to desired value. -m

 

[Vito De Tullio]

Peter Otten wrote:

uhhmm.. I think I misread the example

{1: 2}

yeah, sure it can be useful for the OP...

 

[Chris Angelico]

yeah, sure, but with a fixed value

I mean: if the value is not important, why bother at all trying not to
override it with an if or a lock or other tecniques? doing

d['key'] = 'fixed_value'

multiple times in different threads is not a problem in my eyes...

How fixed is fixed?
{'bar': 2, 'foo': 1}

If list append is guaranteed atomic, and since setdefault is
apparently atomic, then this would be a thread-safe way to maintain a
dictionary of lists:

d={}
lst=d.setdefault("foo",[])
lst.append(1)

Are those guarantees made?

ChrisA

 

[Steven D'Aprano]

I wish to add a key to a dict only if it doesn't already exist, but do
it in a thread-safe manner.

[...]
I'm not entirely sure, but have you investigated dict.setdefault() ?

Great! That's exactly what I need, thanks to everyone who responded.

 

[Roy Smith]

I wish to add a key to a dict only if it doesn't already exist, but do it
in a thread-safe manner.

The naive code is:

if key not in dict:
dict[key] = value


but of course there is a race condition there: it is possible that
another thread may have added the same key between the check and the
store.

How can I add a key in a thread-safe manner?

You want something along the lines of:

from threading import Lock
lock = Lock()
[...]
lock.acquire()
if key not in dict:
dict[key] = value
lock.release()

You probably want to wrap that up in a context manager to ensure the
lock is released if you get an exception. You don't want your entire
program to hang just because somebody handed you a key which wasn't
hashable, for example.