deque and thread-safety

  • Thread starter Christophe Vandeplas
  • Start date
C

Christophe Vandeplas

Hello,

I have a question about deque and thread-safety.

My application has multiple threads running concurrently and doing the
same action (downloading pages)
To know what has already been downloaded I created the variable:
seen = deque('', 1000) (keeps list of max 1000 urls in memory)

In one place of the code I do: seen.append(url)

And another place:
def seenPage()
if url in seen:
return True
return False
From the documentation I understand that deques are thread-safe:
Deques are a generalization of stacks and queues (the name is pronounced “deckâ€
and is short for “double-ended queueâ€). Deques support thread-safe, memory
efficient appends and pops from either side of the deque with approximately the
same O(1) performance in either direction.

It seems that appending to deques is indeed thread-safe, but not
iterating over them.
I've seen a similar, but different, situation here:
http://comments.gmane.org/gmane.comp.python.devel/85487

Forgetting the above url, and considering my situation this behavior
screws up the concept I need:
- Keeping a thread-safe collection of seen urls,
- Being able to check if something is in that collection
- No need to clean the collection to prevent the memory from filling up

So I know I could work around this problem by using a lock.
But then I don't only need to use the lock around the iterator, but
also around the append(), but that defeats the purpose of deque being
thread-safe.

In short, what's your advice:

1/ build a lock around the .append() and the iterator. Using the
already-existing lock in the deque. But HOW?

1/ simply build a lock around the .append() and the iterator.
Defeating the build-in thread-safety.

2/ use another collection that does what I need


Thanks for your expertise.

Christophe
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top