Simple thread-safe counter?

P

Paul Rubin

I'd like to have a function (or other callable object) that returns
0, 1, 2, etc. on repeated calls. That is:

print f() # prints 0
print f() # prints 1
print f() # prints 2
# etc.

There should never be any possibility of any number getting returned
twice, or getting skipped over, even if f is being called from
multiple threads.

What's the simplest and most natural way to do this? I can think of a
few but am not sure that they work. And I can think of some ways that
are sure to work, but are messier than I'd like.
 
T

Tim Peters

[Paul Rubin]
I'd like to have a function (or other callable object) that returns
0, 1, 2, etc. on repeated calls. That is:

print f() # prints 0
print f() # prints 1
print f() # prints 2
# etc.

There should never be any possibility of any number getting returned
twice, or getting skipped over, even if f is being called from
multiple threads.

What's the simplest and most natural way to do this? I can think of a
few but am not sure that they work. And I can think of some ways that
are sure to work, but are messier than I'd like.

The GIL is your friend here:

import itertools
f = itertools.count().next

A similar thing can be done with xrange. But either way sucks if you
call it often enough to exceed the size of a Python short int
(platform C long). The obvious way with an explicit mutex doesn't
have that problem.
 
S

Skip Montanaro

Paul> I'd like to have a function (or other callable object) that
Paul> returns 0, 1, 2, etc. on repeated calls.
...
Paul> There should never be any possibility of any number getting
Paul> returned twice, or getting skipped over, even if f is being called
Paul> from multiple threads.

How about (untested):

import Queue

counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get()
counter.put(i+1)
return i

Obviously, if you want multiple counters for some reason a little
information hiding with a class would help (also untested):

import Queue

class Counter:
def __init__(self, start=0):
self.counter = Queue.Queue()
self.counter.put(start)

def __call__(self):
i = self.counter.get()
self.counter.put(i+1)
return i

Skip
 
P

Paul Rubin

Tim Peters said:
The GIL is your friend here:

import itertools
f = itertools.count().next

Thanks, I was hoping something like this would work but was not sure I
could rely on it.
A similar thing can be done with xrange. But either way sucks if you
call it often enough to exceed the size of a Python short int
(platform C long). The obvious way with an explicit mutex doesn't
have that problem.

Xrange, of course :). I don't need to exceed the size of a short int,
so either of these should work fine. I wonder what measures the Pypy
implementers will take (if any) to make sure these things keep
working, but for now I won't worry about it.

Out of interest, are the above guaranteed to work under Jython? What
I'm doing right now is a short-term thing that will only have to run
under CPython, but I like to do things the right way when I can.
 
A

Artie Gold

Skip said:
Paul> I'd like to have a function (or other callable object) that
Paul> returns 0, 1, 2, etc. on repeated calls.
...
Paul> There should never be any possibility of any number getting
Paul> returned twice, or getting skipped over, even if f is being called
Paul> from multiple threads.

How about (untested):

import Queue

counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get()
I think you need:
i = counter.get(True)
for this to work; otherwise a race condition would raise an exception.
counter.put(i+1)
return i

[snip]

This is, of course dependent upon counter.get() being guaranteed to be
thread safe. (I haven't found anything in the docs specifically relating
to that. Perhaps it's implicit?)

Thanks,
--ag
 
P

Paul Rubin

Skip Montanaro said:
How about (untested):

import Queue
counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get()
counter.put(i+1)
return i

Hmmm, that's a bit messier than I hoped for, but it looks sure to work.

I think for my immediate requirement, I'm going to use xrange as Tim
suggested. However, I can't be sure that will work in non-GIL
implementations.
 
R

Rob Williscroft

Tim Peters wrote in (e-mail address removed) in comp.lang.python:
[Paul Rubin]
I'd like to have a function (or other callable object) that returns
0, 1, 2, etc. on repeated calls. That is:

print f() # prints 0
print f() # prints 1
print f() # prints 2
# etc.

There should never be any possibility of any number getting returned
twice, or getting skipped over, even if f is being called from
multiple threads.

What's the simplest and most natural way to do this? I can think of a
few but am not sure that they work. And I can think of some ways that
are sure to work, but are messier than I'd like.

The GIL is your friend here:

Yes but IIUC the GIL is only CPython, will this work with Jython,
IornPython etc ? (*)
import itertools
f = itertools.count().next

*) I'm either being rhetorical or stupid, not sure which :).

Rob.
 
N

Neil Benn

Paul said:
Hmmm, that's a bit messier than I hoped for, but it looks sure to work.

I think for my immediate requirement, I'm going to use xrange as Tim
suggested. However, I can't be sure that will work in non-GIL
implementations.
Hello,

Simple xrange stuff won't work in Jython as Sython is running
on a JVM which doesn't have a GIL kinda thing. If you wanna return a
sequential number then you'll need something like :

<PSEUDO CODE>

class COUNTER:
DOC : THIS WILL RETURN AN INCRMENTING LIST OF NUMBERS - THREAD SAFE.

FUNCTION INIT:
OBJRLOCK= RLOCK()
INTRETURNNUM = 0

FUNCTION NEXT
TRY{
RLOCK.AQUIRE
}FINALLY{
RLOCK.RELEASE
}

</PSUEDO CODE>

That basically all you'll need, if you make it iteratable of
whatever - you need to wrap the business end of what you are doing
around a recursive lock. Personally I dislike the GIL so I avoid
writing code that takes advantages of it.

Why psuedo code - this is similar to python code I know but it means
I'm not posting untested python code!!

Cheers,

Neil
 
A

Artie Gold

Leif said:
The default value for the "block" argument to Queue.get is True.

Right. I misparsed the entry in the documentation:

"If optional args block is true and timeout is None (the default), block
if necessary..."

Thanks,
--ag
 
A

Aahz

Obviously, if you want multiple counters for some reason a little
information hiding with a class would help (also untested):

import Queue

class Counter:
def __init__(self, start=0):
self.counter = Queue.Queue()
self.counter.put(start)

def __call__(self):
i = self.counter.get()
self.counter.put(i+1)
return i

This is one case where'd recommend using a plan RLock() instead of using
Queue -- the RLock() will be more efficient:

import threading

class Counter:
def __init__(self, start=0, increment=1):
self.counter = start
self.increment = increment
self.lock = threading.RLock()
def __call__(self):
self.lock.acquire()
self.counter += self.increment
i = self.counter
self.lock.release()
return i

There are several tricks one can use to squeeze further efficiency gains
from this, including using Lock() instead of RLock().
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
 
P

Paul Rubin

This is one case where'd recommend using a plan RLock() instead of using
Queue -- the RLock() will be more efficient...

I'm starting to believe the GIL covers up an awful lot of sloppiness
in Python. I wonder if there could be a decorator approach:

@synchronized
def counter():
t = itertools.count()
while True:
yield t.next()

or better yet,

counter = synchronized(itertools.count())
 
T

Tim Peters

[Paul Rubin]
I'm starting to believe the GIL covers up an awful lot of sloppiness
in Python.

The GIL is highly exploitable, and much of CPython does exploit it.

If you don't want to exploit it, that's fine: there was always an
obvious approach using an explicit mutex here, and the only thing
stopping you from using it is a desire to be clever. Exploiting the
GIL in CPython is clever; using an explicit mutex is utterly
straightforward. Pick your poison.
 
H

Heiko Wundram

Am Samstag, 2. April 2005 22:28 schrieb Paul Rubin:
I'm starting to believe the GIL covers up an awful lot of sloppiness
in Python. I wonder if there could be a decorator approach:

@synchronized
def counter():
t = itertools.count()
while True:
yield t.next()

Of course there could:

def synchronized_iterator(f):
def wrapper(*args,**kwargs):
class iterator(object):
def __init__(self,f,args,kwargs):
self.iter = f(*args,**kwargs)
self.lock = threading.RLock()
def __iter__(self):
return self
def next(self):
self.lock.acquire()
try:
return self.iter.next()
finally:
self.lock.release()
return iterator(f,args,kwargs)
return wrapper

@synchronized_iterator
def create_counter():
t = itertools.count()
while True:
yield t.next()

or

counter = synchronized_iterator(itertools.count)

I used a class-based approach, as I don't want to destroy the semantics of
calling the returned wrapper(), which should've already instantiated the
wrapped generator object (which doesn't happen when making wrapper() a
generator itself).

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCTyN0f0bpgh6uVAMRAhl4AJ9eityDJSuzkfywFGC0DGUOiz4hXACePWFP
fi1F0tFXekyTzWbvbHi8zh8=
=flQB
-----END PGP SIGNATURE-----
 
H

Heiko Wundram

Am Sonntag, 3. April 2005 00:57 schrieb Heiko Wundram:
<snip>
or

Make that:

create_counter = syncronized_iterator(itertools.count)

and

counter = create_counter()

to create the actual counter regardless of iterator.

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCTyQNf0bpgh6uVAMRAqdJAJ9Efwzkf9xNau6LaRgcOTKwOpz95ACfe1Y7
iHWep1RM157Dd/pvxlhtYEg=
=zRYe
-----END PGP SIGNATURE-----
 
P

Paul Rubin

Tim Peters said:
If you don't want to exploit it, that's fine: there was always an
obvious approach using an explicit mutex here, and the only thing
stopping you from using it is a desire to be clever. Exploiting the
GIL in CPython is clever; using an explicit mutex is utterly
straightforward. Pick your poison.

For my immediate requirement I'm going to use xrange as you suggested
(thanks!). It will work exactly as I want. I'd be uncomfortable
publishing such a program for long-term use by other people, but this
is just for a short-term hack (famous last words).

I think using an explicit mutex goes against Guido's quote mentioned
yesterday, about avoiding tedious code. But relying on the GIL
results in a brittle program that only works correctly in one Python
implementation. As use of other implementations becomes more
widespread, one of three things has to happen: 1) Python programs will
become uglier and more mistake-prone (from using explicit mutexes all
over the place); 2) Python programs will become unreliable (from
depending on a GIL that doesn't exist any more); or 3) Python will
acquire some new mechanisms for handling reliable synchronization
easily and cleanly, as other languages already have.

I think (3) is the preferable choice and it's reasonable to try to
figure out how such mechanisms should work.
 
M

Michael Hudson

Paul Rubin said:
Thanks, I was hoping something like this would work but was not sure I
could rely on it.


Xrange, of course :). I don't need to exceed the size of a short int,
so either of these should work fine. I wonder what measures the Pypy
implementers will take (if any) to make sure these things keep
working, but for now I won't worry about it.

Well, for now we don't support threads. Easy!

Cheers,
mwh
(no, really, this is for the future)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top