cgi concurrency approaches?

[Paul Rubin]

I'm wondering if folks here have favorite lightweight ways of dealing
with concurrency in cgi's. Take a simple case:

You want to write a cgi that implements a simple counter. The first
time it's called, it prints "1". The next time, "2", etc. The naive
implementation looks something like:

fd = open("counter.num", "rw")
n = int(fd.read())
fd.seek(0, 0)
fd.write("%d"% n+1)
print "Content-type: text/html\n\n"
print n

but of course there's the obvious race condition if two people hit the
cgi at the same time.

Fancier solutions include running an transactional database in another
process and connecting to it, setting up a daemon that remembers the
counter value in memory and serializes access through a socket that
the cgi opens, using a file-system specific hack like linking to
counter file to lock it, having a timeout/retry if the counter is
locked, with a possible hangup if a lock file gets left around by
accident, etc. Each is a big pain in the neck.

Anyone have some simpler approaches?

 

[Josiah Carlson]

I'm wondering if folks here have favorite lightweight ways of dealing
with concurrency in cgi's. Take a simple case:

You want to write a cgi that implements a simple counter. The first
time it's called, it prints "1". The next time, "2", etc. The naive
implementation looks something like:

fd = open("counter.num", "rw")
n = int(fd.read())
fd.seek(0, 0)
fd.write("%d"% n+1)
print "Content-type: text/html\n\n"
print n

but of course there's the obvious race condition if two people hit the
cgi at the same time.

Fancier solutions include running an transactional database in another
process and connecting to it, setting up a daemon that remembers the
counter value in memory and serializes access through a socket that
the cgi opens, using a file-system specific hack like linking to
counter file to lock it, having a timeout/retry if the counter is
locked, with a possible hangup if a lock file gets left around by
accident, etc. Each is a big pain in the neck.

Anyone have some simpler approaches?

I'd just run Xitami (www.xitami.org) and connect a LRWP (Long-Running
Web Process). I'm not familliar with Apache, so I don't know if
mod_python is equivalent.

- Josiah

 

[Rene Pijlman]

I'm wondering if folks here have favorite lightweight ways of dealing
with concurrency in cgi's. Take a simple case:

You want to write a cgi that implements a simple counter. The first
time it's called, it prints "1". The next time, "2", etc. [...]
Anyone have some simpler approaches?

Use mod_python and keep the counter in a Python variable (if the value
needs not be persistent).
http://www.modpython.org/

 

[Matt Goodall]

I'm wondering if folks here have favorite lightweight ways of dealing
with concurrency in cgi's. Take a simple case:

You want to write a cgi that implements a simple counter. The first
time it's called, it prints "1". The next time, "2", etc. [...]
Anyone have some simpler approaches?

Use mod_python and keep the counter in a Python variable (if the value
needs not be persistent).
http://www.modpython.org/

That will only work if you use Apache 2 in threaded mode. Apache 1 and
Apache 2 in forking mode (multiple processes) will have a copy of the
counter object per-process.

Cheers, Matt

--
Matt Goodall, Pollenation Internet Ltd
w: http://www.pollenation.net
e: (e-mail address removed)

Any views expressed are my own and do not necessarily reflect the
views of my employer.

 

[Rene Pijlman]

Matt Goodall:

Rene Pijlman:

That will only work if you use Apache 2 in threaded mode.

Of course, good point.

 

[Alan Kennedy]

[replying to someone else's reply because I missed the original]

[Paul Rubin]

I'm wondering if folks here have favorite lightweight ways of dealing
with concurrency in cgi's, [snip]
but of course there's the obvious race
condition if two people hit the
cgi at the same time.

Fancier solutions include running an transactional database in another
process and connecting to it, setting up a daemon that remembers the
counter value in memory and serializes access through a socket that
the cgi opens, using a file-system specific hack like linking to
counter file to lock it, having a timeout/retry if the counter is
locked, with a possible hangup if a lock file gets left around by
accident, etc. Each is a big pain in the neck.
Anyone have some simpler approaches?

I don't know about a platform independent solution, but if you were
willing to limit yourself to certain varieties of Unix, e.g. Linux, it
would probably be relatively easy to implement a persistent counter
using something like System V message queues, or shared memory +
semaphores. This obviously will work only on systems which support
System V IPC.

Here is a brief page about System V IPC

http://www.science.unitn.it/~fiorella/guidelinux/tlk/node56.html

And here's a module, which I've never used, which purports to provide
a python interface to System V IPC facilities, if available.

http://www.theory.org/~mac4/software/

Given the above facilities, there are two ways I would approach the
problem of a persistent counter.

1. Use a dedicated Queue, and hold the counter value inside a single
message which "lives" on that queue. Those incrementing the counter
read the single message from the queue, increment it and put it back
on the queue. Any processes awaiting access to the "counter message"
would then do a blocking read on the queue. Since all puts and gets of
messages are atomic, you are guaranteed only atomic updates to your
counter. However, you could lock your system up if one of your
accessing CGI processes did not put the counter back again! You can
use timeouts as well, if my memory serves (it's been a long time since
I used System V IPC). You can also get fancy, with priorities, etc.

2. Store the value in pre-created shared memory partition, protected
by a semaphore. Since there are shared memory python modules for
several platforms, you might have a better chance with this approach
on non-unix platforms.

The Python Object Sharing (POSH) module might provide wrappers to some
useful functionality, although the module itself might be a little
heavyweight for providing a simple persistent counter.

http://poshmodule.sourceforge.net/posh/html/posh.html

HTH,