How to lock files (the easiest/best way)?

Discussion in 'Python' started by =?iso-8859-1?q?Elmo_M=E4ntynen?=, Jul 15, 2006.

  1. Is there something better than using fnctl? It seems a bit intimidating
    with a quick look.
     
    =?iso-8859-1?q?Elmo_M=E4ntynen?=, Jul 15, 2006
    #1
    1. Advertising

  2. Elmo Mäntynen enlightened us with:
    > Is there something better than using fnctl? It seems a bit
    > intimidating with a quick look.


    Locking files is a complex business. What do you want to lock? Why?
    Lock it with respect to what? It's easier to lock a file for local
    use, compared to when the file also has to be locked from, say, use
    via the network.

    Sybren
    --
    The problem with the world is stupidity. Not saying there should be a
    capital punishment for stupidity, but why don't we just take the
    safety labels off of everything and let the problem solve itself?
    Frank Zappa
     
    Sybren Stuvel, Jul 15, 2006
    #2
    1. Advertising

  3. On Sat, 15 Jul 2006 23:28:21 +0200, Sybren Stuvel wrote:

    > Elmo Mäntynen enlightened us with:
    >> Is there something better than using fnctl? It seems a bit
    >> intimidating with a quick look.

    >
    > Locking files is a complex business. What do you want to lock? Why?
    > Lock it with respect to what? It's easier to lock a file for local
    > use, compared to when the file also has to be locked from, say, use
    > via the network.
    >
    > Sybren


    Only locally. I want to be able to read/write to a single file from
    multiple possibly parallel processes. Would 'touch lock' (or something
    like that) work reliably (this just occured to me)?
     
    =?iso-8859-1?q?Elmo_M=E4ntynen?=, Jul 15, 2006
    #3
  4. Elmo Mäntynen enlightened us with:
    > Only locally. I want to be able to read/write to a single file from
    > multiple possibly parallel processes. Would 'touch lock' (or
    > something like that) work reliably (this just occured to me)?


    I use a lock directory for that, os.mkdir('/var/lock/somedir').
    If you use a file, you need two steps:
    1) Check whether the lock-file exists
    2) Create the lock-file

    This is not atomic. With a directory, creating it will fail if it
    already exists. This means you can atomically check for the lock, and
    if it doesn't exist already, you've immediately created it too.

    Sybren
    --
    The problem with the world is stupidity. Not saying there should be a
    capital punishment for stupidity, but why don't we just take the
    safety labels off of everything and let the problem solve itself?
    Frank Zappa
     
    Sybren Stuvel, Jul 15, 2006
    #4
  5. On Sat, 15 Jul 2006 23:52:10 +0200, Sybren Stuvel wrote:

    > Elmo Mäntynen enlightened us with:
    >> Only locally. I want to be able to read/write to a single file from
    >> multiple possibly parallel processes. Would 'touch lock' (or
    >> something like that) work reliably (this just occured to me)?

    >
    > I use a lock directory for that, os.mkdir('/var/lock/somedir').
    > If you use a file, you need two steps:
    > 1) Check whether the lock-file exists
    > 2) Create the lock-file
    >
    > This is not atomic. With a directory, creating it will fail if it
    > already exists. This means you can atomically check for the lock, and
    > if it doesn't exist already, you've immediately created it too.
    >
    > Sybren


    Thanks. Is that what atomic basically means?
     
    =?iso-8859-1?q?Elmo_M=E4ntynen?=, Jul 15, 2006
    #5
  6. Elmo Mäntynen enlightened us with:
    > Thanks. Is that what atomic basically means?


    "Atomic" literally means that it can't be broken down into smaller
    bits. In a non-atomic case, this could happen:

    Process A: Check whether lock exists, which it doesn't.
    Process B: Check whether lock exists, which it doesn't.
    Process A: Touch lock-file.
    Process B: Touch lock-file.

    Now both A and B think they've locked the file, and they'll both
    access the protected resource.

    With an atomic locking method, the following happens:

    Process A: Check whether lock exists, which it doesn't. The lock is
    created.
    Process B: The lock is in place, so it refuses to access the protected
    resource.

    Which is what you want.

    Sybren
    --
    The problem with the world is stupidity. Not saying there should be a
    capital punishment for stupidity, but why don't we just take the
    safety labels off of everything and let the problem solve itself?
    Frank Zappa
     
    Sybren Stuvel, Jul 15, 2006
    #6
  7. On 2006-07-15 18:52:10, Sybren Stuvel wrote:

    > Elmo Mäntynen enlightened us with:
    >> Only locally. I want to be able to read/write to a single file from
    >> multiple possibly parallel processes. Would 'touch lock' (or
    >> something like that) work reliably (this just occured to me)?

    >
    > I use a lock directory for that, os.mkdir('/var/lock/somedir').
    > If you use a file, you need two steps:
    > 1) Check whether the lock-file exists
    > 2) Create the lock-file


    cvsnt for example used to use lock files. Now it uses a lock server: a
    server app that just sits there and allows different processes to acquire,
    check for and release locks on files. More implementation work, probably,
    but more efficient.

    Gerhard
     
    Gerhard Fiedler, Jul 15, 2006
    #7
  8. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Paul Rubin Guest

    Sybren Stuvel <> writes:
    > I use a lock directory for that, os.mkdir('/var/lock/somedir').
    > If you use a file, you need two steps:
    > 1) Check whether the lock-file exists
    > 2) Create the lock-file
    > This is not atomic. With a directory, creating it will fail if it
    > already exists. This means you can atomically check for the lock, and
    > if it doesn't exist already, you've immediately created it too.


    The classic way in Unix was to make a link:

    1) link('some_other_file', 'lockfile')

    This was atomic and would fail if lockfile already existed.

    I'm not sure whether it can still be reliable, in current environments
    that can involve remote file systems etc. But I'm not sure if the
    mkdir approach works either.

    Maybe you want to use something like shm or posh?
     
    Paul Rubin, Jul 16, 2006
    #8
  9. Elmo Mäntynen wrote:
    > Is there something better than using fnctl? It seems a bit intimidating
    > with a quick look.


    try the portlocker wrapper from the active state cookbook. I have a
    version which has been slightly updated for more modern pythons. I
    really need to make my darcs repository visible so you could get it.

    Alternatively, you can try the makedir trick. Creating directories are
    atomic operations if it exists, you will get error message. If it
    doesn't, you will get a directory and thereby capture the lock. You
    delete the directory when you release the lock.

    crude but effective. I used it in building a file based queue system.
    You know, I really should learn how to build eggs because I have a whole
    bunch of little pieces of software that would probably be useful to others.

    ---eric
     
    Eric S. Johansson, Jul 16, 2006
    #9
  10. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Jim Segrave Guest

    In article <>,
    Eric S. Johansson <> wrote:
    >Elmo Mäntynen wrote:
    >> Is there something better than using fnctl? It seems a bit intimidating
    >> with a quick look.

    >
    >try the portlocker wrapper from the active state cookbook. I have a
    >version which has been slightly updated for more modern pythons. I
    >really need to make my darcs repository visible so you could get it.
    >
    >Alternatively, you can try the makedir trick. Creating directories are
    >atomic operations if it exists, you will get error message. If it
    >doesn't, you will get a directory and thereby capture the lock. You
    >delete the directory when you release the lock.
    >
    >crude but effective. I used it in building a file based queue system.
    >You know, I really should learn how to build eggs because I have a whole
    >bunch of little pieces of software that would probably be useful to others.


    Here's a more complete file locking scheme which uses a lockfile with
    the O_CREAT and O_EXCL flags to make the creation atomic. If it gets
    the lock, it writes its PID in readable form in the file. It also does
    two other things:

    If you know that no process should lock the file for more than a fixed
    period of time, it will retry once/second as long as the lock file is
    not older than that fixed period of time. If it is older, it will
    report the problem, including the PID of the locking process and exit.

    This caters for a process which has set the lock file and then
    terminates without removing it (which can happen do to an application
    or server crash).

    ------------------------------------------

    import os
    import errno
    import sys
    import time
    import stat

    # the maximum reasonable time for aprocesstobe
    max_wait = 10

    lockfile = "/tmp/mylock"

    while True:
    try:
    fd = os.open(lockfile, os.O_EXCL | os.O_RDWR | os.O_CREAT)
    # we created the lockfile, so we're the owner
    break
    except OSError, e:
    if e.errno != errno.EEXIST:
    # should not occur
    raise

    try:
    # the lock file exists, try to stat it to get its age
    # and read it's contents to report the owner PID
    f = open(lockfile, "r")
    s = os.stat(lockfile)
    except OSError, e:
    if e.errno != errno.EEXIST:
    sys.exit("%s exists but stat() failed: %s" %
    (lockfile, e.strerror))
    # we didn't create the lockfile, so it did exist, but it's
    # gone now. Just try again
    continue

    # we didn't create the lockfile and it's still there, check
    # its age
    now = int(time.time())
    if now - s[stat.ST_MTIME] > max_wait:
    pid = f.readline()
    sys.exit("%s has been locked for more than " \
    "%d seconds (PID %s)" % (lockfile, max_wait,
    pid))

    # it's not been locked too long, wait a while and retry
    f.close()
    time.sleep(1)

    # if we get here. we have the lockfile. Convert the os.open file
    # descriptor into a Python file object and record our PID in it

    f = os.fdopen(fd, "w")
    f.write("%d\n" % os.getpid())
    f.close()




    --
    Jim Segrave ()
     
    Jim Segrave, Jul 16, 2006
    #10
  11. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Jim Segrave Guest

    In article <>,
    Jim Segrave <> wrote:
    > except OSError, e:
    > if e.errno != errno.EEXIST:


    this should read:
    if e.errno != errno.ENOENT:

    (it was left with EEXIST from testing this code by forcing an error,
    as the code for this failure requires a very tight race condition to test)

    > sys.exit("%s exists but stat() failed: %s" %
    > (lockfile, e.strerror))
    > # we didn't create the lockfile, so it did exist, but it's



    --
    Jim Segrave ()
     
    Jim Segrave, Jul 16, 2006
    #11
  12. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Donn Cave Guest

    In article <>,
    Elmo Mäntynen <> wrote:

    > On Sat, 15 Jul 2006 23:52:10 +0200, Sybren Stuvel wrote:
    >
    > > Elmo Mäntynen enlightened us with:
    > >> Only locally. I want to be able to read/write to a single file from
    > >> multiple possibly parallel processes. Would 'touch lock' (or
    > >> something like that) work reliably (this just occured to me)?

    > >
    > > I use a lock directory for that, os.mkdir('/var/lock/somedir').
    > > If you use a file, you need two steps:
    > > 1) Check whether the lock-file exists
    > > 2) Create the lock-file
    > >
    > > This is not atomic. With a directory, creating it will fail if it
    > > already exists. This means you can atomically check for the lock, and
    > > if it doesn't exist already, you've immediately created it too.
    > >
    > > Sybren

    >
    > Thanks. Is that what atomic basically means?


    Yes, and also "race condition". That's why Jim Segrave's
    example code uses O_EXCL with open(2).

    Donn Cave,
     
    Donn Cave, Jul 17, 2006
    #12
  13. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Fuzzyman Guest

    Sybren Stuvel wrote:
    > Elmo Mäntynen enlightened us with:
    > > Only locally. I want to be able to read/write to a single file from
    > > multiple possibly parallel processes. Would 'touch lock' (or
    > > something like that) work reliably (this just occured to me)?

    >
    > I use a lock directory for that, os.mkdir('/var/lock/somedir').
    > If you use a file, you need two steps:
    > 1) Check whether the lock-file exists
    > 2) Create the lock-file
    >
    > This is not atomic. With a directory, creating it will fail if it
    > already exists. This means you can atomically check for the lock, and
    > if it doesn't exist already, you've immediately created it too.


    `Pathutils <http://www.voidspace.org.uk/python/pathutils.html>`_ has a
    simple interface to a system like this.

    Apparently not all platforms guarantee that attempts to create a
    directory will fail if the directory already exists - so pathutils does
    *slightly* more, but it's the same idea.

    All the best,

    Fuzzyman
    http://www.voidspace.org.uk/python/index.shtml


    >
    > Sybren
    > --
    > The problem with the world is stupidity. Not saying there should be a
    > capital punishment for stupidity, but why don't we just take the
    > safety labels off of everything and let the problem solve itself?
    > Frank Zappa
     
    Fuzzyman, Jul 17, 2006
    #13
  14. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Guest

    Elmo Mäntynen wrote:
    > Is there something better than using fnctl? It seems a bit intimidating
    > with a quick look.


    Although fcntl is pretty opaque, it's quite easy to use if all you want
    is a simple exclusive lock for all your data. The thing to keep in
    mind is, you don't have to lock every file you want exclusive access
    to. In fact, doing that provides no added security (since it's an
    advisory lock and programs are free to ignore it). For simple cases,
    locking one file and only accessing your data if you have that lock
    works, and is very simple.

    What I usually do is touch an empty file (say, "lockfile") in my data
    directory. I don't access any files in that directory unless I have a
    lock to the lockfile. This is done simply with (untested):

    import fcntl

    f = open("/path/to/data/directory/lockfile","r")
    try:
    fcntl.flock(f.fileno(),fcntl.LOCK_EX)
    ...access data freely here...
    finally:
    f.close()

    Closing the file should release the lock (unless you have a truly
    horrible operating system).


    Carl Banks
     
    , Jul 17, 2006
    #14
  15. wrote:
    >
    >> ith a quick look.
    >>

    >
    >
    > f = open("/path/to/data/directory/lockfile","r")
    > try:
    > fcntl.flock(f.fileno(),fcntl.LOCK_EX)
    > ...access data freely here...
    > finally:
    > f.close()
    >
    > Closing the file should release the lock (unless you have a truly
    > horrible operating system).
    >
    >

    I also find that fcntl has problems with NFS (or at least, *I* had
    problems using the python fcntl module and nfs - could be that horrible
    operating system, but doing things like that over nfs can be tricky).


    -carl

    --

    Carl J. Van Arsdall

    Build and Release
    MontaVista Software
     
    Carl J. Van Arsdall, Jul 17, 2006
    #15
  16. =?iso-8859-1?q?Elmo_M=E4ntynen?=

    Carl Banks Guest

    Carl J. Van Arsdall wrote:
    > wrote:
    > >
    > > f = open("/path/to/data/directory/lockfile","r")
    > > try:
    > > fcntl.flock(f.fileno(),fcntl.LOCK_EX)
    > > ...access data freely here...
    > > finally:
    > > f.close()
    > >
    > > Closing the file should release the lock (unless you have a truly
    > > horrible operating system).
    > >
    > >

    > I also find that fcntl has problems with NFS (or at least, *I* had
    > problems using the python fcntl module and nfs - could be that horrible
    > operating system, but doing things like that over nfs can be tricky).


    Ah, that's a tough one.

    Apparently, you could lock files on NFS if both the client OS and NFS
    server are up to the task (i.e., sufficiently recent), but good luck
    getting that to fly. And, with NFS, even some of the seemingly
    foolproof methods like "lock directories" aren't necessarily going to
    work. Cross your fingers and hope you have a solid NFS server and
    well-behaved clients.


    Carl Banks
     
    Carl Banks, Jul 18, 2006
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?cm9kY2hhcg==?=

    easiest or best way

    =?Utf-8?B?cm9kY2hhcg==?=, Mar 24, 2005, in forum: ASP .Net
    Replies:
    2
    Views:
    429
    =?Utf-8?B?cm9kY2hhcg==?=
    Mar 25, 2005
  2. Fuzzyman
    Replies:
    3
    Views:
    531
    Andrew MacIntyre
    Dec 5, 2003
  3. Robert Brewer
    Replies:
    0
    Views:
    516
    Robert Brewer
    Dec 5, 2003
  4. k3xji
    Replies:
    7
    Views:
    875
    Gabriel Genellina
    Dec 30, 2008
  5. Petyr David
    Replies:
    17
    Views:
    216
    Petyr David
    Dec 7, 2006
Loading...

Share This Page