Correct file locking techniques

Discussion in 'Perl Misc' started by Robert TV, Jun 27, 2004.

  1. Robert TV

    Robert TV Guest

    Hi, I am asking foir advice on which form of code i've written below is the
    correct and safe way to lock a file. I've done some reading on" use Fcntl
    qw:)DEFAULT :flock);" and I understand the basics. I've also read back on
    many posts regarding file locking and there seems to be much debate. For
    this code, I simply want to increase a value.

    #!/usr/bin/perl

    use Fcntl qw:)DEFAULT :flock);
    use CGI::Carp qw(fatalsToBrowser);

    open (NUMBER, "<number.txt") or die "Can't open file: $!";
    flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
    $value = <NUMBER>;
    close(NUMBER);

    $value = $value++;

    open (NUMBER, ">number.txt") or die "Can't open file: $!";
    flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
    print NUMBER $value;
    close(NUMBER);

    The above code seems to work ok. My question is when the first user locks
    and reads the file, does the locking prevent other users from reading the
    file? The $value is then increased and printed back to the file.

    Another concern of mine is ... User#1 locks file, reads file value, then
    close and releases lock. While User#1 is writing the new value back to the
    file, another user came and read the unlocked file BEFORE the value was
    increased. So he gets the same value as User#1. User#2 then increases the
    value and writes the same data as User#1. In the end, the locking technique
    failed ... am I right?

    Another method I came up with is locking and entire read/write session
    within a what I will call "parent lock" Here is the code:

    #!/usr/bin/perl

    use Fcntl qw:)DEFAULT :flock);
    use CGI::Carp qw(fatalsToBrowser);

    open (PARENT, "<locker.txt") or die "Can't open file: $!"; #an empty file
    flock (PARENT, LOCK_EX) or die "Can't lock file: $!";

    open (NUMBER, "<number.txt") or die "Can't open file: $!";
    flock (NUMBER, LOCK_SH) or die "Can't lock file: $!";
    $value = <NUMBER>;
    close(NUMBER);

    $value = $value++;

    open (NUMBER, ">number.txt") or die "Can't open file: $!";
    flock (NUMBER, LOCK_SH) or die "Can't lock file: $!";
    print NUMBER $value;
    close(NUMBER);

    close(PARENT); #closes the parent lock

    Now User #1 locks the parent file and moves on to the read/write session.
    All other users get stuck back at the parent lock until released. I had to
    use LOCK_SH for the read/write locks for some reason ... if all locks were
    set to EX the program would stall. Is this technique better? Is it
    effective? Or perhaps they are both ineffective?

    In this next example, I don't want to increase a value, just append to the
    file on a new line. So I do this:

    #!/usr/bin/perl

    use Fcntl qw:)DEFAULT :flock);
    use CGI::Carp qw(fatalsToBrowser);

    open (DATA, ">>locker.txt") or die "Can't open file: $!";
    flock (DATA,LOCK_EX) or die "Can't lock file: $!";
    print DATA "$data\n";
    close(DATA);

    Im going to assume that this is ok.

    It's just when increasing a value that concerns me ... multipul users
    reading same value when the read lock is released. Your suggestions are
    greatly welcomes. TIA

    Robert
    Robert TV, Jun 27, 2004
    #1
    1. Advertising

  2. Robert TV

    Ben Morrow Guest

    Quoth "Robert TV" <>:
    > Hi, I am asking foir advice on which form of code i've written below is the
    > correct and safe way to lock a file. I've done some reading on" use Fcntl
    > qw:)DEFAULT :flock);" and I understand the basics. I've also read back on
    > many posts regarding file locking and there seems to be much debate. For
    > this code, I simply want to increase a value.
    >
    > #!/usr/bin/perl
    >
    > use Fcntl qw:)DEFAULT :flock);
    > use CGI::Carp qw(fatalsToBrowser);
    >
    > open (NUMBER, "<number.txt") or die "Can't open file: $!";
    > flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
    > $value = <NUMBER>;
    > close(NUMBER);
    >
    > $value = $value++;
    >
    > open (NUMBER, ">number.txt") or die "Can't open file: $!";
    > flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
    > print NUMBER $value;
    > close(NUMBER);
    >
    > The above code seems to work ok. My question is when the first user locks
    > and reads the file, does the locking prevent other users from reading the
    > file?


    No. It simply prevents other processes from *obtaining a lock* on the
    file.

    > The $value is then increased and printed back to the file.
    >
    > Another concern of mine is ... User#1 locks file, reads file value, then
    > close and releases lock. While User#1 is writing the new value back to the
    > file, another user came and read the unlocked file BEFORE the value was
    > increased. So he gets the same value as User#1. User#2 then increases the
    > value and writes the same data as User#1. In the end, the locking technique
    > failed ... am I right?


    You are. You must read/modify/write under the same, exclusive, lock;
    anyone who is just reading can use a shared lock.

    > Another method I came up with is locking and entire read/write session
    > within a what I will call "parent lock" Here is the code:


    This would be correct, but there is no need for two layers of locks. The
    inner lock is made completely redundant by the outer.

    Correct technique is:

    Processes that will only read:

    open read-only
    flock LOCK_SH
    read the data
    close

    Processes that will read and modify:

    open read-write ('+<')
    flock LOCK_EX
    read the data
    make the modifications
    seek to the start of the file (perldoc -f seek)
    truncate the file (perldoc -f truncate)
    write the new data
    close

    Alternatively, you can use a lockfile instead of proper flock locking;
    this can be simpler, but you have to be careful to clean up after
    yourself:

    sysopen a lockfile with mode O_CREAT | O_EXCL
    if the open fails, sleep for a bit and try again
    open/close/read/write the data file as you will
    unlink the lockfile

    One problem with this is you may wish to check the age of the lockfile
    and summerarily (sp?) unlink it if it appears to be too old...

    > I had to use LOCK_SH for the read/write locks for some reason ... if
    > all locks were set to EX the program would stall.


    On some systems you cannot LOCK_EX a file unless you opened it for
    writing.

    > In this next example, I don't want to increase a value, just append to the
    > file on a new line. So I do this:
    >
    > #!/usr/bin/perl
    >
    > use Fcntl qw:)DEFAULT :flock);
    > use CGI::Carp qw(fatalsToBrowser);
    >
    > open (DATA, ">>locker.txt") or die "Can't open file: $!";
    > flock (DATA,LOCK_EX) or die "Can't lock file: $!";
    > print DATA "$data\n";
    > close(DATA);
    >
    > Im going to assume that this is ok.


    Note that your readers will need to lock as well.

    Ben

    --
    All persons, living or dead, are entirely coincidental.
    Kurt Vonnegut
    Ben Morrow, Jun 27, 2004
    #2
    1. Advertising

  3. Robert TV

    Bob Walton Guest

    Robert TV wrote:

    > Hi, I am asking foir advice on which form of code i've written below is the
    > correct and safe way to lock a file. I've done some reading on" use Fcntl



    I have found that the most portable and least hassle way to lock files
    is to use a lock file. This is a file which is used exclusively for the
    purpose of establishing a lock -- it can be empty, or have any contents
    you want, but the data in the lock file would be unused and unimportant.
    This technique is particularly useful when doing stuff which is a bit
    out of the ordinary, such as locking DBM-type files to which you wish to
    tie a hash, for example.

    If you want to write a file, open the lock file for write (that will, at
    least on some systems, wipe out the contents of the lock file), then
    establish an exclusive lock on it. Then go do whatever you wanted the
    exclusive lock for (write another file, tie a hash to a DBM-type file,
    or any other operation requiring exclusive access to a resource). Then,
    when you are all done with your lock, close the lock file.

    If you want to read a file, open the lock file for reading, and
    establish a shared lock on it. Go read your data file or do whatever
    you wanted the shared lock for. When done, close the lock file.

    To the best of my knowledge, that works flawlessly on at least local
    files on every OS that supports locking (that, by the way, rules out
    Windoze 95/98/98SE). However, you should carefully and thoroughly read
    everything your OS has to say about file locking, particularly if you
    are considering locking files accessed via a network.

    You will want to note that locking a file does not prevent other
    processes from read/writing/deleting/whatevering a file. *All* it does
    is establish a lock -- the other processes have to cooperate with the
    lock in order for it to work. An exclusive lock means other processes
    seeking a lock will block until the exclusive lock is released; a shared
    lock means other processes seeking an exclusive lock will block until
    all the shared locks are released.

    I find it convenient to write a couple of subs to do the locking, like
    [untested]:

    sub lockex{
    open LOCK,">lockfile.txt"
    or die "Oops, couldn't open lockfile.txt for write, $!";
    flock(LOCK,LOCK_EX)
    or die "flock bombed out in lockex, $!";
    }
    sub locksh{
    open LOCK,"lockfile.txt"
    or die "Oops, couldn't open lockfile.txt for read, $!";
    flock(LOCK,LOCK_SH)
    or die "flock bombed out in locksh, $!";
    }
    sub unlock{
    close LOCK or die "Oops, couldn't close lockfile.txt, $!";
    }

    HTH.


    ....
    > Robert


    --
    Bob Walton
    Email: http://bwalton.com/cgi-bin/emailbob.pl
    Bob Walton, Jun 27, 2004
    #3
  4. Robert TV

    Ben Morrow Guest

    Quoth d:
    > Robert TV wrote:
    >
    > > Hi, I am asking foir advice on which form of code i've written below is the
    > > correct and safe way to lock a file. I've done some reading on" use Fcntl

    >
    > I have found that the most portable and least hassle way to lock files
    > is to use a lock file. This is a file which is used exclusively for the
    > purpose of establishing a lock -- it can be empty, or have any contents
    > you want, but the data in the lock file would be unused and unimportant.
    > This technique is particularly useful when doing stuff which is a bit
    > out of the ordinary, such as locking DBM-type files to which you wish to
    > tie a hash, for example.


    You can still perfectly well take a flock lock on the DBM file.

    > If you want to write a file, open the lock file for write (that will, at
    > least on some systems, wipe out the contents of the lock file), then
    > establish an exclusive lock on it.


    The usual reason for using a lockfile is to avoid making locking calls
    altogether... a call to sysopen with O_CREAT | O_EXCL will atomically
    test-and-create the named file, so there is no need to then lock it.

    > To the best of my knowledge, that works flawlessly on at least local
    > files on every OS that supports locking (that, by the way, rules out
    > Windoze 95/98/98SE).


    However, I think that win98 *does* support O_EXCL properly, so that
    method will work.

    > However, you should carefully and thoroughly read
    > everything your OS has to say about file locking, particularly if you
    > are considering locking files accessed via a network.


    Yes. I belive most network filesystems do it properly; the exception is
    NFSv2, and there are (at least on linux) instructions in open(2) for how
    to safely lock files over NFS.

    > I find it convenient to write a couple of subs to do the locking, like
    > [untested]:
    >
    > sub lockex{
    > open LOCK,">lockfile.txt"


    Note that if you choose to put any useful information in the lockfile
    (it is common to put the pid and possibly the hostname of the process
    which created the lock, so later processes can check if you still exist)
    you would need to sysopen it with O_RDWR | O_CREAT to ensure you didn't
    destroy info about someone else's lock.

    > or die "Oops, couldn't open lockfile.txt for write, $!";
    > flock(LOCK,LOCK_EX)
    > or die "flock bombed out in lockex, $!";
    > }
    > sub locksh{
    > open LOCK,"lockfile.txt"
    > or die "Oops, couldn't open lockfile.txt for read, $!";
    > flock(LOCK,LOCK_SH)
    > or die "flock bombed out in locksh, $!";
    > }
    > sub unlock{
    > close LOCK or die "Oops, couldn't close lockfile.txt, $!";
    > }


    Yuck! What happens if you call this twice, for different files? You'll
    implicitly close the global FH and destroy your lock. Try

    {
    my %locks;

    sub lock {
    my ($file, $type) = shift;

    open $locks{$file}, '>', "$file.lock"
    or die "can't create $file.lock: $!";

    flock $locks{$file}, $type
    or die "can't get lock on $file.lock: $!";
    }

    sub unlock {
    close $locks{$_[0]};
    }
    }

    Ben

    --
    Musica Dei donum optimi, trahit homines, trahit deos. |
    Musica truces molit animos, tristesque mentes erigit. |
    Musica vel ipsas arbores et horridas movet feras. |
    Ben Morrow, Jun 27, 2004
    #4
  5. Ben Morrow <> wrote:
    >> If you want to write a file, open the lock file for write (that will, at
    >> least on some systems, wipe out the contents of the lock file), then
    >> establish an exclusive lock on it.


    > The usual reason for using a lockfile is to avoid making locking calls
    > altogether... a call to sysopen with O_CREAT | O_EXCL will atomically
    > test-and-create the named file, so there is no need to then lock it.


    Assuming NFS isn't involved.

    I often use a separate lockfile because I don't want to modify my data
    file directly, lest a crash leave it in an inconsistent state.

    Instead I lock a separate file so that I can move a new file or files
    into place and not have to worry about my lock moving with it.

    > Yes. I belive most network filesystems do it properly; the exception is
    > NFSv2, and there are (at least on linux) instructions in open(2) for how
    > to safely lock files over NFS.


    With increased cleanup requirements also, unfortunately. I tend not to
    do it because things get so messy.

    >> sub lockex{
    >> open LOCK,">lockfile.txt"


    > Note that if you choose to put any useful information in the lockfile
    > (it is common to put the pid and possibly the hostname of the process
    > which created the lock, so later processes can check if you still exist)
    > you would need to sysopen it with O_RDWR | O_CREAT to ensure you didn't
    > destroy info about someone else's lock.


    The same as.. open (LOCK, "<+lockfile.txt").

    --
    Darren Dunham
    Senior Technical Consultant TAOS http://www.taos.com/
    Got some Dr Pepper? San Francisco, CA bay area
    < This line left intentionally blank to confuse you. >
    Darren Dunham, Jul 2, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. joon
    Replies:
    1
    Views:
    511
    Roedy Green
    Jul 8, 2003
  2. Dan

    correct or not correct?

    Dan, Oct 2, 2003, in forum: HTML
    Replies:
    7
    Views:
    433
  3. Timasmith
    Replies:
    4
    Views:
    444
    Bjorn Borud
    Nov 1, 2006
  4. Ludwigi Beethoven
    Replies:
    5
    Views:
    313
    Mike Hall
    Jul 26, 2003
  5. froil
    Replies:
    12
    Views:
    304
    Gunnar Hjalmarsson
    Mar 2, 2006
Loading...

Share This Page