Correct file locking techniques

[Robert TV]

Hi, I am asking foir advice on which form of code i've written below is the
correct and safe way to lock a file. I've done some reading on" use Fcntl
qwDEFAULT :flock);" and I understand the basics. I've also read back on
many posts regarding file locking and there seems to be much debate. For
this code, I simply want to increase a value.

#!/usr/bin/perl

use Fcntl qwDEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);

open (NUMBER, "<number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
$value = <NUMBER>;
close(NUMBER);

$value = $value++;

open (NUMBER, ">number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
print NUMBER $value;
close(NUMBER);

The above code seems to work ok. My question is when the first user locks
and reads the file, does the locking prevent other users from reading the
file? The $value is then increased and printed back to the file.

Another concern of mine is ... User#1 locks file, reads file value, then
close and releases lock. While User#1 is writing the new value back to the
file, another user came and read the unlocked file BEFORE the value was
increased. So he gets the same value as User#1. User#2 then increases the
value and writes the same data as User#1. In the end, the locking technique
failed ... am I right?

Another method I came up with is locking and entire read/write session
within a what I will call "parent lock" Here is the code:

#!/usr/bin/perl

use Fcntl qwDEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);

open (PARENT, "<locker.txt") or die "Can't open file: $!"; #an empty file
flock (PARENT, LOCK_EX) or die "Can't lock file: $!";

open (NUMBER, "<number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_SH) or die "Can't lock file: $!";
$value = <NUMBER>;
close(NUMBER);

$value = $value++;

open (NUMBER, ">number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_SH) or die "Can't lock file: $!";
print NUMBER $value;
close(NUMBER);

close(PARENT); #closes the parent lock

Now User #1 locks the parent file and moves on to the read/write session.
All other users get stuck back at the parent lock until released. I had to
use LOCK_SH for the read/write locks for some reason ... if all locks were
set to EX the program would stall. Is this technique better? Is it
effective? Or perhaps they are both ineffective?

In this next example, I don't want to increase a value, just append to the
file on a new line. So I do this:

#!/usr/bin/perl

use Fcntl qwDEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);

open (DATA, ">>locker.txt") or die "Can't open file: $!";
flock (DATA,LOCK_EX) or die "Can't lock file: $!";
print DATA "$data\n";
close(DATA);

Im going to assume that this is ok.

It's just when increasing a value that concerns me ... multipul users
reading same value when the read lock is released. Your suggestions are
greatly welcomes. TIA

Robert

 

[Ben Morrow]

Hi, I am asking foir advice on which form of code i've written below is the
correct and safe way to lock a file. I've done some reading on" use Fcntl
qwDEFAULT :flock);" and I understand the basics. I've also read back on
many posts regarding file locking and there seems to be much debate. For
this code, I simply want to increase a value.

#!/usr/bin/perl

use Fcntl qwDEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);

open (NUMBER, "<number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
$value = <NUMBER>;
close(NUMBER);

$value = $value++;

open (NUMBER, ">number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
print NUMBER $value;
close(NUMBER);

The above code seems to work ok. My question is when the first user locks
and reads the file, does the locking prevent other users from reading the
file?

No. It simply prevents other processes from *obtaining a lock* on the
file.

The $value is then increased and printed back to the file.

Another concern of mine is ... User#1 locks file, reads file value, then
close and releases lock. While User#1 is writing the new value back to the
file, another user came and read the unlocked file BEFORE the value was
increased. So he gets the same value as User#1. User#2 then increases the
value and writes the same data as User#1. In the end, the locking technique
failed ... am I right?

You are. You must read/modify/write under the same, exclusive, lock;
anyone who is just reading can use a shared lock.

Another method I came up with is locking and entire read/write session
within a what I will call "parent lock" Here is the code:

This would be correct, but there is no need for two layers of locks. The
inner lock is made completely redundant by the outer.

Correct technique is:

Processes that will only read:

open read-only
flock LOCK_SH
read the data
close

Processes that will read and modify:

open read-write ('+<')
flock LOCK_EX
read the data
make the modifications
seek to the start of the file (perldoc -f seek)
truncate the file (perldoc -f truncate)
write the new data
close

Alternatively, you can use a lockfile instead of proper flock locking;
this can be simpler, but you have to be careful to clean up after
yourself:

sysopen a lockfile with mode O_CREAT | O_EXCL
if the open fails, sleep for a bit and try again
open/close/read/write the data file as you will
unlink the lockfile

One problem with this is you may wish to check the age of the lockfile
and summerarily (sp?) unlink it if it appears to be too old...

I had to use LOCK_SH for the read/write locks for some reason ... if
all locks were set to EX the program would stall.

On some systems you cannot LOCK_EX a file unless you opened it for
writing.

In this next example, I don't want to increase a value, just append to the
file on a new line. So I do this:

#!/usr/bin/perl

use Fcntl qwDEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);

open (DATA, ">>locker.txt") or die "Can't open file: $!";
flock (DATA,LOCK_EX) or die "Can't lock file: $!";
print DATA "$data\n";
close(DATA);

Im going to assume that this is ok.

Note that your readers will need to lock as well.

Ben

 

[Bob Walton]

Hi, I am asking foir advice on which form of code i've written below is the
correct and safe way to lock a file. I've done some reading on" use Fcntl

I have found that the most portable and least hassle way to lock files
is to use a lock file. This is a file which is used exclusively for the
purpose of establishing a lock -- it can be empty, or have any contents
you want, but the data in the lock file would be unused and unimportant.
This technique is particularly useful when doing stuff which is a bit
out of the ordinary, such as locking DBM-type files to which you wish to
tie a hash, for example.

If you want to write a file, open the lock file for write (that will, at
least on some systems, wipe out the contents of the lock file), then
establish an exclusive lock on it. Then go do whatever you wanted the
exclusive lock for (write another file, tie a hash to a DBM-type file,
or any other operation requiring exclusive access to a resource). Then,
when you are all done with your lock, close the lock file.

If you want to read a file, open the lock file for reading, and
establish a shared lock on it. Go read your data file or do whatever
you wanted the shared lock for. When done, close the lock file.

To the best of my knowledge, that works flawlessly on at least local
files on every OS that supports locking (that, by the way, rules out
Windoze 95/98/98SE). However, you should carefully and thoroughly read
everything your OS has to say about file locking, particularly if you
are considering locking files accessed via a network.

You will want to note that locking a file does not prevent other
processes from read/writing/deleting/whatevering a file. *All* it does
is establish a lock -- the other processes have to cooperate with the
lock in order for it to work. An exclusive lock means other processes
seeking a lock will block until the exclusive lock is released; a shared
lock means other processes seeking an exclusive lock will block until
all the shared locks are released.

I find it convenient to write a couple of subs to do the locking, like
[untested]:

sub lockex{
open LOCK,">lockfile.txt"
or die "Oops, couldn't open lockfile.txt for write, $!";
flock(LOCK,LOCK_EX)
or die "flock bombed out in lockex, $!";
}
sub locksh{
open LOCK,"lockfile.txt"
or die "Oops, couldn't open lockfile.txt for read, $!";
flock(LOCK,LOCK_SH)
or die "flock bombed out in locksh, $!";
}
sub unlock{
close LOCK or die "Oops, couldn't close lockfile.txt, $!";
}

HTH.


....

 

[Ben Morrow]

Quoth (e-mail address removed):

I have found that the most portable and least hassle way to lock files
is to use a lock file. This is a file which is used exclusively for the
purpose of establishing a lock -- it can be empty, or have any contents
you want, but the data in the lock file would be unused and unimportant.
This technique is particularly useful when doing stuff which is a bit
out of the ordinary, such as locking DBM-type files to which you wish to
tie a hash, for example.

You can still perfectly well take a flock lock on the DBM file.

If you want to write a file, open the lock file for write (that will, at
least on some systems, wipe out the contents of the lock file), then
establish an exclusive lock on it.

The usual reason for using a lockfile is to avoid making locking calls
altogether... a call to sysopen with O_CREAT | O_EXCL will atomically
test-and-create the named file, so there is no need to then lock it.

To the best of my knowledge, that works flawlessly on at least local
files on every OS that supports locking (that, by the way, rules out
Windoze 95/98/98SE).

However, I think that win98 *does* support O_EXCL properly, so that
method will work.

However, you should carefully and thoroughly read
everything your OS has to say about file locking, particularly if you
are considering locking files accessed via a network.

Yes. I belive most network filesystems do it properly; the exception is
NFSv2, and there are (at least on linux) instructions in open(2) for how
to safely lock files over NFS.

I find it convenient to write a couple of subs to do the locking, like
[untested]:

sub lockex{
open LOCK,">lockfile.txt"

Note that if you choose to put any useful information in the lockfile
(it is common to put the pid and possibly the hostname of the process
which created the lock, so later processes can check if you still exist)
you would need to sysopen it with O_RDWR | O_CREAT to ensure you didn't
destroy info about someone else's lock.

or die "Oops, couldn't open lockfile.txt for write, $!";
flock(LOCK,LOCK_EX)
or die "flock bombed out in lockex, $!";
}
sub locksh{
open LOCK,"lockfile.txt"
or die "Oops, couldn't open lockfile.txt for read, $!";
flock(LOCK,LOCK_SH)
or die "flock bombed out in locksh, $!";
}
sub unlock{
close LOCK or die "Oops, couldn't close lockfile.txt, $!";
}

Yuck! What happens if you call this twice, for different files? You'll
implicitly close the global FH and destroy your lock. Try

{
my %locks;

sub lock {
my ($file, $type) = shift;

open $locks{$file}, '>', "$file.lock"
or die "can't create $file.lock: $!";

flock $locks{$file}, $type
or die "can't get lock on $file.lock: $!";
}

sub unlock {
close $locks{$_[0]};
}
}

Ben

 

[Darren Dunham]

The usual reason for using a lockfile is to avoid making locking calls
altogether... a call to sysopen with O_CREAT | O_EXCL will atomically
test-and-create the named file, so there is no need to then lock it.

Assuming NFS isn't involved.

I often use a separate lockfile because I don't want to modify my data
file directly, lest a crash leave it in an inconsistent state.

Instead I lock a separate file so that I can move a new file or files
into place and not have to worry about my lock moving with it.

Yes. I belive most network filesystems do it properly; the exception is
NFSv2, and there are (at least on linux) instructions in open(2) for how
to safely lock files over NFS.

With increased cleanup requirements also, unfortunately. I tend not to
do it because things get so messy.

Note that if you choose to put any useful information in the lockfile
(it is common to put the pid and possibly the hostname of the process
which created the lock, so later processes can check if you still exist)
you would need to sysopen it with O_RDWR | O_CREAT to ensure you didn't
destroy info about someone else's lock.

The same as.. open (LOCK, "<+lockfile.txt").