DB_File and File Locking -- Best Practices

[Fiftyvolts]

I'm writing a set of CGI scripts that create and manage Berkeley DB
files. I need them to be able to read and write to the files with out
having other users klobber them before they get the chance. I want to
use file locking but there are some issues.

I am am unluckily stuck with a slow moving IT department managing the
webserver, and am stuck with perl 5.005 with only the standard modules
installed (and no good hope of getting a new version or module
installed since, quite frankly, they don't know what they are doing and
won't let me take care of myself). This is truly the one large
difficulty in my purpose.

Traditionally file locking was supposed to go like this:

$x = tie %hash, DB_File, 'data', O_RDWR;
open(DB, $x->fd);
flock(DB, LOCK_EX);

but it, as the author of DB_File reveales, someone pointed out that
when you tie the %hash the file is read before the lock is made. He
suggest several modules that can mimic flock, but I am unable to obtain
them in a timely manner. So my question is wheater the following
approach will work properly:

open(DB, 'data');
flock(DB, LOCK_EX);
tie %hash, DB_File, 'data', O_RDWR;

My initial testing seems to indicate that it does, but one never knows
until something breaks. Any thoughts?

Thanks,
MST

 

[ko]

I'm writing a set of CGI scripts that create and manage Berkeley DB
files. I need them to be able to read and write to the files with out
having other users klobber them before they get the chance. I want to
use file locking but there are some issues.

I am am unluckily stuck with a slow moving IT department managing the
webserver, and am stuck with perl 5.005 with only the standard modules
installed (and no good hope of getting a new version or module
installed since, quite frankly, they don't know what they are doing and
won't let me take care of myself). This is truly the one large
difficulty in my purpose.

'perldoc "own module"' from your shell to learn how to install modules
that can be used be your CGI scripts.

[snip]

HTH -keith

 

[dan baker]

I'm writing a set of CGI scripts that create and manage Berkeley DB
files. I need them to be able to read and write to the files with out
having other users klobber them before they get the chance.

---------------

I had similar issues, for different reasons. Among them was that I
needed to be able to do a 'flock' on a PC. I wrote a kludge that will
probably work ok in a low-volume environment. Basically looking for
the existance of a "lock file" before tie()ing the hash, then deleting
the file when done....

here are the subs to lock/unlock to give you some ideas:

# ##############################################################################
sub CreateLockFile { my ( $dbfile ) = @_ ;

# ------------------------------------------------------------------------------
# create a lock file or wait...

my $LockFile = "${dbfile}.lock.txt" ;

# check to see if its already locked
# -----
if ( $cDEBUG eq 'verbose' ) {
print STDERR
"\t entering CreateLockFile, checking for lockfile $LockFile \n" ;
}

my $TryCount = 10 ;
TRY:
if ( -f $LockFile ) {

# whoops, lock file exists... gotta wait
sleep 1;
$TryCount-- ;

if ( $TryCount ) {
goto TRY ;
} else {
# if its more than 60 seconds old, assume its an orphan and
overwrite
@FileStats = stat( $LockFile );
unless ( $FileStats[9] < (time()-60) ) {

ExitToBrowser ( 'warn' ,
"Database $dbfile is locked by another user. ".
"Try again in a minute..." );
}
}
}

# create $LockFile
# -----
unless ( open( FILEIN , ">>$LockFile" )) {

ExitToBrowser ( 'warn' ,
"Could not create a lockfile for database $dbfile because $! ." );
}

print FILEIN "locked at ".scalar(localtime)."\n" ;
close FILEIN ;

if ( $cDEBUG eq 'verbose' ) {
print STDERR "\t created a lockfile $LockFile , returning now... \n"
;
}


# ------------------------------------------------------------------------------

1; }
1;

# ##############################################################################
sub ReleaseLockFile { my ( $dbfile ) = @_ ;

# ------------------------------------------------------------------------------

my $LockFile = "${dbfile}.lock.txt" ;

if ( -f $LockFile ) {
unlink $LockFile ;
} else {
# should never see this....
ExitToBrowser ( 'warn' ,
"Could not find LockFile $LockFile to release it...." );
}

# ##############################################################################
1; }
1;

 

[Ben Morrow]

Quoth (e-mail address removed) (dan baker):

---------------

I had similar issues, for different reasons. Among them was that I
needed to be able to do a 'flock' on a PC. I wrote a kludge that will
probably work ok in a low-volume environment. Basically looking for
the existance of a "lock file" before tie()ing the hash, then deleting
the file when done....

This is a perfectly sensible way to do the locking, but the
implementation is AWFUL. Do you understand the concept of an 'atomic
operation', and why it is relevant to locking?

here are the subs to lock/unlock to give you some ideas:

# ##############################################################################
sub CreateLockFile { my ( $dbfile ) = @_ ;

Whitespace is free. Please use it. Especially newlines and decent
indentation.

# ------------------------------------------------------------------------------
# create a lock file or wait...

my $LockFile = "${dbfile}.lock.txt" ;

# check to see if its already locked
# -----
if ( $cDEBUG eq 'verbose' ) {
print STDERR
"\t entering CreateLockFile, checking for lockfile $LockFile \n" ;
}

my $TryCount = 10 ;
TRY:
if ( -f $LockFile ) {

NO NO NO. Here you have a race condition: you might as well not lock at
all.

use Fcntl;

sub CreateLockFile {
my ($dbfile) = @_;
my $LockFile = "$dbfile.lock.txt";
my $LCK;

if (
my @FileStats = stat $LockFile
and $FileStats[9] < ( time - 60 )
) {
unlink $FileStats
or die "can't unlink stale lockfile $LockFile: $!";
}

while (
not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
# O_EXCL is supposed to be atomic
and $TryCount
) {
sleep 1;
$LCK = undef;
$TryCount--;
}

$LCK or die "can't create lockfile $LockFile: $!";

print $LCK "Pid $$ locked at " . (scalar localtime) . "\n";
}

# ##############################################################################
sub ReleaseLockFile { my ( $dbfile ) = @_ ;

# ------------------------------------------------------------------------------

my $LockFile = "${dbfile}.lock.txt" ;

if ( -f $LockFile ) {

Again, a race condition. Just try to unlink the lockfile: if it didn't
exist unlink will fail.

Ben

 

[dan baker]

This is a perfectly sensible way to do the locking, but the
implementation is AWFUL. Do you understand the concept of an 'atomic
operation', and why it is relevant to locking?

---------
I'm always open to learn... but remember that one of my conditions was
to have this code work on a PC. atomic, flocking, and thread control
is kinda out the window. unless you have better ways to do it in a
Windows environment?

Whitespace is free. Please use it. Especially newlines and decent
indentation.

-------
sorry, but posting source via yahoo does weird things to indents.

not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
# O_EXCL is supposed to be atomic

-------------
I dont think this will work on ActiveState perl running on a PC, will
it?

Again, a race condition. Just try to unlink the lockfile: if it didn't
exist unlink will fail.
-------------

I dont want it to fail.... I want it to try a couple times to skip
over an execution lock that another user may have.


d

 

[Jeff Stampes]

And it won't work over NFS either.

~Jeff

 

[Ben Morrow]

Quoth (e-mail address removed) (dan baker):

---------
I'm always open to learn... but remember that one of my conditions was
to have this code work on a PC. atomic, flocking, and thread control
is kinda out the window. unless you have better ways to do it in a
Windows environment?

If your locking operation isn't atomic there is no point locking. You've
got a race condition anyway so you might as well go ahead and do
whatever without locks and pray.

Get a decent newsreader then.

It will certainly work. Whether O_EXCL is properly atomic or not under
Win9x I don't know; you'd need to check MSDN.

I dont want it to fail.... I want it to try a couple times to skip
over an execution lock that another user may have.

When I say 'fail' I mean 'unlink will return undef if the file was
already unlinked'. It may well anyway, as the code is written: there's
nothing to stop someone else unlinking the file between your -f test and
your unlink. This is what I mean by a 'race condition': you and another
process are racing to unlink the file, and whomever loses will get an
unexpected result (in this case, an error from unlink).

Don't bother with the -f, just do the unlink. If it's there, it'll
delete it; if it's not, it doesn't matter 'cos it's not there. To be
properly careful, you'd check, if unlink failed, that $! is ENOENT and
tell the user if it wasn't. That'll catch things like someone having the
lockfile open so you can't delete it (damnable win32 mandatory file
locks!).

Ben

 

[ko]

I'm always open to learn... but remember that one of my conditions was
to have this code work on a PC. atomic, flocking, and thread control
is kinda out the window. unless you have better ways to do it in a
Windows environment?


sorry, but posting source via yahoo does weird things to indents.

What makes you think that it doesn't work? Did you try running Ben's
code? You *want* to use sysopen() to get it right. I had to make a
couple of changes to get it running:

1. if ( (stat $LockFile)[9] < time - 60 ) {

2. pass $LockFile to unlink().

Then comment out the block that 'unlink's $LockFile and see what
happens. (if the lockfile doesn't already exist, run the script twice)

Reference 'perldoc perlport' if you are uncertain whether a particular
Perl feature/function is available to your OS.

HTH - keith

Also, if the block that unlinks the lockfile is commented out, the
script doesn't die even though $LCK is undefined in the while loop.
(does die if 0 is assigned to $LCK or test for an open filehandle with
fileno() ) Its probably obvious, but I can't figure out why - could
someone please explain? (ActiveState Perl 5.8.4)

Thanks

 

[Rich Grise]

I don't know the Doze-compatible perl system calls, but O_EXCL has
always been an option in MSC/DOS, AFAIK.

Good Luck!
Rich