DB_File and File Locking -- Best Practices

Discussion in 'Perl Misc' started by Fiftyvolts, Jul 16, 2004.

  1. Fiftyvolts

    Fiftyvolts Guest

    I'm writing a set of CGI scripts that create and manage Berkeley DB
    files. I need them to be able to read and write to the files with out
    having other users klobber them before they get the chance. I want to
    use file locking but there are some issues.

    I am am unluckily stuck with a slow moving IT department managing the
    webserver, and am stuck with perl 5.005 with only the standard modules
    installed (and no good hope of getting a new version or module
    installed since, quite frankly, they don't know what they are doing and
    won't let me take care of myself). This is truly the one large
    difficulty in my purpose.

    Traditionally file locking was supposed to go like this:

    $x = tie %hash, DB_File, 'data', O_RDWR;
    open(DB, $x->fd);
    flock(DB, LOCK_EX);

    but it, as the author of DB_File reveales, someone pointed out that
    when you tie the %hash the file is read before the lock is made. He
    suggest several modules that can mimic flock, but I am unable to obtain
    them in a timely manner. So my question is wheater the following
    approach will work properly:

    open(DB, 'data');
    flock(DB, LOCK_EX);
    tie %hash, DB_File, 'data', O_RDWR;

    My initial testing seems to indicate that it does, but one never knows
    until something breaks. Any thoughts?

    Thanks,
    MST
     
    Fiftyvolts, Jul 16, 2004
    #1
    1. Advertising

  2. Fiftyvolts

    ko Guest

    Fiftyvolts wrote:
    > I'm writing a set of CGI scripts that create and manage Berkeley DB
    > files. I need them to be able to read and write to the files with out
    > having other users klobber them before they get the chance. I want to
    > use file locking but there are some issues.
    >
    > I am am unluckily stuck with a slow moving IT department managing the
    > webserver, and am stuck with perl 5.005 with only the standard modules
    > installed (and no good hope of getting a new version or module
    > installed since, quite frankly, they don't know what they are doing and
    > won't let me take care of myself). This is truly the one large
    > difficulty in my purpose.
    >


    'perldoc "own module"' from your shell to learn how to install modules
    that can be used be your CGI scripts.

    [snip]

    HTH -keith
     
    ko, Jul 17, 2004
    #2
    1. Advertising

  3. Fiftyvolts

    dan baker Guest

    "Fiftyvolts" <> wrote in message news:<cd929v$>...
    > I'm writing a set of CGI scripts that create and manage Berkeley DB
    > files. I need them to be able to read and write to the files with out
    > having other users klobber them before they get the chance.

    ---------------

    I had similar issues, for different reasons. Among them was that I
    needed to be able to do a 'flock' on a PC. I wrote a kludge that will
    probably work ok in a low-volume environment. Basically looking for
    the existance of a "lock file" before tie()ing the hash, then deleting
    the file when done....

    here are the subs to lock/unlock to give you some ideas:

    # ##############################################################################
    sub CreateLockFile { my ( $dbfile ) = @_ ;

    # ------------------------------------------------------------------------------
    # create a lock file or wait...

    my $LockFile = "${dbfile}.lock.txt" ;

    # check to see if its already locked
    # -----
    if ( $cDEBUG eq 'verbose' ) {
    print STDERR
    "\t entering CreateLockFile, checking for lockfile $LockFile \n" ;
    }

    my $TryCount = 10 ;
    TRY:
    if ( -f $LockFile ) {

    # whoops, lock file exists... gotta wait
    sleep 1;
    $TryCount-- ;

    if ( $TryCount ) {
    goto TRY ;
    } else {
    # if its more than 60 seconds old, assume its an orphan and
    overwrite
    @FileStats = stat( $LockFile );
    unless ( $FileStats[9] < (time()-60) ) {

    ExitToBrowser ( 'warn' ,
    "Database $dbfile is locked by another user. ".
    "Try again in a minute..." );
    }
    }
    }

    # create $LockFile
    # -----
    unless ( open( FILEIN , ">>$LockFile" )) {

    ExitToBrowser ( 'warn' ,
    "Could not create a lockfile for database $dbfile because $! ." );
    }

    print FILEIN "locked at ".scalar(localtime)."\n" ;
    close FILEIN ;

    if ( $cDEBUG eq 'verbose' ) {
    print STDERR "\t created a lockfile $LockFile , returning now... \n"
    ;
    }


    # ------------------------------------------------------------------------------

    1; }
    1;

    # ##############################################################################
    sub ReleaseLockFile { my ( $dbfile ) = @_ ;

    # ------------------------------------------------------------------------------

    my $LockFile = "${dbfile}.lock.txt" ;

    if ( -f $LockFile ) {
    unlink $LockFile ;
    } else {
    # should never see this....
    ExitToBrowser ( 'warn' ,
    "Could not find LockFile $LockFile to release it...." );
    }

    # ##############################################################################
    1; }
    1;
     
    dan baker, Jul 18, 2004
    #3
  4. Fiftyvolts

    Ben Morrow Guest

    Quoth (dan baker):
    > "Fiftyvolts" <> wrote in message news:<cd929v$>...
    > > I'm writing a set of CGI scripts that create and manage Berkeley DB
    > > files. I need them to be able to read and write to the files with out
    > > having other users klobber them before they get the chance.

    > ---------------
    >
    > I had similar issues, for different reasons. Among them was that I
    > needed to be able to do a 'flock' on a PC. I wrote a kludge that will
    > probably work ok in a low-volume environment. Basically looking for
    > the existance of a "lock file" before tie()ing the hash, then deleting
    > the file when done....


    This is a perfectly sensible way to do the locking, but the
    implementation is AWFUL. Do you understand the concept of an 'atomic
    operation', and why it is relevant to locking?

    > here are the subs to lock/unlock to give you some ideas:
    >
    > # ##############################################################################
    > sub CreateLockFile { my ( $dbfile ) = @_ ;


    Whitespace is free. Please use it. Especially newlines and decent
    indentation.

    > # ------------------------------------------------------------------------------
    > # create a lock file or wait...
    >
    > my $LockFile = "${dbfile}.lock.txt" ;
    >
    > # check to see if its already locked
    > # -----
    > if ( $cDEBUG eq 'verbose' ) {
    > print STDERR
    > "\t entering CreateLockFile, checking for lockfile $LockFile \n" ;
    > }
    >
    > my $TryCount = 10 ;
    > TRY:
    > if ( -f $LockFile ) {


    NO NO NO. Here you have a race condition: you might as well not lock at
    all.

    use Fcntl;

    sub CreateLockFile {
    my ($dbfile) = @_;
    my $LockFile = "$dbfile.lock.txt";
    my $LCK;

    if (
    my @FileStats = stat $LockFile
    and $FileStats[9] < ( time - 60 )
    ) {
    unlink $FileStats
    or die "can't unlink stale lockfile $LockFile: $!";
    }

    while (
    not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
    # O_EXCL is supposed to be atomic
    and $TryCount
    ) {
    sleep 1;
    $LCK = undef;
    $TryCount--;
    }

    $LCK or die "can't create lockfile $LockFile: $!";

    print $LCK "Pid $$ locked at " . (scalar localtime) . "\n";
    }

    > # ##############################################################################
    > sub ReleaseLockFile { my ( $dbfile ) = @_ ;
    >
    > # ------------------------------------------------------------------------------
    >
    > my $LockFile = "${dbfile}.lock.txt" ;
    >
    > if ( -f $LockFile ) {


    Again, a race condition. Just try to unlink the lockfile: if it didn't
    exist unlink will fail.

    Ben

    --
    Although few may originate a policy, we are all able to judge it.
    - Pericles of Athens, c.430 B.C.
     
    Ben Morrow, Jul 18, 2004
    #4
  5. Fiftyvolts

    dan baker Guest

    Ben Morrow <> wrote in message

    > This is a perfectly sensible way to do the locking, but the
    > implementation is AWFUL. Do you understand the concept of an 'atomic
    > operation', and why it is relevant to locking?

    ---------
    I'm always open to learn... but remember that one of my conditions was
    to have this code work on a PC. atomic, flocking, and thread control
    is kinda out the window. unless you have better ways to do it in a
    Windows environment?


    > Whitespace is free. Please use it. Especially newlines and decent
    > indentation.

    -------
    sorry, but posting source via yahoo does weird things to indents.




    > not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
    > # O_EXCL is supposed to be atomic

    -------------
    I dont think this will work on ActiveState perl running on a PC, will
    it?




    > Again, a race condition. Just try to unlink the lockfile: if it didn't
    > exist unlink will fail.
    > -------------

    I dont want it to fail.... I want it to try a couple times to skip
    over an execution lock that another user may have.


    d
     
    dan baker, Jul 19, 2004
    #5
  6. Fiftyvolts

    Jeff Stampes Guest

    dan baker wrote:

    > Ben Morrow <> wrote in message
    >
    >> not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
    >> # O_EXCL is supposed to be atomic

    > -------------
    > I dont think this will work on ActiveState perl running on a PC, will
    > it?


    And it won't work over NFS either.

    ~Jeff
     
    Jeff Stampes, Jul 19, 2004
    #6
  7. Fiftyvolts

    Ben Morrow Guest

    Quoth (dan baker):
    > Ben Morrow <> wrote in message
    >
    > > This is a perfectly sensible way to do the locking, but the
    > > implementation is AWFUL. Do you understand the concept of an 'atomic
    > > operation', and why it is relevant to locking?

    > ---------
    > I'm always open to learn... but remember that one of my conditions was
    > to have this code work on a PC. atomic, flocking, and thread control
    > is kinda out the window. unless you have better ways to do it in a
    > Windows environment?


    If your locking operation isn't atomic there is no point locking. You've
    got a race condition anyway so you might as well go ahead and do
    whatever without locks and pray.

    > > Whitespace is free. Please use it. Especially newlines and decent
    > > indentation.

    > -------
    > sorry, but posting source via yahoo does weird things to indents.


    Get a decent newsreader then.

    > > not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
    > > # O_EXCL is supposed to be atomic

    > -------------
    > I dont think this will work on ActiveState perl running on a PC, will
    > it?


    It will certainly work. Whether O_EXCL is properly atomic or not under
    Win9x I don't know; you'd need to check MSDN.

    > > Again, a race condition. Just try to unlink the lockfile: if it didn't
    > > exist unlink will fail.
    > > -------------

    > I dont want it to fail.... I want it to try a couple times to skip
    > over an execution lock that another user may have.


    When I say 'fail' I mean 'unlink will return undef if the file was
    already unlinked'. It may well anyway, as the code is written: there's
    nothing to stop someone else unlinking the file between your -f test and
    your unlink. This is what I mean by a 'race condition': you and another
    process are racing to unlink the file, and whomever loses will get an
    unexpected result (in this case, an error from unlink).

    Don't bother with the -f, just do the unlink. If it's there, it'll
    delete it; if it's not, it doesn't matter 'cos it's not there. To be
    properly careful, you'd check, if unlink failed, that $! is ENOENT and
    tell the user if it wasn't. That'll catch things like someone having the
    lockfile open so you can't delete it (damnable win32 mandatory file
    locks!).

    Ben

    --
    Musica Dei donum optimi, trahit homines, trahit deos. |
    Musica truces molit animos, tristesque mentes erigit. |
    Musica vel ipsas arbores et horridas movet feras. |
     
    Ben Morrow, Jul 19, 2004
    #7
  8. Fiftyvolts

    ko Guest

    dan baker wrote:
    > Ben Morrow <> wrote in message
    >
    >>This is a perfectly sensible way to do the locking, but the
    >>implementation is AWFUL. Do you understand the concept of an 'atomic
    >>operation', and why it is relevant to locking?

    >
    > I'm always open to learn... but remember that one of my conditions was
    > to have this code work on a PC. atomic, flocking, and thread control
    > is kinda out the window. unless you have better ways to do it in a
    > Windows environment?
    >
    >>Whitespace is free. Please use it. Especially newlines and decent
    >>indentation.

    >
    > sorry, but posting source via yahoo does weird things to indents.
    >
    >> not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
    >> # O_EXCL is supposed to be atomic

    >
    > -------------
    > I dont think this will work on ActiveState perl running on a PC, will
    > it?


    What makes you think that it doesn't work? Did you try running Ben's
    code? You *want* to use sysopen() to get it right. I had to make a
    couple of changes to get it running:

    1. if ( (stat $LockFile)[9] < time - 60 ) {

    2. pass $LockFile to unlink().

    Then comment out the block that 'unlink's $LockFile and see what
    happens. (if the lockfile doesn't already exist, run the script twice)

    Reference 'perldoc perlport' if you are uncertain whether a particular
    Perl feature/function is available to your OS.

    HTH - keith

    Also, if the block that unlinks the lockfile is commented out, the
    script doesn't die even though $LCK is undefined in the while loop.
    (does die if 0 is assigned to $LCK or test for an open filehandle with
    fileno() ) Its probably obvious, but I can't figure out why - could
    someone please explain? (ActiveState Perl 5.8.4)

    Thanks
     
    ko, Jul 20, 2004
    #8
  9. Fiftyvolts

    Rich Grise Guest

    dan baker wrote:

    > Ben Morrow <> wrote in message

    ....
    >> not sysopen $LCK, $LockFile, O_WRONLY | O_CREAT | O_EXCL
    >> # O_EXCL is supposed to be atomic

    > -------------
    > I dont think this will work on ActiveState perl running on a PC, will
    > it?
    >

    I don't know the Doze-compatible perl system calls, but O_EXCL has
    always been an option in MSC/DOS, AFAIK.

    Good Luck!
    Rich
     
    Rich Grise, Aug 8, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Timasmith
    Replies:
    4
    Views:
    459
    Bjorn Borud
    Nov 1, 2006
  2. 187
    Replies:
    4
    Views:
    212
    Paul Marquess
    Apr 6, 2004
  3. botfood
    Replies:
    5
    Views:
    185
    botfood
    Apr 26, 2006
  4. botfood
    Replies:
    3
    Views:
    145
    J. Gleixner
    Sep 18, 2006
  5. Chicken McNuggets

    Best book on C gotchas and best practices?

    Chicken McNuggets, Jul 31, 2013, in forum: C Programming
    Replies:
    9
    Views:
    270
    Fred J. Tydeman
    Aug 5, 2013
Loading...

Share This Page