Problem with FileLock

Discussion in 'Java' started by alejandrina, Jul 13, 2007.

  1. alejandrina

    alejandrina Guest

    Hi experts,

    We need to update a file (on a file server) from many different
    machines. To synchromize the updates I am using FileLock.

    Everything works as advertised (ie if a machine gets the lock,
    it writes to the file while the other machines wait, everything is
    written in the proper order). The same test, using Linux machines,
    fails: the file is clobbered (meaning one update gets on top of
    another). No Exceptions are thrown; in fact the debug statements
    indicate that all the machines are acquiring the exclusive lock)

    Anyone can shed light? Here is the critical method with some
    debug statements:

    public void write (String s) throws Exception {
    ByteBuffer bb = stringToByteBuffer(s);

    //lock the file and wait till we can
    FileChannel channel = fos.getChannel();
    FileLock lock = null;
    try {
    while ((lock = channel.tryLock()) == null) {
    System.out.println (Utils.getHostname() + " Failed lock...wait");
    Thread.sleep(100);
    }

    System.out.println (Utils.getHostname() + " Locked:" + lock);
    System.out.println (Utils.getHostname() + " Lock type is "+
    ((lock.isShared())?"shared":"exclusive"));

    System.out.println (Utils.getHostname() + " Is lock valid: " +
    lock.isValid());

    //write the title first if noone's done it
    //and they asked for one
    if (channel.size() <= 0 && bbTitle != null)
    channel.write (bbTitle);
    channel.write(bb);

    } catch (Exception e) {
    throw (e);
    } finally {
    if (lock != null) {
    System.out.println (Utils.getHostname() + " Releasing lock");
    lock.release();
    }
    }

    }
     
    alejandrina, Jul 13, 2007
    #1
    1. Advertising

  2. alejandrina

    Oliver Wong Guest

    "alejandrina" <> wrote in message
    news:...
    > Hi experts,
    >
    > We need to update a file (on a file server) from many different
    > machines. To synchromize the updates I am using FileLock.
    >
    > Everything works as advertised (ie if a machine gets the lock,
    > it writes to the file while the other machines wait, everything is
    > written in the proper order). The same test, using Linux machines,
    > fails: the file is clobbered (meaning one update gets on top of
    > another). No Exceptions are thrown; in fact the debug statements
    > indicate that all the machines are acquiring the exclusive lock)


    http://java.sun.com/javase/6/docs/api/java/nio/channels/FileLock.html
    <quote>
    Whether or not a lock actually prevents another program from accessing the
    content of the locked region is system-dependent and therefore
    unspecified. The native file-locking facilities of some systems are merely
    advisory, meaning that programs must cooperatively observe a known locking
    protocol in order to guarantee data integrity. On other systems native
    file locks are mandatory, meaning that if one program locks a region of a
    file then other programs are actually prevented from accessing that region
    in a way that would violate the lock. On yet other systems, whether native
    file locks are advisory or mandatory is configurable on a per-file basis.
    To ensure consistent and correct behavior across platforms, it is strongly
    recommended that the locks provided by this API be used as if they were
    advisory locks.
    [...]
    On some systems, closing a channel releases all locks held by the Java
    virtual machine on the underlying file regardless of whether the locks
    were acquired via that channel or via another channel open on the same
    file. It is strongly recommended that, within a program, a unique channel
    be used to acquire all locks on any given file.
    [...]
    In general, great care should be taken when locking files that reside on
    network filesystems.
    </quote>

    - Oliver
     
    Oliver Wong, Jul 13, 2007
    #2
    1. Advertising

  3. alejandrina

    alejandrina Guest

    On Jul 13, 6:55 pm, "Oliver Wong" <> wrote:
    > "alejandrina" <> wrote in message
    >
    > news:...
    >
    > > Hi experts,

    >
    > > We need to update a file (on a file server) from many different
    > > machines. To synchromize the updates I am using FileLock.

    >
    > > Everything works as advertised (ie if a machine gets the lock,
    > > it writes to the file while the other machines wait, everything is
    > > written in the proper order). The same test, using Linux machines,
    > > fails: the file is clobbered (meaning one update gets on top of
    > > another). No Exceptions are thrown; in fact the debug statements
    > > indicate that all the machines are acquiring the exclusive lock)

    >
    > http://java.sun.com/javase/6/docs/api/java/nio/channels/FileLock.html
    > <quote>
    > Whether or not a lock actually prevents another program from accessing the
    > content of the locked region is system-dependent and therefore
    > unspecified. The native file-locking facilities of some systems are merely
    > advisory, meaning that programs must cooperatively observe a known locking
    > protocol in order to guarantee data integrity. On other systems native
    > file locks are mandatory, meaning that if one program locks a region of a
    > file then other programs are actually prevented from accessing that region
    > in a way that would violate the lock. On yet other systems, whether native
    > file locks are advisory or mandatory is configurable on a per-file basis.
    > To ensure consistent and correct behavior across platforms, it is strongly
    > recommended that the locks provided by this API be used as if they were
    > advisory locks.
    > [...]
    > On some systems, closing a channel releases all locks held by the Java
    > virtual machine on the underlying file regardless of whether the locks
    > were acquired via that channel or via another channel open on the same
    > file. It is strongly recommended that, within a program, a unique channel
    > be used to acquire all locks on any given file.
    > [...]
    > In general, great care should be taken when locking files that reside on
    > network filesystems.
    > </quote>
    >
    > - Oliver


    Yes, I have read all this. So, what is the real meaning of "advisory"?
    Can you or can't you issue a lock on a file on Linux?

    Which systems are "advisory" and why doesn't anyone spell this out?
     
    alejandrina, Jul 14, 2007
    #3
  4. alejandrina wrote:
    > On Jul 13, 6:55 pm, "Oliver Wong" <> wrote:

    ....
    > Yes, I have read all this. So, what is the real meaning of "advisory"?

    ....

    "Advisory", in conjunction with locking, usually means that threads that
    do not choose to play by the rules can go ahead and access the protected
    resource regardless of the state of the locks.

    If that is the intended interpretation, I don't think the quoted passage
    helps with understanding your problem, because you have already checked
    that your threads only access the resource while in possession of an
    exclusive lock on it.

    Have you tried calling force() after the write? Maybe there is some
    buffering in the FileChannel that is delaying the effect of the write
    past the release of the lock.

    Patricia
     
    Patricia Shanahan, Jul 14, 2007
    #4
  5. alejandrina

    Lew Guest

    alejandrina wrote:
    > Yes, I have read all this. So, what is the real meaning of "advisory"?


    "Advisory" means that any program can find out if another program has a lock
    on a file, but need not respect it.

    > Can you or can't you issue a lock on a file on Linux?


    Yes, but you can't enforce it.

    > Which systems are "advisory"


    You have to check the docs for each system. Perhaps Google is your friend
    (GIYF) on this question - have you tried it?

    > and why doesn't anyone spell this out?


    Why should they?

    Besides, they do. It's in the docs for each system.

    --
    Lew
     
    Lew, Jul 14, 2007
    #5
  6. On Fri, 13 Jul 2007 14:34:14 -0700, alejandrina wrote:
    > We need to update a file (on a file server) from many different
    > machines. To synchromize the updates I am using FileLock.
    >
    > Everything works as advertised (ie if a machine gets the lock, it
    > writes to the file while the other machines wait, everything is
    > written in the proper order). The same test, using Linux machines,
    > fails: the file is clobbered (meaning one update gets on top of
    > another). No Exceptions are thrown; in fact the debug statements
    > indicate that all the machines are acquiring the exclusive lock)


    If you are accessing the files over NFS, then I believe (but am not
    sure) that

    - the locks may be unknown to the server, and consequently to other
    NFS clients.

    - each client might be caching its updates, which are not guaranteed
    to be visible to other clients without a close/open sequence in
    between (if even then). This could result in corruption even if the
    locking is in fact working as you expect it to.

    What happens when you run two instances of your application on the
    *same* host?

    I would recommend that you update the file from a single (server)
    process that does so on behalf of the other (client) processes, so the
    updates are atomic and strictly serialized, without any need for
    locking.

    /gordon

    --
     
    Gordon Beaton, Jul 14, 2007
    #6
  7. alejandrina

    Lew Guest

    Gordon Beaton wrote:
    > I would recommend that you update the file from a single (server)
    > process that does so on behalf of the other (client) processes, so the
    > updates are atomic and strictly serialized, without any need for
    > locking.


    This is an excellent pattern (it was called "resource manager" in QNX, I don't
    know of the GoF name for it if there is one) to protect a single, serialized
    resource from trouble with multiple clients. The resource manager listens for
    service requests for the resource, and mediates all such requests.

    --
    Lew
     
    Lew, Jul 14, 2007
    #7
  8. alejandrina

    Guest

    On Jul 14, 10:35 am, Lew <> wrote:
    > Gordon Beaton wrote:
    > > I would recommend that you update the file from a single (server)
    > > process that does so on behalf of the other (client) processes, so the
    > > updates are atomic and strictly serialized, without any need for
    > > locking.

    >
    > This is an excellent pattern (it was called "resource manager" in QNX, I don't
    > know of the GoF name for it if there is one) to protect a single, serialized
    > resource from trouble with multiple clients. The resource manager listens for
    > service requests for the resource, and mediates all such requests.
    >
    > --
    > Lew


    I concur, I have had a lot of trouble trying to get file locks working
    in a server environment (you can run into very nasty problems like a
    file that doesn't unlock). Using a shared object to take care of IO
    works, much, much better.

    ----
    Ben
    http://www.plink-search.com
     
    , Jul 14, 2007
    #8
  9. alejandrina

    Nigel Wade Guest

    Gordon Beaton wrote:

    > On Fri, 13 Jul 2007 14:34:14 -0700, alejandrina wrote:
    >> We need to update a file (on a file server) from many different
    >> machines. To synchromize the updates I am using FileLock.
    >>
    >> Everything works as advertised (ie if a machine gets the lock, it
    >> writes to the file while the other machines wait, everything is
    >> written in the proper order). The same test, using Linux machines,
    >> fails: the file is clobbered (meaning one update gets on top of
    >> another). No Exceptions are thrown; in fact the debug statements
    >> indicate that all the machines are acquiring the exclusive lock)

    >
    > If you are accessing the files over NFS, then I believe (but am not
    > sure) that
    >
    > - the locks may be unknown to the server, and consequently to other
    > NFS clients.
    >
    > - each client might be caching its updates, which are not guaranteed
    > to be visible to other clients without a close/open sequence in
    > between (if even then). This could result in corruption even if the
    > locking is in fact working as you expect it to.
    >
    > What happens when you run two instances of your application on the
    > *same* host?
    >
    > I would recommend that you update the file from a single (server)
    > process that does so on behalf of the other (client) processes, so the
    > updates are atomic and strictly serialized, without any need for
    > locking.
    >


    For NFS rpc.lockd must be running on the server, it must be contactable using
    RPC (i.e. the dynamic RPC port it is registered at must not be blocked by a
    firewall), each client needs to co-operates by sending the lock request, and
    waiting for the lock responses. Even then it might not work.

    NFS was designed to be stateless, principally so that it could survive a server
    re-boot or network drop-out. File locking is inherently stateful (hence the
    need for the additional service rpc.lockd) and the two don't mix well.
    Basically, don't attempt file locking over NFS unless you have plenty of time
    to spare to debug it thoroughly. There are many failure modes and you should
    not rely on it without rigorous testing of how you code reacts to each failure
    mode.

    --
    Nigel Wade, System Administrator, Space Plasma Physics Group,
    University of Leicester, Leicester, LE1 7RH, UK
    E-mail :
    Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
     
    Nigel Wade, Jul 16, 2007
    #9
  10. alejandrina

    alejandrina Guest

    On Jul 16, 5:31 am, Nigel Wade <> wrote:
    > Gordon Beaton wrote:
    > > On Fri, 13 Jul 2007 14:34:14 -0700, alejandrina wrote:
    > >> We need to update a file (on a file server) from many different
    > >> machines. To synchromize the updates I am using FileLock.

    >
    > >> Everything works as advertised (ie if a machine gets the lock, it
    > >> writes to the file while the other machines wait, everything is
    > >> written in the proper order). The same test, using Linux machines,
    > >> fails: the file is clobbered (meaning one update gets on top of
    > >> another). No Exceptions are thrown; in fact the debug statements
    > >> indicate that all the machines are acquiring the exclusive lock)

    >
    > > If you are accessing the files over NFS, then I believe (but am not
    > > sure) that

    >
    > > - the locks may be unknown to the server, and consequently to other
    > > NFS clients.

    >
    > > - each client might be caching its updates, which are not guaranteed
    > > to be visible to other clients without a close/open sequence in
    > > between (if even then). This could result in corruption even if the
    > > locking is in fact working as you expect it to.

    >
    > > What happens when you run two instances of your application on the
    > > *same* host?

    >
    > > I would recommend that you update the file from a single (server)
    > > process that does so on behalf of the other (client) processes, so the
    > > updates are atomic and strictly serialized, without any need for
    > > locking.

    >
    > For NFS rpc.lockd must be running on the server, it must be contactable using
    > RPC (i.e. the dynamic RPC port it is registered at must not be blocked by a
    > firewall), each client needs to co-operates by sending the lock request, and
    > waiting for the lock responses. Even then it might not work.
    >
    > NFS was designed to be stateless, principally so that it could survive a server
    > re-boot or network drop-out. File locking is inherently stateful (hence the
    > need for the additional service rpc.lockd) and the two don't mix well.
    > Basically, don't attempt file locking over NFS unless you have plenty of time
    > to spare to debug it thoroughly. There are many failure modes and you should
    > not rely on it without rigorous testing of how you code reacts to each failure
    > mode.
    >
    > --
    > Nigel Wade, System Administrator, Space Plasma Physics Group,
    > University of Leicester, Leicester, LE1 7RH, UK
    > E-mail :
    > Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555


    AHHHHHHHHHH. That's the kind of info we need! Thanks so much...I'll
    continue this discussion with the sys admin.
     
    alejandrina, Jul 16, 2007
    #10
  11. alejandrina

    alejandrina Guest

    On Jul 14, 9:51 am, Patricia Shanahan <> wrote:
    > alejandrina wrote:
    > > On Jul 13, 6:55 pm, "Oliver Wong" <> wrote:

    > ...
    > > Yes, I have read all this. So, what is the real meaning of "advisory"?

    >
    > ...
    >
    > "Advisory", in conjunction with locking, usually means that threads that
    > do not choose to play by the rules can go ahead and access the protected
    > resource regardless of the state of the locks.
    >
    > If that is the intended interpretation, I don't think the quoted passage
    > helps with understanding your problem, because you have already checked
    > that your threads only access the resource while in possession of an
    > exclusive lock on it.
    >
    > Have you tried calling force() after the write? Maybe there is some
    > buffering in the FileChannel that is delaying the effect of the write
    > past the release of the lock.
    >
    > Patricia


    That would completely negate the effect of the lock, wouldn't it?
     
    alejandrina, Jul 16, 2007
    #11
  12. alejandrina wrote:
    > On Jul 14, 9:51 am, Patricia Shanahan <> wrote:
    >> alejandrina wrote:
    >>> On Jul 13, 6:55 pm, "Oliver Wong" <> wrote:

    >> ...
    >>> Yes, I have read all this. So, what is the real meaning of "advisory"?

    >> ...
    >>
    >> "Advisory", in conjunction with locking, usually means that threads that
    >> do not choose to play by the rules can go ahead and access the protected
    >> resource regardless of the state of the locks.
    >>
    >> If that is the intended interpretation, I don't think the quoted passage
    >> helps with understanding your problem, because you have already checked
    >> that your threads only access the resource while in possession of an
    >> exclusive lock on it.
    >>
    >> Have you tried calling force() after the write? Maybe there is some
    >> buffering in the FileChannel that is delaying the effect of the write
    >> past the release of the lock.
    >>
    >> Patricia

    >
    > That would completely negate the effect of the lock, wouldn't it?
    >


    No, the lock would still serve to serialize the periods during which a
    thread is entitled to write to the file. The force is related to making
    sure the write really happens between during that period.

    However, given other posts I'm not sure this is worth trying until after
    you have checked that the lock demon is running on the NFS server.

    Patricia
     
    Patricia Shanahan, Jul 16, 2007
    #12
  13. alejandrina

    alejandrina Guest

    On Jul 16, 10:31 am, alejandrina <> wrote:
    > On Jul 16, 5:31 am, Nigel Wade <> wrote:
    >
    >
    >
    > > Gordon Beaton wrote:
    > > > On Fri, 13 Jul 2007 14:34:14 -0700, alejandrina wrote:
    > > >> We need to update a file (on a file server) from many different
    > > >> machines. To synchromize the updates I am using FileLock.

    >
    > > >> Everything works as advertised (ie if a machine gets the lock, it
    > > >> writes to the file while the other machines wait, everything is
    > > >> written in the proper order). The same test, using Linux machines,
    > > >> fails: the file is clobbered (meaning one update gets on top of
    > > >> another). No Exceptions are thrown; in fact the debug statements
    > > >> indicate that all the machines are acquiring the exclusive lock)

    >
    > > > If you are accessing the files over NFS, then I believe (but am not
    > > > sure) that

    >
    > > > - the locks may be unknown to the server, and consequently to other
    > > > NFS clients.

    >
    > > > - each client might be caching its updates, which are not guaranteed
    > > > to be visible to other clients without a close/open sequence in
    > > > between (if even then). This could result in corruption even if the
    > > > locking is in fact working as you expect it to.

    >
    > > > What happens when you run two instances of your application on the
    > > > *same* host?

    >
    > > > I would recommend that you update the file from a single (server)
    > > > process that does so on behalf of the other (client) processes, so the
    > > > updates are atomic and strictly serialized, without any need for
    > > > locking.

    >
    > > For NFS rpc.lockd must be running on the server, it must be contactable using
    > > RPC (i.e. the dynamic RPC port it is registered at must not be blocked by a
    > > firewall), each client needs to co-operates by sending the lock request, and
    > > waiting for the lock responses. Even then it might not work.

    >
    > > NFS was designed to be stateless, principally so that it could survive a server
    > > re-boot or network drop-out. File locking is inherently stateful (hence the
    > > need for the additional service rpc.lockd) and the two don't mix well.
    > > Basically, don't attempt file locking over NFS unless you have plenty of time
    > > to spare to debug it thoroughly. There are many failure modes and you should
    > > not rely on it without rigorous testing of how you code reacts to each failure
    > > mode.

    >
    > > --
    > > Nigel Wade, System Administrator, Space Plasma Physics Group,
    > > University of Leicester, Leicester, LE1 7RH, UK
    > > E-mail :
    > > Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555

    >
    > AHHHHHHHHHH. That's the kind of info we need! Thanks so much...I'll
    > continue this discussion with the sys admin.



    We were running Samba...When that went away, everything worked
    perfectly.
     
    alejandrina, Jul 30, 2007
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. David Zimmerman

    confused abt FileLock behavior

    David Zimmerman, Jul 28, 2003, in forum: Java
    Replies:
    3
    Views:
    457
  2. Replies:
    0
    Views:
    5,304
  3. Replies:
    4
    Views:
    8,552
    toddwh50
    Mar 11, 2010
  4. Harold Yarmouth

    Apparent bug in FileLock

    Harold Yarmouth, Nov 19, 2008, in forum: Java
    Replies:
    1
    Views:
    346
    Harold Yarmouth
    Nov 20, 2008
  5. Josef 'Jupp' SCHUGT

    epoch/2 bug in filelock.rb?

    Josef 'Jupp' SCHUGT, Jan 11, 2004, in forum: Ruby
    Replies:
    1
    Views:
    139
    Josef 'Jupp' SCHUGT
    Jan 14, 2004
Loading...

Share This Page