Problem with FileLock

[alejandrina]

Hi experts,

We need to update a file (on a file server) from many different
machines. To synchromize the updates I am using FileLock.

Everything works as advertised (ie if a machine gets the lock,
it writes to the file while the other machines wait, everything is
written in the proper order). The same test, using Linux machines,
fails: the file is clobbered (meaning one update gets on top of
another). No Exceptions are thrown; in fact the debug statements
indicate that all the machines are acquiring the exclusive lock)

Anyone can shed light? Here is the critical method with some
debug statements:

public void write (String s) throws Exception {
ByteBuffer bb = stringToByteBuffer(s);

//lock the file and wait till we can
FileChannel channel = fos.getChannel();
FileLock lock = null;
try {
while ((lock = channel.tryLock()) == null) {
System.out.println (Utils.getHostname() + " Failed lock...wait");
Thread.sleep(100);
}

System.out.println (Utils.getHostname() + " Locked:" + lock);
System.out.println (Utils.getHostname() + " Lock type is "+
((lock.isShared())?"shared":"exclusive"));

System.out.println (Utils.getHostname() + " Is lock valid: " +
lock.isValid());

//write the title first if noone's done it
//and they asked for one
if (channel.size() <= 0 && bbTitle != null)
channel.write (bbTitle);
channel.write(bb);

} catch (Exception e) {
throw (e);
} finally {
if (lock != null) {
System.out.println (Utils.getHostname() + " Releasing lock");
lock.release();
}
}

}

 

[Oliver Wong]

Hi experts,

We need to update a file (on a file server) from many different
machines. To synchromize the updates I am using FileLock.

Everything works as advertised (ie if a machine gets the lock,
it writes to the file while the other machines wait, everything is
written in the proper order). The same test, using Linux machines,
fails: the file is clobbered (meaning one update gets on top of
another). No Exceptions are thrown; in fact the debug statements
indicate that all the machines are acquiring the exclusive lock)

http://java.sun.com/javase/6/docs/api/java/nio/channels/FileLock.html
<quote>
Whether or not a lock actually prevents another program from accessing the
content of the locked region is system-dependent and therefore
unspecified. The native file-locking facilities of some systems are merely
advisory, meaning that programs must cooperatively observe a known locking
protocol in order to guarantee data integrity. On other systems native
file locks are mandatory, meaning that if one program locks a region of a
file then other programs are actually prevented from accessing that region
in a way that would violate the lock. On yet other systems, whether native
file locks are advisory or mandatory is configurable on a per-file basis.
To ensure consistent and correct behavior across platforms, it is strongly
recommended that the locks provided by this API be used as if they were
advisory locks.
[...]
On some systems, closing a channel releases all locks held by the Java
virtual machine on the underlying file regardless of whether the locks
were acquired via that channel or via another channel open on the same
file. It is strongly recommended that, within a program, a unique channel
be used to acquire all locks on any given file.
[...]
In general, great care should be taken when locking files that reside on
network filesystems.
</quote>

- Oliver

 

[alejandrina]

Hi experts,

We need to update a file (on a file server) from many different
machines. To synchromize the updates I am using FileLock.

Everything works as advertised (ie if a machine gets the lock,
it writes to the file while the other machines wait, everything is
written in the proper order). The same test, using Linux machines,
fails: the file is clobbered (meaning one update gets on top of
another). No Exceptions are thrown; in fact the debug statements
indicate that all the machines are acquiring the exclusive lock)

http://java.sun.com/javase/6/docs/api/java/nio/channels/FileLock.html
<quote>
Whether or not a lock actually prevents another program from accessing the
content of the locked region is system-dependent and therefore
unspecified. The native file-locking facilities of some systems are merely
advisory, meaning that programs must cooperatively observe a known locking
protocol in order to guarantee data integrity. On other systems native
file locks are mandatory, meaning that if one program locks a region of a
file then other programs are actually prevented from accessing that region
in a way that would violate the lock. On yet other systems, whether native
file locks are advisory or mandatory is configurable on a per-file basis.
To ensure consistent and correct behavior across platforms, it is strongly
recommended that the locks provided by this API be used as if they were
advisory locks.
[...]
On some systems, closing a channel releases all locks held by the Java
virtual machine on the underlying file regardless of whether the locks
were acquired via that channel or via another channel open on the same
file. It is strongly recommended that, within a program, a unique channel
be used to acquire all locks on any given file.
[...]
In general, great care should be taken when locking files that reside on
network filesystems.
</quote>

- Oliver

Yes, I have read all this. So, what is the real meaning of "advisory"?
Can you or can't you issue a lock on a file on Linux?

Which systems are "advisory" and why doesn't anyone spell this out?

 

[Patricia Shanahan]

Yes, I have read all this. So, what is the real meaning of "advisory"?

....

"Advisory", in conjunction with locking, usually means that threads that
do not choose to play by the rules can go ahead and access the protected
resource regardless of the state of the locks.

If that is the intended interpretation, I don't think the quoted passage
helps with understanding your problem, because you have already checked
that your threads only access the resource while in possession of an
exclusive lock on it.

Have you tried calling force() after the write? Maybe there is some
buffering in the FileChannel that is delaying the effect of the write
past the release of the lock.

Patricia

 

[Lew]

Yes, I have read all this. So, what is the real meaning of "advisory"?

"Advisory" means that any program can find out if another program has a lock
on a file, but need not respect it.

Can you or can't you issue a lock on a file on Linux?

Yes, but you can't enforce it.

Which systems are "advisory"

You have to check the docs for each system. Perhaps Google is your friend
(GIYF) on this question - have you tried it?

and why doesn't anyone spell this out?

Why should they?

Besides, they do. It's in the docs for each system.

 

[Gordon Beaton]

We need to update a file (on a file server) from many different
machines. To synchromize the updates I am using FileLock.

Everything works as advertised (ie if a machine gets the lock, it
writes to the file while the other machines wait, everything is
written in the proper order). The same test, using Linux machines,
fails: the file is clobbered (meaning one update gets on top of
another). No Exceptions are thrown; in fact the debug statements
indicate that all the machines are acquiring the exclusive lock)

If you are accessing the files over NFS, then I believe (but am not
sure) that

- the locks may be unknown to the server, and consequently to other
NFS clients.

- each client might be caching its updates, which are not guaranteed
to be visible to other clients without a close/open sequence in
between (if even then). This could result in corruption even if the
locking is in fact working as you expect it to.

What happens when you run two instances of your application on the
*same* host?

I would recommend that you update the file from a single (server)
process that does so on behalf of the other (client) processes, so the
updates are atomic and strictly serialized, without any need for
locking.

/gordon

--

 

[Lew]

I would recommend that you update the file from a single (server)
process that does so on behalf of the other (client) processes, so the
updates are atomic and strictly serialized, without any need for
locking.

This is an excellent pattern (it was called "resource manager" in QNX, I don't
know of the GoF name for it if there is one) to protect a single, serialized
resource from trouble with multiple clients. The resource manager listens for
service requests for the resource, and mediates all such requests.

 

[bencoe]

This is an excellent pattern (it was called "resource manager" in QNX, I don't
know of the GoF name for it if there is one) to protect a single, serialized
resource from trouble with multiple clients. The resource manager listens for
service requests for the resource, and mediates all such requests.

I concur, I have had a lot of trouble trying to get file locks working
in a server environment (you can run into very nasty problems like a
file that doesn't unlock). Using a shared object to take care of IO
works, much, much better.

 

[Nigel Wade]

If you are accessing the files over NFS, then I believe (but am not
sure) that

- the locks may be unknown to the server, and consequently to other
NFS clients.

- each client might be caching its updates, which are not guaranteed
to be visible to other clients without a close/open sequence in
between (if even then). This could result in corruption even if the
locking is in fact working as you expect it to.

What happens when you run two instances of your application on the
*same* host?

I would recommend that you update the file from a single (server)
process that does so on behalf of the other (client) processes, so the
updates are atomic and strictly serialized, without any need for
locking.

For NFS rpc.lockd must be running on the server, it must be contactable using
RPC (i.e. the dynamic RPC port it is registered at must not be blocked by a
firewall), each client needs to co-operates by sending the lock request, and
waiting for the lock responses. Even then it might not work.

NFS was designed to be stateless, principally so that it could survive a server
re-boot or network drop-out. File locking is inherently stateful (hence the
need for the additional service rpc.lockd) and the two don't mix well.
Basically, don't attempt file locking over NFS unless you have plenty of time
to spare to debug it thoroughly. There are many failure modes and you should
not rely on it without rigorous testing of how you code reacts to each failure
mode.

 

[alejandrina]

For NFS rpc.lockd must be running on the server, it must be contactable using
RPC (i.e. the dynamic RPC port it is registered at must not be blocked by a
firewall), each client needs to co-operates by sending the lock request, and
waiting for the lock responses. Even then it might not work.

NFS was designed to be stateless, principally so that it could survive a server
re-boot or network drop-out. File locking is inherently stateful (hence the
need for the additional service rpc.lockd) and the two don't mix well.
Basically, don't attempt file locking over NFS unless you have plenty of time
to spare to debug it thoroughly. There are many failure modes and you should
not rely on it without rigorous testing of how you code reacts to each failure
mode.

--
Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : (e-mail address removed)
Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555

AHHHHHHHHHH. That's the kind of info we need! Thanks so much...I'll
continue this discussion with the sys admin.

 

[alejandrina]

...

"Advisory", in conjunction with locking, usually means that threads that
do not choose to play by the rules can go ahead and access the protected
resource regardless of the state of the locks.

If that is the intended interpretation, I don't think the quoted passage
helps with understanding your problem, because you have already checked
that your threads only access the resource while in possession of an
exclusive lock on it.

Have you tried calling force() after the write? Maybe there is some
buffering in the FileChannel that is delaying the effect of the write
past the release of the lock.

Patricia

That would completely negate the effect of the lock, wouldn't it?

 

[Patricia Shanahan]

That would completely negate the effect of the lock, wouldn't it?

No, the lock would still serve to serialize the periods during which a
thread is entitled to write to the file. The force is related to making
sure the write really happens between during that period.

However, given other posts I'm not sure this is worth trying until after
you have checked that the lock demon is running on the NFS server.

Patricia

 

[alejandrina]

AHHHHHHHHHH. That's the kind of info we need! Thanks so much...I'll
continue this discussion with the sys admin.

We were running Samba...When that went away, everything worked
perfectly.