Exclusive access to a file

[fastcars]

I have a C program scanning a directory for files that get dumped there
by a java program.

What is happening is that the C program is picking these files up
before they are completely written. They then fail to parse and cause
data loss in the system. My intended solution is to exclusively lock
the file that is being written to in the java program, so the C
program will always get a complete file.


The java program is executing the file copy as below.
The C program below that again.

The java program doesnt see the C programs file lock under linux
And the C program doesnt see the java program file locks
2.6.10-1.741_FC3 #1 Thu Jan 13 16:35:56 EST 2005 x86_64 x86_64 x86_64
GNU/Linux

And the C program doesnt see the java locks.
Each small program below locks a file and waits for input from
keyboard.

The C program when run twice detects the lock in the second instance
and exits as you would expect.
I know the locking is advisory only on some platforms, however is linux
one of these ?

I cant find any docs saying this is the case or otherwise

Any one any ideas?


public static void copyFileWithLock(File from, File to,String
errorLine) throws Exception {
FileInputStream fis = null;
FileChannel fc = null;
Exception errt = null;
MyByteBuffer mbb = new MyByteBuffer();

try{
fis = new FileInputStream(from);
fc = new RandomAccessFile(to,"rw").getChannel();

FileLock lock = null;
try{
System.out.println("size :" + fc.size());
lock = fc.lock(0,fc.size(),false);
byte [] data = new byte[1024*4];
int read = 0;

System.out.println("have lock, waiting");
System.in.read();
while ( (read=fis.read(data))>0)
mbb.add(data,0,read);

if(errorLine!=null && errorLine.length()>0){
errorLine = "\n" + errorLine +"\n";
mbb.add(errorLine.getBytes(),0, errorLine.length());
}

fc.write(mbb.getBuffer());
} finally{
if(lock!=null)
lock.release();
}
}finally{
try{
if(fis!=null)
fis.close();
}catch(Exception err){
errt = err;
}
if(fc!=null)
fc.close();
}

if(errt!=null)
throw errt;
}



#include <sys/types.h>
#include <dirent.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/file.h>
#include <fcntl.h>



int main(char **argc){

printf("hello world\n");
int fd = open("/tmp/nextto.ini", O_RDONLY);
if(fd<0){
printf("cannot open file\n");
} else {


if (flock(fd, LOCK_EX | LOCK_NB) < 0){
printf("unable to lock file\n");
} else {
printf("locked ok \n");
int waitChar = fgetc(stdin);
}
close(fd);
}
printf("exiting\n");

}

 

[Thomas Weidenfeller]

I have a C program scanning a directory for files that get dumped there
by a java program.

What is happening is that the C program is picking these files up
before they are completely written. They then fail to parse and cause
data loss in the system. My intended solution is to exclusively lock
the file that is being written to in the java program, so the C
program will always get a complete file.

This is all fare too complicated for my liking.

Just give the file a temporary name, e.g. file.temp or file.current
while you write it, and after you finished writing rename the file to
its final name. Let the reader only scan for files which don't end in
'.temp' or '.current' or whatever.

if (flock(fd, LOCK_EX | LOCK_NB) < 0){

You could try fcntl() or lockf() instead, but see above. I wouldn't use
locking at all.

/Thomas

 

[Roedy Green]

What is happening is that the C program is picking these files up
before they are completely written. They then fail to parse and cause
data loss in the system. My intended solution is to exclusively lock
the file that is being written to in the java program, so the C
program will always get a complete file.

I had a similar problem in the Replicator,
http://mindprod.com/webstarts/replicator.html

The way I handled it is I first write all the files. Then when I am
sure they are all in place, I write a manifest, a tiny file listing
the names of the files in the just-prepared batch. I write this in a
single atomic IO. See http://mindprod.com/products1.html#HUNKIO

It lists the files I want others to see. I am then free to add more
files without worrying that someone will use the set before it is
complete. They always access only files in the manifest.

Similarly I can hide logically deleted files from new users without
cutting off old users, by removing them from the manifest. I can
delete them physically a day later.

If you read the Replicator source code, look at the enum called
ZDStatus which controls this. What I do is more elaborate than what
you need since I have to deal with FTP upload and its flakiness.

You might be interested in this too:
http://mindprod.com/jgloss/smartftp.html
about how to do atomic FTP uploads, which is roughly what you are
doing without the FTP.

 

[fastcars]

I dont have a choice on the C part of the equation.

However renaming sounds like a good option

Thanks

 

[Nigel Wade]

I have a C program scanning a directory for files that get dumped there
by a java program.

What is happening is that the C program is picking these files up
before they are completely written. They then fail to parse and cause
data loss in the system. My intended solution is to exclusively lock
the file that is being written to in the java program, so the C
program will always get a complete file.


The java program is executing the file copy as below.
The C program below that again.

The java program doesnt see the C programs file lock under linux
And the C program doesnt see the java program file locks
2.6.10-1.741_FC3 #1 Thu Jan 13 16:35:56 EST 2005 x86_64 x86_64 x86_64
GNU/Linux

And the C program doesnt see the java locks.
Each small program below locks a file and waits for input from
keyboard.

The C program when run twice detects the lock in the second instance
and exits as you would expect.
I know the locking is advisory only on some platforms, however is linux
one of these ?

I cant find any docs saying this is the case or otherwise

Any one any ideas?

There are multiple types of file locks in UNIX/Linux, the main ones being based
on flock() and lockf(). Since the documentation for FileLock in Java doesn't
specify which one is used you will need to employ a trial-and-error approach
until you find the right one.

I'd guess that Java has used the POSIX file locks (lockf) as these are generally
preferred over other implementations. Also, the fact that there is a tryLock()
method implies there is an ability to test if a lock is in place without
blocking, and POSIX locks provide this.

However, the approach suggested by Thomas (to use temporary filenames) is better
than trying to rely on system dependent file locks.

 

[James McGill]

What is happening is that the C program is picking these files up
before they are completely written. They then fail to parse and cause
data loss in the system. My intended solution is to exclusively lock
the file that is being written to in the java program, so the C
program will always get a complete file.

You need to put the file in place with an atomic operation, unless you
can work out some reliable locking semantics, some form of IPC, or
something in the file format itself to indicate content length or EOF.

You have to do *something*, or else it's in the category of the Halting
Problem -- unsolvable.

Writing to a temp file first, and then moving it to the final location,
will work, if it's an atomic operation on your filesystem. It might not
be, on NFS for example!

I'd call it a design flaw in your C that it has no means of verifying
the integrity of its input, and doesn't make any attempt at flow
control, and just assumes that because a file is created, it isn't being
written to (or, worse, being randomly accessed and written to by
multiple processes!)

Instead of file IO you could make the C program listen on a socket and
stream the data that way. Or you could signal the C program with some
other file, perhaps providing an MD5 sum as well so the recipient has
some verification of the data. Or you could end the file with some
string that's guaranteed not to be in the data, but is guaranteed to be
at the end of the file.

If you can put constraints on the amount of time allowed to write the
file, you can have the C program wait this long in a nonblocking read
loop, and agree to continue if that amount of time has passed. This can
be reasonable if you're waiting for scp or writing files that are always
close to a certain size.

Since the 1.4 JDK there's been a java interface to the OS-native locking
semantics. Whether it works for you depends on your OS, the locking
implementation of your C libs, and your filesystem. In practice, on a
real OS, it's essentially a JNI to fcntl(), which should work nicely
between the JVM and C programs, even on decent NFS implementations.

http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileLock.html


James

 

[Oliver Wong]

There are multiple types of file locks in UNIX/Linux, the main ones being
based
on flock() and lockf(). Since the documentation for FileLock in Java
doesn't
specify which one is used you will need to employ a trial-and-error
approach
until you find the right one.

Note though that if you do this, you're depending on undocumented
behaviour, which is dangerous because it's subject to change with newer
releases of Java. If you decided to go this route, at least document the
fact somewhere, so that 5, 10 or 20 years down the road, when some
maintenance programmer (might be you!) gets called in at 3:00AM because the
program mysteriously broke, you'll have some clues as to where to start
looking. E.g. Did the machine recently auto-update its copy of Java?

- Oliver