Exclusive access to a file

Discussion in 'Java' started by fastcars, Feb 17, 2006.

  1. fastcars

    fastcars Guest

    I have a C program scanning a directory for files that get dumped there
    by a java program.

    What is happening is that the C program is picking these files up
    before they are completely written. They then fail to parse and cause
    data loss in the system. My intended solution is to exclusively lock
    the file that is being written to in the java program, so the C
    program will always get a complete file.


    The java program is executing the file copy as below.
    The C program below that again.

    The java program doesnt see the C programs file lock under linux
    And the C program doesnt see the java program file locks
    2.6.10-1.741_FC3 #1 Thu Jan 13 16:35:56 EST 2005 x86_64 x86_64 x86_64
    GNU/Linux

    And the C program doesnt see the java locks.
    Each small program below locks a file and waits for input from
    keyboard.

    The C program when run twice detects the lock in the second instance
    and exits as you would expect.
    I know the locking is advisory only on some platforms, however is linux
    one of these ?

    I cant find any docs saying this is the case or otherwise

    Any one any ideas?


    public static void copyFileWithLock(File from, File to,String
    errorLine) throws Exception {
    FileInputStream fis = null;
    FileChannel fc = null;
    Exception errt = null;
    MyByteBuffer mbb = new MyByteBuffer();

    try{
    fis = new FileInputStream(from);
    fc = new RandomAccessFile(to,"rw").getChannel();

    FileLock lock = null;
    try{
    System.out.println("size :" + fc.size());
    lock = fc.lock(0,fc.size(),false);
    byte [] data = new byte[1024*4];
    int read = 0;

    System.out.println("have lock, waiting");
    System.in.read();
    while ( (read=fis.read(data))>0)
    mbb.add(data,0,read);

    if(errorLine!=null && errorLine.length()>0){
    errorLine = "\n" + errorLine +"\n";
    mbb.add(errorLine.getBytes(),0, errorLine.length());
    }

    fc.write(mbb.getBuffer());
    } finally{
    if(lock!=null)
    lock.release();
    }
    }finally{
    try{
    if(fis!=null)
    fis.close();
    }catch(Exception err){
    errt = err;
    }
    if(fc!=null)
    fc.close();
    }

    if(errt!=null)
    throw errt;
    }



    #include <sys/types.h>
    #include <dirent.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    #include <unistd.h>
    #include <sys/file.h>
    #include <fcntl.h>



    int main(char **argc){

    printf("hello world\n");
    int fd = open("/tmp/nextto.ini", O_RDONLY);
    if(fd<0){
    printf("cannot open file\n");
    } else {


    if (flock(fd, LOCK_EX | LOCK_NB) < 0){
    printf("unable to lock file\n");
    } else {
    printf("locked ok \n");
    int waitChar = fgetc(stdin);
    }
    close(fd);
    }
    printf("exiting\n");

    }
     
    fastcars, Feb 17, 2006
    #1
    1. Advertising

  2. fastcars wrote:
    > I have a C program scanning a directory for files that get dumped there
    > by a java program.
    >
    > What is happening is that the C program is picking these files up
    > before they are completely written. They then fail to parse and cause
    > data loss in the system. My intended solution is to exclusively lock
    > the file that is being written to in the java program, so the C
    > program will always get a complete file.


    This is all fare too complicated for my liking.

    Just give the file a temporary name, e.g. file.temp or file.current
    while you write it, and after you finished writing rename the file to
    its final name. Let the reader only scan for files which don't end in
    '.temp' or '.current' or whatever.


    > if (flock(fd, LOCK_EX | LOCK_NB) < 0){


    You could try fcntl() or lockf() instead, but see above. I wouldn't use
    locking at all.

    /Thomas
    --
    The comp.lang.java.gui FAQ:
    ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
    http://www.uni-giessen.de/faq/archiv/computer-lang.java.gui.faq/
     
    Thomas Weidenfeller, Feb 17, 2006
    #2
    1. Advertising

  3. fastcars

    Roedy Green Guest

    On 16 Feb 2006 23:21:05 -0800, "fastcars" <>
    wrote, quoted or indirectly quoted someone who said :

    >What is happening is that the C program is picking these files up
    >before they are completely written. They then fail to parse and cause
    >data loss in the system. My intended solution is to exclusively lock
    >the file that is being written to in the java program, so the C
    >program will always get a complete file.


    I had a similar problem in the Replicator,
    http://mindprod.com/webstarts/replicator.html

    The way I handled it is I first write all the files. Then when I am
    sure they are all in place, I write a manifest, a tiny file listing
    the names of the files in the just-prepared batch. I write this in a
    single atomic IO. See http://mindprod.com/products1.html#HUNKIO

    It lists the files I want others to see. I am then free to add more
    files without worrying that someone will use the set before it is
    complete. They always access only files in the manifest.

    Similarly I can hide logically deleted files from new users without
    cutting off old users, by removing them from the manifest. I can
    delete them physically a day later.

    If you read the Replicator source code, look at the enum called
    ZDStatus which controls this. What I do is more elaborate than what
    you need since I have to deal with FTP upload and its flakiness.

    You might be interested in this too:
    http://mindprod.com/jgloss/smartftp.html
    about how to do atomic FTP uploads, which is roughly what you are
    doing without the FTP.






    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
     
    Roedy Green, Feb 17, 2006
    #3
  4. fastcars

    fastcars Guest

    I dont have a choice on the C part of the equation.

    However renaming sounds like a good option

    Thanks
     
    fastcars, Feb 17, 2006
    #4
  5. fastcars

    Nigel Wade Guest

    fastcars wrote:

    > I have a C program scanning a directory for files that get dumped there
    > by a java program.
    >
    > What is happening is that the C program is picking these files up
    > before they are completely written. They then fail to parse and cause
    > data loss in the system. My intended solution is to exclusively lock
    > the file that is being written to in the java program, so the C
    > program will always get a complete file.
    >
    >
    > The java program is executing the file copy as below.
    > The C program below that again.
    >
    > The java program doesnt see the C programs file lock under linux
    > And the C program doesnt see the java program file locks
    > 2.6.10-1.741_FC3 #1 Thu Jan 13 16:35:56 EST 2005 x86_64 x86_64 x86_64
    > GNU/Linux
    >
    > And the C program doesnt see the java locks.
    > Each small program below locks a file and waits for input from
    > keyboard.
    >
    > The C program when run twice detects the lock in the second instance
    > and exits as you would expect.
    > I know the locking is advisory only on some platforms, however is linux
    > one of these ?
    >
    > I cant find any docs saying this is the case or otherwise
    >
    > Any one any ideas?
    >


    There are multiple types of file locks in UNIX/Linux, the main ones being based
    on flock() and lockf(). Since the documentation for FileLock in Java doesn't
    specify which one is used you will need to employ a trial-and-error approach
    until you find the right one.

    I'd guess that Java has used the POSIX file locks (lockf) as these are generally
    preferred over other implementations. Also, the fact that there is a tryLock()
    method implies there is an ability to test if a lock is in place without
    blocking, and POSIX locks provide this.

    However, the approach suggested by Thomas (to use temporary filenames) is better
    than trying to rely on system dependent file locks.

    --
    Nigel Wade, System Administrator, Space Plasma Physics Group,
    University of Leicester, Leicester, LE1 7RH, UK
    E-mail :
    Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
     
    Nigel Wade, Feb 17, 2006
    #5
  6. fastcars

    James McGill Guest

    On Thu, 2006-02-16 at 23:21 -0800, fastcars wrote:
    >
    > What is happening is that the C program is picking these files up
    > before they are completely written. They then fail to parse and cause
    > data loss in the system. My intended solution is to exclusively lock
    > the file that is being written to in the java program, so the C
    > program will always get a complete file.


    You need to put the file in place with an atomic operation, unless you
    can work out some reliable locking semantics, some form of IPC, or
    something in the file format itself to indicate content length or EOF.

    You have to do *something*, or else it's in the category of the Halting
    Problem -- unsolvable.

    Writing to a temp file first, and then moving it to the final location,
    will work, if it's an atomic operation on your filesystem. It might not
    be, on NFS for example!

    I'd call it a design flaw in your C that it has no means of verifying
    the integrity of its input, and doesn't make any attempt at flow
    control, and just assumes that because a file is created, it isn't being
    written to (or, worse, being randomly accessed and written to by
    multiple processes!)

    Instead of file IO you could make the C program listen on a socket and
    stream the data that way. Or you could signal the C program with some
    other file, perhaps providing an MD5 sum as well so the recipient has
    some verification of the data. Or you could end the file with some
    string that's guaranteed not to be in the data, but is guaranteed to be
    at the end of the file.

    If you can put constraints on the amount of time allowed to write the
    file, you can have the C program wait this long in a nonblocking read
    loop, and agree to continue if that amount of time has passed. This can
    be reasonable if you're waiting for scp or writing files that are always
    close to a certain size.

    Since the 1.4 JDK there's been a java interface to the OS-native locking
    semantics. Whether it works for you depends on your OS, the locking
    implementation of your C libs, and your filesystem. In practice, on a
    real OS, it's essentially a JNI to fcntl(), which should work nicely
    between the JVM and C programs, even on decent NFS implementations.

    http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileLock.html


    James
     
    James McGill, Feb 17, 2006
    #6
  7. fastcars

    Oliver Wong Guest

    "Nigel Wade" <> wrote in message
    news:dt4h7v$2fj$...
    >
    > There are multiple types of file locks in UNIX/Linux, the main ones being
    > based
    > on flock() and lockf(). Since the documentation for FileLock in Java
    > doesn't
    > specify which one is used you will need to employ a trial-and-error
    > approach
    > until you find the right one.


    Note though that if you do this, you're depending on undocumented
    behaviour, which is dangerous because it's subject to change with newer
    releases of Java. If you decided to go this route, at least document the
    fact somewhere, so that 5, 10 or 20 years down the road, when some
    maintenance programmer (might be you!) gets called in at 3:00AM because the
    program mysteriously broke, you'll have some clues as to where to start
    looking. E.g. Did the machine recently auto-update its copy of Java?

    - Oliver
     
    Oliver Wong, Feb 17, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. MSDN
    Replies:
    0
    Views:
    686
  2. Vijay
    Replies:
    1
    Views:
    658
    Daniel
    May 13, 2005
  3. Steven D'Aprano
    Replies:
    3
    Views:
    1,857
    Jeffrey Straszheim
    Dec 6, 2008
  4. bino_oetomo

    Non exclusive file access

    bino_oetomo, Apr 19, 2007, in forum: Ruby
    Replies:
    1
    Views:
    117
    Phillip Gawlowski
    Apr 19, 2007
  5. Yash

    Exclusive file access

    Yash, Sep 22, 2004, in forum: Perl Misc
    Replies:
    0
    Views:
    113
Loading...

Share This Page