Accessing file while it is being copied

[Chris]

My app processes files as they get dropped into a directory. There's a
loop that keeps checking if a file is available, and if it is, then the
file gets read.

The trouble is that if the app detects a new file before it has been
fully copied from another location, and it starts processing, then I get
errors. Somehow, I need to be able to detect when the copy process has
completed.

I tried to test for an exclusive file lock using NIO, but it did not
work. In order to lock a file, you have to open it first, and I get a
FileNotFoundException when I create a FileInputStream on the file.

java.io.FileNotFoundException: C:\mydir\product.xml (The process cannot
access the file because it is being used by another process)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)

How can I know when the copy process is complete. (Also, this has to
work on both Windows and FreeBSD).

 

[opalpa opalpa]

Possible solutions:
A) make a second directory for links to files
D) create a sentinel file for every file copied in; that is say file
getting copied in is named "lots_o_data.text" have the process
performing copy put in another file named something like
"lots_o_data.text.finished".



opalpa
(e-mail address removed)
http://opalpa.info/

 

[Eric Sosman]

opalpa (e-mail address removed) http://opalpa.info wrote On 03/05/07 16:13,:

Possible solutions:
A) make a second directory for links to files
D) create a sentinel file for every file copied in; that is say file
getting copied in is named "lots_o_data.text" have the process
performing copy put in another file named something like
"lots_o_data.text.finished".

G) Copy the file using the name "lots_o_data.temporary"
and after copying rename it to "lots_o_data.text". Teach
your program to ignore files named "....temporary".

 

[Knute Johnson]

My app processes files as they get dropped into a directory. There's a
loop that keeps checking if a file is available, and if it is, then the
file gets read.

The trouble is that if the app detects a new file before it has been
fully copied from another location, and it starts processing, then I get
errors. Somehow, I need to be able to detect when the copy process has
completed.

I tried to test for an exclusive file lock using NIO, but it did not
work. In order to lock a file, you have to open it first, and I get a
FileNotFoundException when I create a FileInputStream on the file.

java.io.FileNotFoundException: C:\mydir\product.xml (The process cannot
access the file because it is being used by another process)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)

How can I know when the copy process is complete. (Also, this has to
work on both Windows and FreeBSD).

It stops throwing the exception. What's the problem here? If it throws
this exception just wait for a few seconds and try again.

 

[Chris]

It stops throwing the exception. What's the problem here? If it throws
this exception just wait for a few seconds and try again.

Because it's almost always a bad idea to have exceptions thrown in the
normal course of execution. Exceptions are just that -- unusual cases.
Besides, there's no guarantee that this method will work in a
cross-platform way. Maybe a particular operation system doesn't throw
the exception. Then my app would start processing a partially-copied
file, leading to all kinds of problems.

 

[opalpa opalpa]

I believe BSD, being a unix, would not throw the exception.

As for throwing exceptions, I've personally changed my mind on
throwing exceptions in course of execution. I used to treat exception
as a synonym for error. Currently I occasionally treat exceptions as
a control flow mechanism.


opalpa
(e-mail address removed)
http://opalpa.info/

 

[Knute Johnson]

Because it's almost always a bad idea to have exceptions thrown in the
normal course of execution. Exceptions are just that -- unusual cases.

I think an incomplete file would be an unusual situation, it's certainly
not what you want to find.

Besides, there's no guarantee that this method will work in a
cross-platform way. Maybe a particular operation system doesn't throw
the exception. Then my app would start processing a partially-copied
file, leading to all kinds of problems.

I think you're toast. The only other idea I have is to check
periodically to see if the file size stops changing. But I'm not sure
that is good either because I have seen file systems that set the length
of the file before completing the write. You might try reading it again
and again until it's checksum becomes stable.

You might just try using both the exception and the lock. See if that
works on most OS.

 

[opalpa opalpa]

I believe BSD, being a unix, would not throw the exception.

The possible absence of this exception is another reason to keep the
operation of coping the files and the operation of processing the
files seperate in sequence. If the format of the file permits a
subsequence to be a valid file then reading a file too early could
chop of information.

---

I'm suprised to learn FileNotFoundException covers inaccseibility to
file. I would have guessed that IOException gets thrown.

opalpa
(e-mail address removed)
http://opalpa.info/

 

[Chris Uppal]

opalpa (e-mail address removed) http://opalpa.info wrote On 03/05/07 16:13,:

G) Copy the file using the name "lots_o_data.temporary"
and after copying rename it to "lots_o_data.text". Teach
your program to ignore files named "....temporary".

J) Use a self-delimiting format for the file contents (checksum, end-of-file
marker, anything which comes "for free" with the internal file format (such as
well-formdedness), MIME-style encapsulation of the "real" contents...)

-- chris