Accessing file while it is being copied

Discussion in 'Java' started by Chris, Mar 5, 2007.

  1. Chris

    Chris Guest

    My app processes files as they get dropped into a directory. There's a
    loop that keeps checking if a file is available, and if it is, then the
    file gets read.

    The trouble is that if the app detects a new file before it has been
    fully copied from another location, and it starts processing, then I get
    errors. Somehow, I need to be able to detect when the copy process has
    completed.

    I tried to test for an exclusive file lock using NIO, but it did not
    work. In order to lock a file, you have to open it first, and I get a
    FileNotFoundException when I create a FileInputStream on the file.

    java.io.FileNotFoundException: C:\mydir\product.xml (The process cannot
    access the file because it is being used by another process)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)

    How can I know when the copy process is complete. (Also, this has to
    work on both Windows and FreeBSD).
    Chris, Mar 5, 2007
    #1
    1. Advertising

  2. Possible solutions:
    A) make a second directory for links to files
    D) create a sentinel file for every file copied in; that is say file
    getting copied in is named "lots_o_data.text" have the process
    performing copy put in another file named something like
    "lots_o_data.text.finished".



    opalpa

    http://opalpa.info/
    opalpa http://opalpa.info, Mar 5, 2007
    #2
    1. Advertising

  3. Chris

    Eric Sosman Guest

    opalpa http://opalpa.info wrote On 03/05/07 16:13,:
    > Possible solutions:
    > A) make a second directory for links to files
    > D) create a sentinel file for every file copied in; that is say file
    > getting copied in is named "lots_o_data.text" have the process
    > performing copy put in another file named something like
    > "lots_o_data.text.finished".


    G) Copy the file using the name "lots_o_data.temporary"
    and after copying rename it to "lots_o_data.text". Teach
    your program to ignore files named "....temporary".

    --
    Eric Sosman, Mar 5, 2007
    #3
  4. Chris wrote:
    > My app processes files as they get dropped into a directory. There's a
    > loop that keeps checking if a file is available, and if it is, then the
    > file gets read.
    >
    > The trouble is that if the app detects a new file before it has been
    > fully copied from another location, and it starts processing, then I get
    > errors. Somehow, I need to be able to detect when the copy process has
    > completed.
    >
    > I tried to test for an exclusive file lock using NIO, but it did not
    > work. In order to lock a file, you have to open it first, and I get a
    > FileNotFoundException when I create a FileInputStream on the file.
    >
    > java.io.FileNotFoundException: C:\mydir\product.xml (The process cannot
    > access the file because it is being used by another process)
    > at java.io.FileInputStream.open(Native Method)
    > at java.io.FileInputStream.<init>(FileInputStream.java:106)
    >
    > How can I know when the copy process is complete. (Also, this has to
    > work on both Windows and FreeBSD).


    It stops throwing the exception. What's the problem here? If it throws
    this exception just wait for a few seconds and try again.

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Mar 5, 2007
    #4
  5. Chris

    Chris Guest

    Knute Johnson wrote:
    > Chris wrote:
    >> My app processes files as they get dropped into a directory. There's a
    >> loop that keeps checking if a file is available, and if it is, then
    >> the file gets read.
    >>
    >> The trouble is that if the app detects a new file before it has been
    >> fully copied from another location, and it starts processing, then I
    >> get errors. Somehow, I need to be able to detect when the copy process
    >> has completed.
    >>
    >> I tried to test for an exclusive file lock using NIO, but it did not
    >> work. In order to lock a file, you have to open it first, and I get a
    >> FileNotFoundException when I create a FileInputStream on the file.
    >>
    >> java.io.FileNotFoundException: C:\mydir\product.xml (The process
    >> cannot access the file because it is being used by another process)
    >> at java.io.FileInputStream.open(Native Method)
    >> at java.io.FileInputStream.<init>(FileInputStream.java:106)
    >>
    >> How can I know when the copy process is complete. (Also, this has to
    >> work on both Windows and FreeBSD).

    >
    > It stops throwing the exception. What's the problem here? If it throws
    > this exception just wait for a few seconds and try again.
    >


    Because it's almost always a bad idea to have exceptions thrown in the
    normal course of execution. Exceptions are just that -- unusual cases.
    Besides, there's no guarantee that this method will work in a
    cross-platform way. Maybe a particular operation system doesn't throw
    the exception. Then my app would start processing a partially-copied
    file, leading to all kinds of problems.
    Chris, Mar 6, 2007
    #5
  6. I believe BSD, being a unix, would not throw the exception.

    As for throwing exceptions, I've personally changed my mind on
    throwing exceptions in course of execution. I used to treat exception
    as a synonym for error. Currently I occasionally treat exceptions as
    a control flow mechanism.


    opalpa

    http://opalpa.info/
    opalpa http://opalpa.info, Mar 6, 2007
    #6
  7. Chris wrote:
    > Knute Johnson wrote:
    >> Chris wrote:
    >>> My app processes files as they get dropped into a directory. There's
    >>> a loop that keeps checking if a file is available, and if it is, then
    >>> the file gets read.
    >>>
    >>> The trouble is that if the app detects a new file before it has been
    >>> fully copied from another location, and it starts processing, then I
    >>> get errors. Somehow, I need to be able to detect when the copy
    >>> process has completed.
    >>>
    >>> I tried to test for an exclusive file lock using NIO, but it did not
    >>> work. In order to lock a file, you have to open it first, and I get a
    >>> FileNotFoundException when I create a FileInputStream on the file.
    >>>
    >>> java.io.FileNotFoundException: C:\mydir\product.xml (The process
    >>> cannot access the file because it is being used by another process)
    >>> at java.io.FileInputStream.open(Native Method)
    >>> at java.io.FileInputStream.<init>(FileInputStream.java:106)
    >>>
    >>> How can I know when the copy process is complete. (Also, this has to
    >>> work on both Windows and FreeBSD).

    >>
    >> It stops throwing the exception. What's the problem here? If it
    >> throws this exception just wait for a few seconds and try again.
    >>

    >
    > Because it's almost always a bad idea to have exceptions thrown in the
    > normal course of execution. Exceptions are just that -- unusual cases.


    I think an incomplete file would be an unusual situation, it's certainly
    not what you want to find.

    > Besides, there's no guarantee that this method will work in a
    > cross-platform way. Maybe a particular operation system doesn't throw
    > the exception. Then my app would start processing a partially-copied
    > file, leading to all kinds of problems.


    I think you're toast. The only other idea I have is to check
    periodically to see if the file size stops changing. But I'm not sure
    that is good either because I have seen file systems that set the length
    of the file before completing the write. You might try reading it again
    and again until it's checksum becomes stable.

    You might just try using both the exception and the lock. See if that
    works on most OS.

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Mar 6, 2007
    #7
  8. > I believe BSD, being a unix, would not throw the exception.

    The possible absence of this exception is another reason to keep the
    operation of coping the files and the operation of processing the
    files seperate in sequence. If the format of the file permits a
    subsequence to be a valid file then reading a file too early could
    chop of information.

    ---

    I'm suprised to learn FileNotFoundException covers inaccseibility to
    file. I would have guessed that IOException gets thrown.

    opalpa

    http://opalpa.info/
    opalpa http://opalpa.info, Mar 6, 2007
    #8
  9. Chris

    Chris Uppal Guest

    Eric Sosman wrote:
    > opalpa http://opalpa.info wrote On 03/05/07 16:13,:
    > > Possible solutions:
    > > A) make a second directory for links to files
    > > D) create a sentinel file for every file copied in; that is say file
    > > getting copied in is named "lots_o_data.text" have the process
    > > performing copy put in another file named something like
    > > "lots_o_data.text.finished".

    >
    > G) Copy the file using the name "lots_o_data.temporary"
    > and after copying rename it to "lots_o_data.text". Teach
    > your program to ignore files named "....temporary".


    J) Use a self-delimiting format for the file contents (checksum, end-of-file
    marker, anything which comes "for free" with the internal file format (such as
    well-formdedness), MIME-style encapsulation of the "real" contents...)

    -- chris
    Chris Uppal, Mar 6, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. BH
    Replies:
    4
    Views:
    709
    sampsons
    Jul 17, 2003
  2. J Gao
    Replies:
    0
    Views:
    1,210
    J Gao
    Jul 28, 2003
  3. Hans Kesting
    Replies:
    0
    Views:
    1,170
    Hans Kesting
    Jan 5, 2004
  4. Hooyoo
    Replies:
    9
    Views:
    505
    Hooyoo
    Aug 28, 2007
  5. Paul Archer
    Replies:
    7
    Views:
    164
    Paul Archer
    Jul 18, 2009
Loading...

Share This Page