How to Monitor the uploading a file using java

[GMHK]

Hi,

How do we know when the file is completely uploaded or not, using Java?

The scenario is something like this: i have one folder, i have
listener to that folder which does constantly monitor that folder for
new files.

My program is working fine for smaller files that are loaded to that
folder.

Suppose, if a huge file is being added and if my program is invoked in
middle of that, it copies only the half the file instead, it was
supposed to copy the completely downloaded file.

can u tell me how to monitor whether the file is completely uplodaed or
not?

Thanks and Regards
HariKiran

 

[jan V]

can u tell me how to monitor whether the file is completely uplodaed or
not?

Is this a Zen question?

How are the uploads performed PRECISELY? What kinds of files are you
uploading. PRECISELY?

Don't you realise your OP contains too little hard information for any
reasonable answer to be formulated?

 

[GMHK]

Hi,
The scenaro is like this.
I dont have a control on how to upload the file into the directory, but
my listener program will listening to that directory, which is created
to store the uploaded files.
files can be uploaded by means of copying, a new file can be
created,etc..

 

[Ingo R. Homann]

Hi,

Hi,
The scenaro is like this.
I dont have a control on how to upload the file into the directory, but
my listener program will listening to that directory, which is created
to store the uploaded files.
files can be uploaded by means of copying, a new file can be
created,etc..

So, if I understand you correctly, there are two programs: One program
(that you cant modify) puts files into a directory. Another program
should realize if the file is complete or not.

In general: I don't think there is a 100% safe solution that works on
every OS.

Some ideas:

(A) JNI
(B) Something with java.nio.*? (Requires at least Java 1.4)
(C) Reading the size of the file periodically and see if it does not
change any longer for some seconds (or minutes?)
(D) Read the files contents and see if it is not currupt (of course only
possible for certain file formats)
(E) Trying to delete or rename the file. If it works, it was complete
(Does not work on many OSs)

Hth,
Ingo

 

[jan V]

I dont have a control on how to upload the file into the directory, but

my listener program will listening to that directory, which is created
to store the uploaded files.
files can be uploaded by means of copying, a new file can be
created,etc..

OK, so you don't have control over how the files get there... but what about
the file types? I asked you what file types are involved, but you didn't
answer this key question. Some techniques may only be viable if known types
are involved (so that existing decoders can see if the whole stream is
present or not). If you also don't know the types involved, then can you
please explain what the dickens you're trying to do, because it sounds very
mysterious...

 

[Hemal Pandya]

Hi,
The scenaro is like this.
I dont have a control on how to upload the file into the directory, but
my listener program will listening to that directory, which is created
to store the uploaded files.
files can be uploaded by means of copying, a new file can be
created,etc..

If you know that the uploading application keeps the file open while
uploading, you can use OS specific methods to identify if a file is
open by any process (there might be permission issues).

Otherwise, I suspect it is hard to devise a general solution to
identify if a file has been completely uploaded. You could probably use
Jan's approach of testing file data integrity.

Also consider storing last sync time and current modification date on
the files. But with his approach you /will/ copy incomplete files.

 

[jan V]

Hey HariKiran, maybe there;s a non-programmatic solution to your problem:

IF

a) the uploading program can be configured to upload the same files to
another destination
b) you can configure that program to also upload the files to your imaginary
Java program's destination folder

... am I missing something? Maybe bandwidth/delay considerations would make
this approach too slow/expensive.

 

[Gordon Beaton]

How do we know when the file is completely uploaded or not, using Java?

Basically you have two choices: educated guesses, or help from the
uploading process itself.

If you can, have the uploading process upload to a different name (an
additional suffix will do) or different directory. It can then rename
the finished file to the final destination (an atomic operation) where
your application detects it and can safely assume that it is complete.

/gordon

 

[Stefan Schulz]

If you can, have the uploading process upload to a different name (an
additional suffix will do) or different directory. It can then rename
the finished file to the final destination (an atomic operation) where
your application detects it and can safely assume that it is complete.

This is only atomic if you do not cross filesystem boundaries. Therefore,
a different suffix might be the best guess.

 

[Roedy Green]

can u tell me how to monitor whether the file is completely uplodaed or
not?

If Java is doing the download, it often knows the size to start, and
it knows how many bytes it has written to disk.

see http://mindprod.com/products1.html#FILETRANSFER
http://mindprod.com/applets/fileio.html

 

[Roedy Green]

I dont have a control on how to upload the file into the directory, but
my listener program will listening to that directory, which is created
to store the uploaded files.

You have to determine yourself independently how long the files SHOULD
be. If the files are zip files, you can check they were fully
downloaded by checking zip integrity.

You can monitor how much has happened in the last X seconds by doing a
File.list() and using the File.length() method.

See http://mindprod.com/jgloss/file.html
http://mindprod.com/jgloss/timer.html
http://mindprod.com/jgloss/zip.html

 

[Roedy Green]

Otherwise, I suspect it is hard to devise a general solution to
identify if a file has been completely uploaded.

The problem is that a file that was aborted part way through download,
and one that was complete have no special distinguishing mark. The OS
does not track file integrity. Some formats, e.g. zip, have a way of
telling, but not in general.

I am assuming you mean downloaded, not uploaded.

If you truly meant uploaded, you have a much more difficult problem.

How far through the upload is, is totally private to the other app.
The best you could do is see if the file was in use, but not how much
was uploaded. There is nothing in the files to give a hint.

If you were absolutely desperate to solve the problem, here are some
approaches:

1. disassemble the other program and add monitoring.

2. write your own file uploader. Give the other program dummy work to
do if necessary. (This is the easiest of the three approaches). File
upload is not rocket science, even FTP upload.)

3. Write a TCP/IP monitor and snoop on the traffic out of the other
program. Collect stats.

4. Hire me to solve it for you.

 

[jan V]

3. Write a TCP/IP monitor and snoop on the traffic out of the other

program. Collect stats.

**write** a TCP/IP monitor? Not in Java you won't. In any event, what would
be the point when there's plenty of packet sniffers etcc.. out there
already?

 

[gerrards8]

If you can, have the uploading process upload to a different name (an
additional suffix will do) or different directory. It can then rename
the finished file to the final destination (an atomic operation) where
your application detects it and can safely assume that it is complete.

/gordon

That's exactly right.

If the client doesn't have the ability to rename the file, then upload
another empty file with a different extension as an indicator that the
upload task is now complete. This will be the file you want to monitor
(filter your monitor on this 2nd extension).

If none of the above options are feasible, then you would have no option
but to monitor the size of the file, and determine an acceptable span of
time where if the size doesn't change (within that time period), then a
file uploaded event is triggered. (this is *very* ugly, but works if you
trust your system/network environment, specially if the constant size
time span is large, which would slow down your server's response).

 

[Roedy Green]

can u tell me how to monitor whether the file is completely uplodaed or
not?

Another possible approach if you have control over the code accepting
the uploads is to ask it for progress.