determing if FTP process is finished

[Brock Heinz]

Hi All,

I have a pretty interesting problem. First I'll explain my scenario,
and then I'll pose my question.

I've written a TimerTask that regularly checks for a file on my FTP
server. In each cycle the task must determine 1. If the file exists
and then 2. if the file isn't currently being written to by an FTP
process. I want to ensure that I only act upon files that have been
fully transfered to my FTP server.

The first part is simple - I can easily determine if the file is
present. The second part though is causing me headaches. How can I
know if the FTP process is done? Does anyone have any experience with
this? Is it very platform / process (FTP in this case) dependent?

Here is what I was thinking, but my tests are not turning anything
positive up...

private boolean fileAvailable(File file) {
//ensure that the file exists
if (file.exists()) {
FileChannel fc = new RandomAccessFile(file, "rw").getChannel();
//attempt a lock
try {
//this is an exclusive lock
fc.lock();
} catch (Throwable t) {
//ftp still has file locked...
return false;
}
//able to lock file - release
fc.close();
return true;
} else {
//no file
return false;
}
}

Thanks,
Brock

 

[Paul Lutus]

Hi All,

I have a pretty interesting problem. First I'll explain my scenario,
and then I'll pose my question.

I've written a TimerTask that regularly checks for a file on my FTP
server. In each cycle the task must determine 1. If the file exists
and then 2. if the file isn't currently being written to by an FTP
process. I want to ensure that I only act upon files that have been
fully transfered to my FTP server.

The first part is simple - I can easily determine if the file is
present. The second part though is causing me headaches. How can I
know if the FTP process is done?

Very simply, without (a) knowing in advance the length of the file, or (b)
somehow inquiring with the FTP client program directly, you cannot know for
sure that the file transfer is complete. This is because an FTP transfer
might be delayed unreasonably, or even get interrupted for a day or longer
when accessing a busy FTP server.

Period, end of story.

 

[Sudsy]

Brock Heinz wrote:

The first part is simple - I can easily determine if the file is
present. The second part though is causing me headaches. How can I
know if the FTP process is done? Does anyone have any experience with
this? Is it very platform / process (FTP in this case) dependent?

If you'd searched the archives (you really should have) then you'd
have found a fairly common approach. It consists of uploading to a
file with a particular extension and then renaming the file when
upload completes. The consumer knows not to touch files with the
known extension since those transfers are in-progress.
There should also be a reaper to remove partial transfers after a
reasonable period of time has elapsed.

 

[Brock Heinz]

Brock Heinz wrote:


If you'd searched the archives (you really should have) then you'd
have found a fairly common approach.

Yes, Sudsy - I did read the archives. The double file approach isn't
an option in my environment. With FileChannel(s) being new to JDK
1.4, a good deal of the archives don't pertain.... I was hoping I
could solve this problem programatically.

It consists of uploading to a
file with a particular extension and then renaming the file when
upload completes. The consumer knows not to touch files with the
known extension since those transfers are in-progress.
There should also be a reaper to remove partial transfers after a
reasonable period of time has elapsed.

-Brock

 

[Gordon Beaton]

The first part is simple - I can easily determine if the file is
present. The second part though is causing me headaches. How can I
know if the FTP process is done? Does anyone have any experience with
this? Is it very platform / process (FTP in this case) dependent?

Any solution to your problem will likely be platform dependent, but
unfortunately you didn't specify your platform.

Here is what I was thinking, but my tests are not turning anything
positive up...
[...]

if (file.exists()) {
FileChannel fc = new RandomAccessFile(file, "rw").getChannel();
//attempt a lock

Unless the server is also locking the file (and I see no reason why it
should be) then you should successfully acquire the lock every time.

If your ftp server is running on some flavour of unix and your
application is running on the same host, then you might be able to
determine the status of a file by using (for example) the process file
system or a tool like lsof, both of which can tell you about files
currently open by other processes.

Some ftp servers use setproctitle() (or some equivalent mechanism) to
write information to the command line shown by ps, allowing you to see
e.g. the status of each connection (IDLE, RETR foo.tar, etc).

In general though, it's hard to guess what another application is up
to when it wasn't designed to share that information, so solutions
like the above are more or less hacks.

IMO the cleanest and most correct solution is to implement an ftp
server into your application, or modify an existing one to notify you
when an upload is complete.

/gordon