File locking

[MadDogMcGee]

Hi,

I want to process the contents of a file, but only when it's copied across
to the work folder. Some of the files take a few seconds to bring across.
How do I detect that the file is complete and ready for processing?

Thanks.

 

[A. Sinan Unur]

I want to process the contents of a file, but only when it's copied
across to the work folder. Some of the files take a few seconds to
bring across. How do I detect that the file is complete and ready for
processing?

Please note that you are asking an operating system specific question (I am
assuming that the copying of the file is being done outside of your
program. Otherwise the answer would to wait until the copying is done).

Based on your other posts and newsreader, I am assuming you are on a Win32
machine. In this case, you might want to utilize the Windows API to wait
for the file. See http://tinyurl.com/f6a5

Sinan.

 

[Brian McCauley]

I want to process the contents of a file, but only when it's copied across
to the work folder. Some of the files take a few seconds to bring across.
How do I detect that the file is complete and ready for processing?

If the other process is holding a lock on the file (either explicitly or
implicitly as a feature of the OS) you can try locking it.

However the more portable, reliable and widely used approach is to copy
the file using a temporary name then rename it to its final name.

None of ghis, of course, has anything to do with Perl.

If your OS provides a flock(2) like interface to lock files then you can
use the Perl flock() function the access it.

 

[Lawson Hanson]

Hi,

I want to process the contents of a file, but only when it's copied across
to the work folder. Some of the files take a few seconds to bring across.
How do I detect that the file is complete and ready for processing?

Thanks.

One way might be to do something like:

1. Obtain the size (byte count) of the source file
2. Obtain the MD5 message digest checksum
of the source file
3. Copy the file from the source to the target
4. Check the size of the target file every so often
5. When the target file has the same byte count as
the source file, then obtain the MD5 message
digest checksum of the target file
6. Compare the MD5 message digest checksums of the
two files, and proceed only if they are equal.

That might be a bit of overkill, but it should enable
a fairly robust solution to the problem.

Regards,

Lawson

 

[MadDogMcGee]

I seem to have solved my problem:


$file = "test.iso";

while (1) {

if (open(FILE, "< $file")) {
print "Transfer completed\n";
close (FILE);

}
else {
print "Not ready\n";
}

}



Is there anything undesirable about this approach?

Notes: I'm on an NT based system, and the file is being copied by a
different program. I don't want to work on the file with Perl until it's
finished being copied across.

Hi,

I want to process the contents of a file, but only when it's copied across

to the work folder. Some of the files take a few seconds to bring across.

How do I detect that the file is complete and ready for processing?

 

[Tintin]

I seem to have solved my problem:


$file = "test.iso";

while (1) {

if (open(FILE, "< $file")) {
print "Transfer completed\n";
close (FILE);

}
else {
print "Not ready\n";
}

}



Is there anything undesirable about this approach?

Did you intend on having an infinite loop?

Your test for file existence, not copy completion (unless NTFS buffers
everything before copying)

 

[MadDogMcGee]

Did you intend on having an infinite loop?

Your test for file existence, not copy completion (unless NTFS buffers
everything before copying)

Yes, I meant it to be an infinite loop, but just as an example of using open
for determining copy completion. I will write the actual code tomorrow,
during work hours. I tested the above on Windows XP and it worked the way I
wanted.

 

[A. Sinan Unur]

I tested the above on Windows XP and it worked the way I wanted.

This is a very tight loop with an expensive system call in it. If you are
not going to use OS facilities to wait for completeion, you should at
least sleep a second between calls to avoid hogging the CPU.

 

[Joe Smith]

Your test for file existence, not copy completion (unless NTFS buffers
everything before copying)

The Windows operating system not only has a mandatory write lock, it
has a mandatory read lock. Opening a file for reading will fail
if the process doing the writing has not yet closed the file.

 

[Brian McCauley]

[ Please don't post TOFU. It is considered rude an will predispose
people not to want to help you. ]

I seem to have solved my problem:

$file = "test.iso";

while (1) {

if (open(FILE, "< $file")) {
print "Transfer completed\n";
close (FILE);

}
else {
print "Not ready\n";
}

}



Is there anything undesirable about this approach?

It's a busy loop. Busy loops are undesirable on multi-taking system.
You should pause for a non-zero time between unsucessful opens.

Notes: I'm on an NT based system, and the file is being copied by a
different program. I don't want to work on the file with Perl until it's
finished being copied across.

This only works if the process writing the file is taking a manatory
exclusive lock. It also does not destinguish between a successful
transfer and an one that dies before incompletion.

If it seems to be working then it would appear that the process writing
the file is taking a manatory exclusive lock. I'm not a expert on the
intracacies of Windows' locking semantics.

As I and others have said this is not really a Perl question.

 

[MadDogMcGee]

[ Please don't post TOFU. It is considered rude an will predispose
people not to want to help you. ]

I seem to have solved my problem:

$file = "test.iso";

while (1) {

if (open(FILE, "< $file")) {
print "Transfer completed\n";
close (FILE);

}
else {
print "Not ready\n";
}

}



Is there anything undesirable about this approach?

It's a busy loop. Busy loops are undesirable on multi-taking system.
You should pause for a non-zero time between unsucessful opens.

Notes: I'm on an NT based system, and the file is being copied by a
different program. I don't want to work on the file with Perl until it's
finished being copied across.

This only works if the process writing the file is taking a manatory
exclusive lock. It also does not destinguish between a successful
transfer and an one that dies before incompletion.

If it seems to be working then it would appear that the process writing
the file is taking a manatory exclusive lock. I'm not a expert on the
intracacies of Windows' locking semantics.

As I and others have said this is not really a Perl question.

I didn't intend the while loop to be the "approach", just the "open" part.
Communication is often not "simple". I didn't plan to use the while loop
with no sleep in the actual program.
So anyway, open works fine for me, thanks everyone for your help.