Dectecting dir changes

[chuck]

I need to write a daemon for Solaris that monitors a directory for
incoming FTP transfers. Under certain conditions, when the transfer is
complete I need to send an email notification, and do other stuff.
Win32 provides FindFirstChangeNotification(), but as best I can tell
this isn't supported on Solaris.

I am thinking of using the approach suggested here
http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html
which is:

import os, time
path_to_watch = "."
before = dict ([(f, None) for f in os.listdir (path_to_watch)])
while 1:
time.sleep (10)
after = dict ([(f, None) for f in os.listdir (path_to_watch)])
added = [f for f in after if not f in before]
removed = [f for f in before if not f in after]
if added: print "Added: ", ", ".join (added)
if removed: print "Removed: ", ", ".join (removed)
before = after

My concern with this is that a change may be detected before the ftp
daemon process is done writing the file to disk. I don't want to take
any action until the file is written and closed. I know that I could
pole a new file looping to check to see if it's file size is changing
but the timing of such a loop is subject to I/O buffering and is
otherwise not elegant.

Googling shows other solutions using fcntl
(http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/217829) but it
appears that this only works on Linux.

While I'm at it I'm going to throw in a grump about the Python
documentation of fcntl. The doco indicates to read the source for
fcntl.py to lookup the constants representing the different types of
events/signals that are avaiable. However fcntl on some platforms
seems to be implemented as a binary leaving no way to look up the
contants for the platform.

Suggestions?

 

[Fredrik Lundh]

I need to write a daemon for Solaris that monitors a directory for
incoming FTP transfers. Under certain conditions, when the transfer is
complete I need to send an email notification, and do other stuff.
Win32 provides FindFirstChangeNotification(), but as best I can tell
this isn't supported on Solaris.

I am thinking of using the approach suggested here
http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html
which is:

import os, time
path_to_watch = "."
before = dict ([(f, None) for f in os.listdir (path_to_watch)])
while 1:
time.sleep (10)
after = dict ([(f, None) for f in os.listdir (path_to_watch)])
added = [f for f in after if not f in before]
removed = [f for f in before if not f in after]
if added: print "Added: ", ", ".join (added)
if removed: print "Removed: ", ", ".join (removed)
before = after

My concern with this is that a change may be detected before the ftp
daemon process is done writing the file to disk. I don't want to take
any action until the file is written and closed. I know that I could
pole a new file looping to check to see if it's file size is changing
but the timing of such a loop is subject to I/O buffering and is
otherwise not elegant.

there is no reliable way to do this, unless you can modify the sending
application to use a temporary filename during transfer (or find an FTP
server that does this automatically), or you're willing to write your own
FTP responder.

checking the mtime for new files might be good enough for some use
cases (poll the directory at regular intervals; if a file is newer than X
seconds, assume it's still being updated).

</F>

 

[Sybren Stuvel]

chuck enlightened us with:

The doco indicates to read the source for fcntl.py to lookup the
constants representing the different types of events/signals that
are avaiable. However fcntl on some platforms seems to be
implemented as a binary leaving no way to look up the contants for
the platform.

'pydoc fcntl' works fine on my system, even though the module is a
binary .so file.

Sybren

 

[chuck]

Is this on Solaris?

I think you may have missed my point. I don't have fcntl.py on my
Solaris box so how do I know what signals that I can used to monitor a
directory for modification. In other words will the following work?

fcntl.fcntl(self.fd, fcntl.F_NOTIFY,
fcntl.DN_DELETE|fcntl.DN_CREATE|fcntl.DN_MULTISHOT)

Without fcntl source, which the python documentation suggests that I
look at, I don't know if any of these constants apply to the Solaris
platform.

 

[Fredrik Lundh]

I think you may have missed my point. I don't have fcntl.py on my
Solaris box so how do I know what signals that I can used to monitor a
directory for modification. In other words will the following work?

fcntl.fcntl(self.fd, fcntl.F_NOTIFY,
fcntl.DN_DELETE|fcntl.DN_CREATE|fcntl.DN_MULTISHOT)

Without fcntl source, which the python documentation suggests that I
look at, I don't know if any of these constants apply to the Solaris
platform.

$ man fcntl

</F>

 

[chuck]

ty - more useful than 'works here'

 

[Steve Holden]

I need to write a daemon for Solaris that monitors a directory for
incoming FTP transfers. Under certain conditions, when the transfer is
complete I need to send an email notification, and do other stuff.
Win32 provides FindFirstChangeNotification(), but as best I can tell
this isn't supported on Solaris.
[...]

Suggestions?

Write an FTP server in Python, then it will know exactly when each file
transfer is complete, and it can do the mailing itself!

regards
Steve

 

[Zem]

Why not use SGI's FAM (File Alteration Monitor)? It works under
Linux...and I think I saw Solaris support somewhere. Under Linux, the
FAM daemon doesn't use inefficient polling of the filesystem instead it
monitors file changes through the kernel. Under Solaris, it'd probably
fall back to polling.

If you can use it, then you can use the nice python-fam module. Very
simple to work with and generally works very well. I wrote a python
app that monitored a directory for file creation and deletion and sent
emails based on certain events.

http://python-fam.sourceforge.net/

Good luck.