wait until change

[Tom]

Hi,

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

What would be the appropriate command to check if the file changed? And
how can I implement "wait until" into my program? I googled a lot to
find any hints, but couldn't find anything helpful. But I found
something about a python that was stabbed in NY in 2000. Thought
that was funny.

I would deeply appreciate, if you could maybe include some little
fragments of code to show me how to use your suggestions. I am
relatively new to python and I often have problems with Pythons Library
Reference, because it doesn't show examples and is very technically.
That makes it difficult for me to really understand that. It also would
be great, if you could tell me which modules I have to include. I hope
all that is not asked too much. I assume that both problems can be
solved with a few easy lines of code. I just don't know them.

Thanks a lot for your help and suggestions.
Regards Tom

 

[Lane LiaBraaten]

Hi,

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

What would be the appropriate command to check if the file changed? And
how can I implement "wait until" into my program? I googled a lot to
find any hints, but couldn't find anything helpful. But I found
something about a python that was stabbed in NY in 2000. Thought
that was funny.

I would use os.stat() which returns all sorts of info about the file including
modification time.

More info at: http://web.pydoc.org/1.5.2/os.html

time=os.stat(file)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again

I would deeply appreciate, if you could maybe include some little
fragments of code to show me how to use your suggestions. I am
relatively new to python and I often have problems with Pythons Library
Reference, because it doesn't show examples and is very technically.
That makes it difficult for me to really understand that. It also would
be great, if you could tell me which modules I have to include. I hope
all that is not asked too much. I assume that both problems can be
solved with a few easy lines of code. I just don't know them.

Thanks a lot for your help and suggestions.
Regards Tom

I'm sure there are a number of ways to do this.

Good Luck,
LGL

 

[Peter Hansen]

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

Would it be sufficient to have your program periodically wake up and
check the timestamp on the file? To do that, a combination of time.sleep()
and os.stat() would be sufficient, and quite cross-platform.

-Peter

 

[Tom]

I would use os.stat() which returns all sorts of info about the file including
modification time.

More info at: http://web.pydoc.org/1.5.2/os.html

time=os.stat(file)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again

Hi Lane,

thanks for your answer it works great and gives me a lot of new ideas
that I can work with.
That "while 1" command looks interessting. I never used while that way.
The Library Reference doesn't explain it. But am I right if I assume
that this is a loop that continues forever? Despite using break of
course. I ask, because your suggestion works, but unfortunately uses
100% of my CPU power. :-( Well, it works, but is there maybe another way
to do the same without using so much of my resources? That would be
great.

Thank's again.
Tom

 

[Peter Hansen]

thanks for your answer it works great and gives me a lot of new ideas
that I can work with.
That "while 1" command looks interessting. I never used while that way.
The Library Reference doesn't explain it. But am I right if I assume
that this is a loop that continues forever? Despite using break of
course. I ask, because your suggestion works, but unfortunately uses
100% of my CPU power. :-( Well, it works, but is there maybe another way
to do the same without using so much of my resources? That would be
great.

Use time.sleep().

-Peter

 

[Tom]

Use time.sleep().

-Peter

If I use sleep, I have to know how long the process should sleep, but
sometimes it should sleep for seconds and sometimes for minutes. So it
would be good if I can put it to sleep and somehow wake it up if the
file changed.

Tom

 

[Tim Hochberg]

If I use sleep, I have to know how long the process should sleep, but
sometimes it should sleep for seconds and sometimes for minutes. So it
would be good if I can put it to sleep and somehow wake it up if the
file changed.

If you use something like::

ctime = os.stat(file).st_ctime
while True:
if os.stat(file).st_ctime > ctime:
break
time.sleep(1.0)
# Do stuff...

The load on your CPU will be negligible and your program will wake up at
most 1 second after the modification

-tim

 

[Peter Hansen]

If I use sleep, I have to know how long the process should sleep, but
sometimes it should sleep for seconds and sometimes for minutes. So it
would be good if I can put it to sleep and somehow wake it up if the
file changed.

See Tim's excellent reply.

Also, note that there are non-cross-platform solutions that basically
notify your program "as soon as" (possibly without any latency guarantees)
a change has happened. They are harder to use and quite likely unnecessary
in your case, and the time.sleep() approach Tim points out is simpler
and therefore the best thing to start with.

You do have to confirm of course that this other application won't
be writing to the file so often that you might run into trouble.

Does this other program write updates infrequently, and does your
Python analysis script run quickly enough, that if your code wakes
up within a few seconds of each update it will complete its work
soon enough to avoid problems?

-Peter

 

[Werner Schiendl]

Hi,

What's the problem?

Just wait as long as you can accept to be "late" when your file changes.

So if you can accept e. g. 5 seconds delay, you'll change Lane's example to:

time=os.stat(file)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again
else:
time.sleep(5)


This will drastically reduce your CPU load ;-)

hth

Werner

 

[Dave Benjamin]

Also, note that there are non-cross-platform solutions that basically
notify your program "as soon as" (possibly without any latency guarantees)
a change has happened. They are harder to use and quite likely unnecessary
in your case, and the time.sleep() approach Tim points out is simpler
and therefore the best thing to start with.

Here's one that uses PythonWin and WMI. I wrote this awhile ago because I
was experimenting with writing a templating system that would process input
files "on the fly" whenever they changed. It seems to drive my CPU usage up,
though - not sure why, because I'm pretty sure that events.NextEvent()
blocks until an event comes. Anyway, I don't know if this is useful to
anyone, but here it is... (apologies for the wide margins):

import time, win32com; time.sleep(1)
from win32com.client import GetObject

interval = 1
filenames = [r'c:\test.txt', r'c:\output.txt']

class FileChangeEvent:
def __init__(self, filename, new_size, old_size):
self.filename = filename
self.new_size = new_size
self.old_size = old_size

def on_change(event):
print event.__dict__

wmi = GetObject(r'winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2')
filespec = ' OR '.join(["TargetInstance.Name='%s'" % filename.replace('\\', '\\\\') for filename in filenames])
query = "SELECT * FROM __InstanceModificationEvent WITHIN %d WHERE TargetInstance ISA 'CIM_DataFile' AND (%s)" % (interval, filespec)
events = wmi.ExecNotificationQuery(query)

while True:
event = events.NextEvent()
on_change(FileChangeEvent(str(event.TargetInstance.Name),
int(event.TargetInstance.FileSize),
int(event.PreviousInstance.FileSize)))

 

[Matthew Wilson]

I just took a midterm in my operating systems class, and a big topic was
"busy waiting" and semaphores.

And now, today, a real-world example! Anyway, the time.sleep() method
is probably good enough, but how would a semaphore solution work in
python? A semaphore solution is one where the process that uses the
updated file is asleep until another process says "Hey! Wake up! I
updated the file!". This avoids all those check_for_update() calls
which bog down the CPU.

Anyone?

 

[Peter Hansen]

I just took a midterm in my operating systems class, and a big topic was
"busy waiting" and semaphores.

And now, today, a real-world example! Anyway, the time.sleep() method
is probably good enough, but how would a semaphore solution work in
python? A semaphore solution is one where the process that uses the
updated file is asleep until another process says "Hey! Wake up! I
updated the file!". This avoids all those check_for_update() calls
which bog down the CPU.

In this case, it appears the OP doesn't have the capability of changing
the other application, so what you suggest is out.

Dave's example with WMI uses the *operating system* to essentially
signal semaphore-like that an update has been made. As I suggested,
it's obviously much more complicated than the simple time.sleep()
plus os.stat() approach, but there are times when it's the right
way to approach this problem.

-Peter

 

[Thomas Heller]

In this case, it appears the OP doesn't have the capability of changing
the other application, so what you suggest is out.

Dave's example with WMI uses the *operating system* to essentially
signal semaphore-like that an update has been made. As I suggested,
it's obviously much more complicated than the simple time.sleep()
plus os.stat() approach, but there are times when it's the right
way to approach this problem.

Just for completeness, it should be mentioned that the win32 api
functions FindFirstChangeNotification and FindNextChangeNotification
could also be used. They are wrapped in win32all, and probably could
also be made to work with ctypes. Platform dependend, sure, but maybe a
much more lightweight approach than using wmi (and, for me at least,
probably better understandable).

-Peter

Thomas

 

[Michael Geary]

Just for completeness, it should be mentioned that the win32 api
functions FindFirstChangeNotification and FindNextChangeNotification
could also be used. They are wrapped in win32all, and probably could
also be made to work with ctypes. Platform dependend, sure, but maybe a
much more lightweight approach than using wmi (and, for me at least,
probably better understandable).

And to be *really* complete, Windows NT/2000/XP has the
ReadDirectoryChangesW function, which has some improvements over the
Find...ChangeNotification functions. Not available on Win95/98/Me, though.

-Mike

 

[Heiko Wundram]

And to be *really* complete, Windows NT/2000/XP has the
ReadDirectoryChangesW function, which has some improvements over the
Find...ChangeNotification functions. Not available on Win95/98/Me, though.

-Mike

And to be really, really complete, Unix (in my case Linux, afaik also
works under Solaris and HP-UX) has FAM (the file alteration monitor),
which works as a user-space daemon to which a program can attach to get
a file/directory change notification. The daemon itself uses a special
kernel module to get these notification right from the VFS (virtual file
system).

I recall that there was a Python-wrapper somewhere for the fam client
library.

HTH!

Heiko.

 

[Aahz]

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

What would be the appropriate command to check if the file changed? And
how can I implement "wait until" into my program? I googled a lot to
find any hints, but couldn't find anything helpful. But I found
something about a python that was stabbed in NY in 2000. Thought
that was funny.

os.stat() is the correct answer, but I question the other responses that
suggest using time.sleep(). Unless your application stores a lot of
data in memory, you might be better off using your OS to periodically
run your application; your application stores the result of os.stat() in
a private file. On Unix-like systems, the facility is called cron;
dunno what that would be on Windows. This would simplify your program;
it also means your program would automatically start when the system
gets rebooted.

 

[Bryan]

os.stat() is the correct answer, but I question the other responses that
suggest using time.sleep(). Unless your application stores a lot of
data in memory, you might be better off using your OS to periodically
run your application; your application stores the result of os.stat() in
a private file. On Unix-like systems, the facility is called cron;
dunno what that would be on Windows. This would simplify your program;
it also means your program would automatically start when the system
gets rebooted.

i'm currently running a python script that automatically runs when the system gets rebooted on windows. i use the
"scheduled tasks" control panel applet. you can have your program scheduled to run weekly, monthly, daily, once, at
system startup, at logon, or when idle. you can specify start/end times, durations, how to repeat the task, plus a
bunch more options.

bryan

 

[Mark Roach]

And to be really, really complete, Unix (in my case Linux, afaik also
works under Solaris and HP-UX) has FAM (the file alteration monitor),
which works as a user-space daemon to which a program can attach to get
a file/directory change notification. The daemon itself uses a special
kernel module to get these notification right from the VFS (virtual file
system).

I recall that there was a Python-wrapper somewhere for the fam client
library.

There also appears to be a python wrapper for dnotify which is the
kernel-level file-change notification system so that you don't have to
have famd and portmap running just to get file notifications... It appears
to be apart of twisted according to this message:
http://twistedmatrix.com/pipermail/twisted-python/2003-April/003873.html

-Mark

 

[Anand Pillai]

If you dont want to use sleep(), try this mechanism
based on 'Timer' class in the threading module.

1. Create a timer thread that calls a function
which checks for the time stamp using os.stat().
2. Schedule the timer to wake up after 'n' seconds
and call this function.
3. If the call succeeds all and well, otherwise in
the function, create a new timer and set it again
to wake itself up after another 'n' seconds.

A timer can be used only once so you need to make a new timer
for every wake-up call. Perhaps not very good on memory,
but it might save you the CPU cycles.

You can wrap the whole thing in a class with the timer
bound to a local class variable, and the function as
another class attribute.

HTH.

-Anand

I would use os.stat() which returns all sorts of info about the file including
modification time.

More info at: http://web.pydoc.org/1.5.2/os.html

time=os.stat(file)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again

Hi Lane,

thanks for your answer it works great and gives me a lot of new ideas
that I can work with.
That "while 1" command looks interessting. I never used while that way.
The Library Reference doesn't explain it. But am I right if I assume
that this is a loop that continues forever? Despite using break of
course. I ask, because your suggestion works, but unfortunately uses
100% of my CPU power. :-( Well, it works, but is there maybe another way
to do the same without using so much of my resources? That would be
great.

Thank's again.
Tom

 

[Anand Pillai]

I thought this was fairly straightforward, but when
I tried to write one I figured out that the standard
python Timers cannot be used.

Anyway here is how you would go about doing it.
This piece of code uses a time out as an exit condition,
whereas in the scenario given by the O.P it would be
the file's timestamp verification.

----------------------------------------------------
from threading import Thread
import time

class ModifiedTimer(Thread):
""" Modified timer with a maximum timeout
and repeated calls """

def __init__(self, interval, maxtime, target):
self.__interval = interval
self.__maxtime = maxtime
self.__tottime = 0.0
self.__target = target
self.__flag = 0
Thread.__init__(self, None, 'mytimer', None)

def run(self):

while self.__flag==0:
time.sleep(self.__interval)
self.__target()
self.__tottime += self.__interval

if self.__tottime >= self.__maxtime:
print 'Timing out...'
self.end()

def end(self):
self.__flag=1

class MyTimer:

def __init__(self):
self.__interval = 5.0

def __timerfunc(self):
print 'Hello, this is the timer function!'

def make_timer(self, interval):
self.__interval = interval
self.__timer = ModifiedTimer(self.__interval, 60.0, self.__timerfunc)
self.__timer.start()

if __name__=="__main__":
t=MyTimer()
t.make_timer(10.0)

----------------------------------------------------------------

The code works as expected. I tested on Windows 98
using python 2.3.

-Anand Pillai

If you dont want to use sleep(), try this mechanism
based on 'Timer' class in the threading module.

1. Create a timer thread that calls a function
which checks for the time stamp using os.stat().
2. Schedule the timer to wake up after 'n' seconds
and call this function.
3. If the call succeeds all and well, otherwise in
the function, create a new timer and set it again
to wake itself up after another 'n' seconds.

A timer can be used only once so you need to make a new timer
for every wake-up call. Perhaps not very good on memory,
but it might save you the CPU cycles.

You can wrap the whole thing in a class with the timer
bound to a local class variable, and the function as
another class attribute.

HTH.

-Anand

I would use os.stat() which returns all sorts of info about the file including
modification time.

More info at: http://web.pydoc.org/1.5.2/os.html

time=os.stat(file)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again

Hi Lane,

thanks for your answer it works great and gives me a lot of new ideas
that I can work with.
That "while 1" command looks interessting. I never used while that way.
The Library Reference doesn't explain it. But am I right if I assume
that this is a loop that continues forever? Despite using break of
course. I ask, because your suggestion works, but unfortunately uses
100% of my CPU power. :-( Well, it works, but is there maybe another way
to do the same without using so much of my resources? That would be
great.

Thank's again.
Tom