Efficient polling of a flatfile

[Ike]

Just looking for feedback on what you might consider the most efficient
means to poll a flat file.

I need to have my app look at a flat file, say, every second or so, and read
any data that may have been appended to it.

Wondering what others might consider an efficient means of doing this in
Java. Would it be wise to merely look at the file size everysecond or so,
and, if that has changed, to then read it, starting from that byte which was
the previous size of the file on the previous polling a second ago?

Is there a more efficent way to do this? Thanks, Ike

 

[Fahd Shariff]

Hi,

An alternative approach might be to:

- Add a timer to your app.
- At each interval your app would check the last modified time i.e.
file.lastModified() to find out if it has changed.
- If there is a change it would notify all listeners.
- The listener would then read the appended text. If the text is
delimited you could use the substring method.

Otherwise your byte method looks fine to.

 

[HK]

Wondering what others might consider an efficient means of doing this in
Java. Would it be wise to merely look at the file size everysecond or so,
and, if that has changed, to then read it, starting from that byte which was
the previous size of the file on the previous polling a second ago?

Hopefully you really only want to read bytes. For any data type
which is longer than one byte you may run into the problem that
the new data ends in the middle of an item (int, char, etc).
This happens if you do not have 100% control over the output
buffering of the application which writes your file. Consequently
you must do your own buffering when reading to handle incomplete data
items at the end of the buffer. A likely trap is char data, because
some encodings are more than one byte long.

Sometimes the scheme you describe cannot be avoided, but if at
all possible, I would try to redirect or duplicate the output
of the program which generates the data so that the application
we are talking about can just read the data. If you are on a UNIX
like system, pipes, named pipes, tee come to mind. In general,
sockets are always an option.

Harald.

 

[Dotty]

Hi,

An alternative approach might be to:

- Add a timer to your app.
- At each interval your app would check the last modified time i.e.
file.lastModified() to find out if it has changed.
- If there is a change it would notify all listeners.
- The listener would then read the appended text. If the text is
delimited you could use the substring method.

Otherwise your byte method looks fine to.

But is the resolution of the file time only one second?
Maybe check the length since others only append.
What do you do if you try to access the file when it is
being used by another process?

 

[Jacob]

Just looking for feedback on what you might consider the most efficient
means to poll a flat file.

I need to have my app look at a flat file, say, every second or so, and read
any data that may have been appended to it.

Wondering what others might consider an efficient means of doing this in
Java. Would it be wise to merely look at the file size everysecond or so,
and, if that has changed, to then read it, starting from that byte which was
the previous size of the file on the previous polling a second ago?

Is there a more efficent way to do this? Thanks, Ike

You can check the "fileaccessor" module of http://geosoft.no/software
which does exactly this.

The moudle is generic, so it checks the lastModified tag only. What
you suggest is more specific as you imply knowledge of *how* your
source file changes with time. This might be important if the file is
large and/or performance is extremely important.

 

[Ike]

Wow, nice set of classes at this site Jacob! I'm still concenerned about
what Dotty says regarding what might occur if a file is open in another
process when I go to read it. -Ike