Monitoring directory for new files on Solaris

Discussion in 'Java' started by Mikhail Teterin, Mar 11, 2008.

  1. Hello!

    An application we need to write has to monitor a directory and process any
    files appearing in there -- as quickly as possible.

    The usual approach to this task is to rescan the directory every once in a
    while, which is horrible -- it loads the system keeping the program in RAM
    and still has latency measured in seconds...

    How can I ask the kernel to notify my program, when the directory is updated
    in any way -- via something like poll(), or select(), or

    http://developers.sun.com/solaris/articles/event_completion.html

    ?

    Is there a Java way of doing this already (without periodic polling), or
    will I need to make my own JNI class?

    Thanks for any pointers!

    -mi
    Mikhail Teterin, Mar 11, 2008
    #1
    1. Advertising

  2. Mikhail Teterin

    Wolfgang Guest

    Mikhail Teterin schrieb:
    > Hello!
    >
    > An application we need to write has to monitor a directory and process any
    > files appearing in there -- as quickly as possible.
    >
    > The usual approach to this task is to rescan the directory every once in a
    > while, which is horrible -- it loads the system keeping the program in RAM
    > and still has latency measured in seconds...


    what is a rescan in your sense? to scan an mostly empty dir, why should
    that be slow, dont you empty dir after processing? Which language your
    app use, pure java?

    first idea (not java): install gnu find, scan all 5 seconds for file
    newer then 5 sec., but take care (read last sentence).

    second idea: in filesystem terms: just look for changes in the
    directory, like (again just system commands): "cat /dir > /tmp/dirread1
    ; sleep 5 ; cat /dir > /tmp/dirread2 ; diff dirread1 dirread2" (sorry,
    at the moment I can not proof what cat does on dirs, maybe dd does the
    trick).
    are there any routines in C or java for just reading the dir themself,
    not the inodes in it?

    In my opinion: the more difficult part in your software are the file
    locking during processing the files and to ashure not to process files
    which are still in access of other processes, e.g. which still not
    complete or just not closed by the writing process.

    W.
    Wolfgang, Mar 11, 2008
    #2
    1. Advertising

  3. Mikhail Teterin wrote:
    > Hello!
    >
    > An application we need to write has to monitor a directory and process any
    > files appearing in there -- as quickly as possible.
    >
    > The usual approach to this task is to rescan the directory every once in a
    > while, which is horrible -- it loads the system keeping the program in RAM
    > and still has latency measured in seconds...
    >
    > How can I ask the kernel to notify my program, when the directory is updated
    > in any way -- via something like poll(), or select(), or
    >
    > http://developers.sun.com/solaris/articles/event_completion.html
    >
    > ?
    >
    > Is there a Java way of doing this already (without periodic polling), or
    > will I need to make my own JNI class?
    >
    > Thanks for any pointers!
    >
    > -mi


    I seem to remember that on Unix the modification time of the directory
    normally changes when there is any change to its content. This may be a
    bit quicker than listing the content. While repeated listing the
    directory seems ugly, it isn't all that slow or inefficient (even from
    Java) as everything is cached. This is especially true if processed
    files are removed, thus leaving the directory empty.
    Warning, the directory modification time does not change on Windows, so
    don't try this approach there.

    Mark Thornton
    Mark Thornton, Mar 11, 2008
    #3
  4. On Tue, 11 Mar 2008 18:52:55 -0000, Mikhail Teterin
    <> wrote:

    > Hello!
    >
    > An application we need to write has to monitor a directory and process
    > any
    > files appearing in there -- as quickly as possible.
    >
    > The usual approach to this task is to rescan the directory every once in
    > a
    > while, which is horrible -- it loads the system keeping the program in
    > RAM
    > and still has latency measured in seconds...
    >
    > How can I ask the kernel to notify my program, when the directory is
    > updated
    > in any way -- via something like poll(), or select(), or
    >
    > http://developers.sun.com/solaris/articles/event_completion.html
    >
    > ?
    >
    > Is there a Java way of doing this already (without periodic polling), or
    > will I need to make my own JNI class?


    from the docs for JNotify:

    "JNotify works on both Windows (Windows 2000, XP, Vista) and Linux with
    INotify support (Kernel 2.6.14 and above)."

    This probably won't help you much if you're on Solaris.

    java-fam may give you some ideas.

    Mark
    Mark Clements, Mar 11, 2008
    #4
  5. In comp.unix.solaris Mikhail Teterin <> wrote:

    > An application we need to write has to monitor a directory and process any
    > files appearing in there -- as quickly as possible.


    > The usual approach to this task is to rescan the directory every once in a
    > while, which is horrible -- it loads the system keeping the program in RAM
    > and still has latency measured in seconds...


    > How can I ask the kernel to notify my program, when the directory is updated
    > in any way -- via something like poll(), or select(), or


    > http://developers.sun.com/solaris/articles/event_completion.html


    Is there something like kqueue() in Solaris?
    In FreeBSD, many people find the wait_on utility very useful.
    See http://freebsd.unixfreunde.de/sources/wait_on-1.1.tar.gz

    --
    Victor Sudakov, VAS4-RIPE, VAS47-RIPN
    2:5005/49@fidonet http://vas.tomsk.ru/
    Victor Sudakov, Mar 12, 2008
    #5
  6. In article <fr7r7o$2rg8$>,
    Victor Sudakov <-spam-here.tomsk.su> wrote:

    > In comp.unix.solaris Mikhail Teterin <> wrote:
    >
    > > An application we need to write has to monitor a directory and process any
    > > files appearing in there -- as quickly as possible.

    >
    > > The usual approach to this task is to rescan the directory every once in a
    > > while, which is horrible -- it loads the system keeping the program in RAM
    > > and still has latency measured in seconds...

    >
    > > How can I ask the kernel to notify my program, when the directory is updated
    > > in any way -- via something like poll(), or select(), or

    >
    > > http://developers.sun.com/solaris/articles/event_completion.html

    >
    > Is there something like kqueue() in Solaris?
    > In FreeBSD, many people find the wait_on utility very useful.
    > See http://freebsd.unixfreunde.de/sources/wait_on-1.1.tar.gz


    There isn't anything in Solaris that does this but with OpenSolaris, you
    could mod the kernel or maybe create a Dtrace script that "wakes up"
    when a directory is modified.

    --
    DeeDee, don't press that button! DeeDee! NO! Dee...
    Michael Vilain, Mar 12, 2008
    #6
  7. Mikhail Teterin

    Wolfgang Guest

    Wolfgang schrieb:
    >
    >
    > second idea: in filesystem terms: just look for changes in the
    > directory,


    I forgot (or in german: den Wald vor lauter Baeumen nicht sehen):
    as mentioned in another post, unix has also for directorys as for every
    inode the atime/mtime/ctime. I think to monitor just the mtime of the
    dir give you the trigger to scan for new files, if you dont have subdirs.

    W.
    Wolfgang, Mar 12, 2008
    #7
  8. Mikhail Teterin

    Guest

    On Mar 11, 4:38 pm, Wolfgang <> wrote:
    > Mikhail Teterin schrieb:
    >
    > > Hello!

    >
    > > An application we need to write has to monitor a directory and process any
    > > files appearing in there -- as quickly as possible.

    >
    > > The usual approach to this task is to rescan the directory every once in a
    > > while, which is horrible -- it loads the system keeping the program in RAM
    > > and still has latency measured in seconds...

    >
    > what is a rescan in your sense? to scan an mostly empty dir, why should
    > that be slow, dont you empty dir after processing? Which language your
    > app use, pure java?
    >
    > first idea (not java): install gnu find, scan all 5 seconds for file
    > newer then 5 sec., but take care (read last sentence).


    GNU find has option -cmin, but I could not find option for second.

    Victor
    , Mar 12, 2008
    #8
  9. On Tue, 11 Mar 2008 14:52:55 -0400, Mikhail Teterin wrote:
    > An application we need to write has to monitor a directory and
    > process any files appearing in there -- as quickly as possible.
    >
    > The usual approach to this task is to rescan the directory every
    > once in a while, which is horrible -- it loads the system keeping
    > the program in RAM and still has latency measured in seconds...


    Three suggestions:

    - Poll the last modified time of the directory itself at a suitable
    interval. It will be updated when changes are made to the directory
    (e.g. files added, removed, renamed). When you see that something
    has changed, rescan the directory to see what it was.

    - I believe Solaris has an mechanism for subscribing to file system
    events. Write a JNI wrapper for that.

    - Modify the application producing these files, so that it can notify
    your application.

    /gordon

    --
    Gordon Beaton, Mar 12, 2008
    #9
  10. On Wed, 12 Mar 2008 08:09:41 -0700 (PDT), wrote:
    > GNU find has option -cmin, but I could not find option for second.


    Remember what the most recent file was that you processed, and use
    "-cnewer" instead. If you have to remove the files as you process
    them, you can create a marker file of your own solely for this
    purpose.

    /gordon

    --
    Gordon Beaton, Mar 12, 2008
    #10
  11. On 12 Mar 2008 15:30:33 GMT, Gordon Beaton wrote:
    > On Wed, 12 Mar 2008 08:09:41 -0700 (PDT), wrote:
    >> GNU find has option -cmin, but I could not find option for second.

    >
    > Remember what the most recent file was that you processed, and use
    > "-cnewer" instead. If you have to remove the files as you process
    > them, you can create a marker file of your own solely for this
    > purpose.


    On the other hand, there is no need to use find when your existing
    application already does exactly this kind of scanning. Find doesn't
    offer any shortcuts here.

    /gordon

    --
    Gordon Beaton, Mar 12, 2008
    #11
  12. In comp.unix.solaris Gordon Beaton <> wrote:
    > - I believe Solaris has an mechanism for subscribing to file system
    > events.


    Could you please elaborate.

    --
    Victor Sudakov, VAS4-RIPE, VAS47-RIPN
    2:5005/49@fidonet http://vas.tomsk.ru/
    Victor Sudakov, Mar 13, 2008
    #12
  13. Mikhail Teterin

    Ian Collins Guest

    Victor Sudakov wrote:
    > In comp.unix.solaris Gordon Beaton <> wrote:
    >> - I believe Solaris has an mechanism for subscribing to file system
    >> events.

    >
    > Could you please elaborate.
    >

    If you are using Solaris Express there's this:

    http://blogs.sun.com/praks/entry/file_events_notification

    --
    Ian Collins.
    Ian Collins, Mar 13, 2008
    #13
  14. In comp.unix.solaris Ian Collins <> wrote:
    > >> - I believe Solaris has an mechanism for subscribing to file system
    > >> events.

    > >
    > > Could you please elaborate.
    > >

    > If you are using Solaris Express there's this:


    > http://blogs.sun.com/praks/entry/file_events_notification


    This is not official, I presume. Not supported by Sun, is it?


    --
    Victor Sudakov, VAS4-RIPE, VAS47-RIPN
    2:5005/49@fidonet http://vas.tomsk.ru/
    Victor Sudakov, Mar 13, 2008
    #14
  15. Victor Sudakov <-spam-here.tomsk.su> writes:

    >In comp.unix.solaris Ian Collins <> wrote:
    >> >> - I believe Solaris has an mechanism for subscribing to file system
    >> >> events.
    >> >
    >> > Could you please elaborate.
    >> >

    >> If you are using Solaris Express there's this:


    >> http://blogs.sun.com/praks/entry/file_events_notification


    >This is not official, I presume. Not supported by Sun, is it?



    What do you mean by "not official"? It's not in Solaris 10 or
    earlier if that is what you mean.


    Casper
    --
    Expressed in this posting are my opinions. They are in no way related
    to opinions held by my employer, Sun Microsystems.
    Statements on Sun products included here are not gospel and may
    be fiction rather than truth.
    Casper H.S. Dik, Mar 13, 2008
    #15
  16. In comp.unix.solaris Casper H.S. Dik <> wrote:
    > >> >> - I believe Solaris has an mechanism for subscribing to file system
    > >> >> events.
    > >> >
    > >> > Could you please elaborate.
    > >> >
    > >> If you are using Solaris Express there's this:


    > >> http://blogs.sun.com/praks/entry/file_events_notification


    > >This is not official, I presume. Not supported by Sun, is it?



    > What do you mean by "not official"? It's not in Solaris 10 or
    > earlier if that is what you mean.


    I mean it's not from Sun but from some volunteer contributor, is it?


    --
    Victor Sudakov, VAS4-RIPE, VAS47-RIPN
    2:5005/49@fidonet http://vas.tomsk.ru/
    Victor Sudakov, Mar 14, 2008
    #16
  17. Mikhail Teterin

    Thommy M. Guest

    On Mar 14, 1:38 pm, Victor Sudakov <-spam-here.tomsk.su>
    wrote:
    > In comp.unix.solaris Casper H.S. Dik <> wrote:
    >
    > > >> >> - I believe Solaris has an mechanism for subscribing to file system
    > > >> >> events.

    >
    > > >> > Could you please elaborate.

    >
    > > >> If you are using Solaris Express there's this:
    > > >>http://blogs.sun.com/praks/entry/file_events_notification
    > > >This is not official, I presume. Not supported by Sun, is it?

    > > What do you mean by "not official"? It's not in Solaris 10 or
    > > earlier if that is what you mean.

    >
    > I mean it's not from Sun but from some volunteer contributor, is it?


    Well, as he has his blog on sun.com he is more than a volunteer... ;-)
    Thommy M., Mar 14, 2008
    #17
  18. Ian Collins wrote:
    > If you are using Solaris Express there's this:
    >
    > http://blogs.sun.com/praks/entry/file_events_notification


    Yes, this would be what I'm looking for, except Sun is not shipping it -- it
    will not be in the production hardware for years :(

    Damn it, *BSD had kqueue/kevent for years already, but I'm stuck on
    Solaris...

    -mi
    Mikhail Teterin, Mar 14, 2008
    #18
  19. Wolfgang wrote:
    > what is a rescan in your sense? to scan an mostly empty dir, why should
    > that be slow, dont you empty dir after processing? Which language your
    > app use, pure java?


    > first idea (not java): install gnu find, scan all 5 seconds for file
    > newer then 5 sec., but take care (read last sentence).


    The actual rescan is not difficult to implement -- as others (and yourself)
    have pointed out, one just needs to check the directory's mtime.

    However, the problems common with all /periodic/ algorithms is that they are
    inherently inefficient. You want to be event driven: getting woken up, when
    stuff happens, instead of -- as bored children do to their parents when
    riding in a car -- continuously asking the kernel: "Are we there yet? No?
    How about now? Are we there yet?"

    For example, you suggested 5 seconds, so let's go with that... Polling
    means, your program must always reside in memory -- the OS can not really
    page it out, because the program needs to run every 5 seconds. On the other
    hand, your average latency will still be 2.5 seconds (and up to 5), which
    may well be too long for your application...

    Moving the 5 seconds figure into any direction eases one of these two
    negatives while exacerbating the other...

    Better OSes have a way for programs to request notification from the OS --
    the kernel knows, when it changes things, so why can't it tell the process,
    if asked nicely? Unfortunately, Solaris has not joined this club yet,
    according to the link kindly posted by Ian Collins in this thread:

    http://blogs.sun.com/praks/entry/file_events_notification

    For now, however, no good solution appears to exist and I will have to use
    polling...

    Yours,

    -mi
    Mikhail Teterin, Mar 14, 2008
    #19
  20. In comp.unix.solaris Mikhail Teterin <> wrote:
    > http://blogs.sun.com/praks/entry/file_events_notification
    >
    > For now, however, no good solution appears to exist and I will have to use
    > polling...


    It's not the best solution, but you could use 'dtrace' on Solaris 10.
    It would not require polling, but recieves probes as events. That said,
    I don't think you can guarantee its correctness. You'd want to fall
    back on polling (at a longer frequency) as a safety check.

    --
    Darren Dunham
    Senior Technical Consultant TAOS http://www.taos.com/
    Got some Dr Pepper? San Francisco, CA bay area
    < This line left intentionally blank to confuse you. >
    Darren Dunham, Mar 14, 2008
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?SmVmZiBSZWVk?=

    Failed to Start Monitoring Directory Changes

    =?Utf-8?B?SmVmZiBSZWVk?=, Feb 15, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    1,761
    =?Utf-8?B?SmVmZiBSZWVk?=
    Feb 15, 2004
  2. =?Utf-8?B?ZXRyb3BpYw==?=

    Failed to start monitoring changes to <directory?>

    =?Utf-8?B?ZXRyb3BpYw==?=, Apr 18, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    548
    =?Utf-8?B?ZXRyb3BpYw==?=
    Apr 18, 2006
  3. Replies:
    0
    Views:
    348
  4. Florian Lindner

    Monitoring a directory for changes

    Florian Lindner, Sep 20, 2005, in forum: Python
    Replies:
    2
    Views:
    326
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Sep 20, 2005
  5. zdk
    Replies:
    1
    Views:
    291
    Ian Collins
    Jul 27, 2006
Loading...

Share This Page