File Monitoring modules

Discussion in 'Perl Misc' started by nagandla@gmail.com, Jan 22, 2008.

  1. Guest

    Hi All,

    For monitoring files in directories i am using SGI::FAM module...
    this module is monitoring perfectlt but taking more 90% CPU usage
    while monitoring ...
    so any one can suggest me the better module to monitor the files in
    directoties....here one more thing i want to monitor more than a
    single directory at a time...

    Regards ,
    Lakshmi Nagandla
     
    , Jan 22, 2008
    #1
    1. Advertising

  2. Ben Morrow Guest

    Quoth :
    >
    > For monitoring files in directories i am using SGI::FAM module...
    > this module is monitoring perfectlt but taking more 90% CPU usage
    > while monitoring ...


    It is polling, rather than using a kernel-based notification mechanism.
    What OS are you on?

    > so any one can suggest me the better module to monitor the files in
    > directoties....


    There are OS-specific interfaces to most of the file-change services I
    know of (kqueue, inotify, Win32's ChangeNotify). If you know what your
    OS supports, you can search for a direct interface to that.

    > here one more thing i want to monitor more than a
    > single directory at a time...


    All of the above will do that.

    Ben
     
    Ben Morrow, Jan 22, 2008
    #2
    1. Advertising

  3. Guest

    On Jan 22, 3:08 pm, Ben Morrow <> wrote:
    > Quoth :
    >
    >
    >
    > > For monitoring files in directories i am using SGI::FAM module...
    > > this module is monitoring perfectlt but taking more 90% CPU usage
    > > while monitoring ...

    >
    > It is polling, rather than using a kernel-based notification mechanism.
    > What OS are you on?
    >
    > > so any one can suggest me the better module to monitor the files in
    > > directoties....

    >
    > There are OS-specific interfaces to most of the file-change services I
    > know of (kqueue, inotify, Win32's ChangeNotify). If you know what your
    > OS supports, you can search for a direct interface to that.
    >
    > > here one more thing i want to monitor more than a
    > > single directory at a time...

    >
    > All of the above will do that.
    >
    > Ben



    My OS is Linux,,Dapper Drake Ubuntu System
     
    , Jan 22, 2008
    #3
  4. Ben Morrow Guest

    Quoth :
    > On Jan 22, 3:08 pm, Ben Morrow <> wrote:
    > > Quoth :
    > >
    > > > For monitoring files in directories i am using SGI::FAM module...
    > > > this module is monitoring perfectlt but taking more 90% CPU usage
    > > > while monitoring ...

    > >
    > > It is polling, rather than using a kernel-based notification mechanism.
    > > What OS are you on?

    >
    > My OS is Linux,,Dapper Drake Ubuntu System


    Huh. fam(3) ought to work perfectly well under Linux... If you write a
    minimal C program which watches the same directories, does it have the
    same behaviour? If so, this is a problem with fam on your system, and
    you need to take it up with the developers, if there are any. If there
    are, they'll probably tell you to switch to gamin; making SGI::FAM work
    with gamin has been on my todo list for a while now... :)

    Ben
     
    Ben Morrow, Jan 22, 2008
    #4
  5. Guest

    On Jan 22, 6:16 pm, Ben Morrow <> wrote:
    > Quoth :
    >
    > > On Jan 22, 3:08 pm, Ben Morrow <> wrote:
    > > > Quoth :

    >
    > > > > For monitoring files in directories i am using SGI::FAM module...
    > > > > this module is monitoring perfectlt but taking more 90% CPU usage
    > > > > while monitoring ...

    >
    > > > It is polling, rather than using a kernel-based notification mechanism.
    > > > What OS are you on?

    >
    > > My OS is Linux,,Dapper Drake Ubuntu System

    >
    > Huh. fam(3) ought to work perfectly well under Linux... If you write a
    > minimal C program which watches the same directories, does it have the
    > same behaviour? If so, this is a problem with fam on your system, and
    > you need to take it up with the developers, if there are any. If there
    > are, they'll probably tell you to switch to gamin; making SGI::FAM work
    > with gamin has been on my todo list for a while now... :)
    >
    > Ben



    Ya thank U Ben,

    Just now i found one more module for monitoring is File::Monitor....
    It is also all most all same as fam...and also taking equivalent time
    to moniotr when a file is newly created or deleted....
    and also it is taking very less CPU usage (below 20%) while monitoring
    directories...
    Have a look on that....and suggest me that whether can i use that...
     
    , Jan 22, 2008
    #5
  6. Peter Scott Guest

    On Mon, 21 Jan 2008 23:29:20 -0800, nagandla wrote:
    > For monitoring files in directories i am using SGI::FAM module...
    > this module is monitoring perfectlt but taking more 90% CPU usage
    > while monitoring ...
    > so any one can suggest me the better module to monitor the files in
    > directoties....here one more thing i want to monitor more than a
    > single directory at a time...


    Maybe
    http://search.cpan.org/~twerner/Linux-Inotify-0.05/lib/Linux/Inotify.pm

    --
    Peter Scott
    http://www.perlmedic.com/
    http://www.perldebugged.com/
     
    Peter Scott, Jan 22, 2008
    #6
  7. Ted Zlatanov Guest

    On Mon, 21 Jan 2008 23:29:20 -0800 (PST) wrote:

    n> For monitoring files in directories i am using SGI::FAM module...
    n> this module is monitoring perfectlt but taking more 90% CPU usage
    n> while monitoring ... so any one can suggest me the better module to
    n> monitor the files in directoties....here one more thing i want to
    n> monitor more than a single directory at a time...

    It could be a bug in your code. Can you show the code?

    Use `top' or something similar to see what's actually using the CPU. Is
    it the FAM daemon or your code? You could be in a busy loop...

    Ted
     
    Ted Zlatanov, Jan 22, 2008
    #7
  8. Guest

    On Jan 23, 1:59 am, Ted Zlatanov <> wrote:
    > On Mon, 21 Jan 2008 23:29:20 -0800 (PST) wrote:
    >
    > n> For monitoring files in directories i am using SGI::FAM module...
    > n> this module is monitoring perfectlt but taking more 90% CPU usage
    > n> while monitoring ... so any one can suggest me the better module to
    > n> monitor the files in directoties....here one more thing i want to
    > n> monitor more than a single directory at a time...
    >
    > It could be a bug in your code. Can you show the code?
    >
    > Use `top' or something similar to see what's actually using the CPU. Is
    > it the FAM daemon or your code? You could be in a busy loop...
    >
    > Ted




    Here is the code to monitor directories using Fam....
    use strict;
    use warnings;

    use SGI::FAM;
    my $fam;
    my $terminate = 0;
    my $path = '/data';
    my %interested_dirs = ();
    sub today_datetime
    {
    my($ss,$mi,$hh,$day,$mon,$year) = localtime();
    $mon++;
    $year+=1900;
    my $now = "$year-$mon-$day:$hh:$mi:$ss";
    return($now);
    }

    opendir(DIR,$path);
    my @file_s = readdir(DIR);
    foreach(@file_s)
    {
    if ($_ !~ /\.$/)
    {
    if(-d "$path/$_")
    {
    # print "$_ <-> $path/$_ \n ";
    $interested_dirs{$_} = $path."/".$_ ;
    }
    }
    }
    closedir(DIR);

    until ($terminate)
    {
    unless (defined($fam))
    {
    eval { $fam = new SGI::FAM; };
    if ($@)
    {
    print "Failed to connect to FAMd - starting forcibly...\n\n\n";
    system("/usr/local/sbin/famd");
    next;
    }


    die "FAM Failed: $!\n" unless (defined($fam));
    foreach (values %interested_dirs)
    {
    $fam->monitor($_);
    #print "$_\n";
    }
    }

    # check for FAM events
    if ($fam->pending)
    {
    my $event;
    eval { $event = $fam->next_event; };

    if ($@)
    {
    print "FAM error: rebuilding cache...\n\n\n";
    $fam = undef;
    next;
    }
    #print "Received: " . $event->type . " for " . $event->filename .
    "\n";
    &process_fam_event($event);
    }

    }
    sub process_fam_event
    {
    my $event = shift;
    my $filename = $event->filename;
    my $dir = $fam->which($event);
    my $now = &today_datetime;
    if($event->type eq 'create')
    {
    if($filename ne $dir)
    {
    my $subdir = "$dir/$filename";
    while(`/usr/sbin/lsof +d $dir | grep "$filename"`)
    {
    next;
    }

    if(-d $subdir)
    {
    print "Creating $subdir \n";
    eval { $fam->monitor($subdir); };
    if($@)
    {
    print "Fam Error : Failed to monitor $subdir : $@ \n";
    }
    }
    }

    print "Createed--------NewFile : $filename , Dir : $dir $now\n";

    }elsif($event->type eq 'delete')
    {
    if($filename ne $dir)
    {
    my $subdir = "$dir/$filename";
    while(`/usr/sbin/lsof +d $dir | grep "$filename"`)
    {
    next;
    }

    if(-d $subdir)
    {
    print "Deleting $subdir \n";
    eval { $fam->cancel($subdir); };
    if($@)
    {
    print "Fam Error : Failed to monitor $subdir : $@ \n";
    }
    }
    }

    print "Delete : $filename , Dir : $dir $now\n";
    }

    }


    Nagandla
     
    , Jan 24, 2008
    #8
  9. Ted Zlatanov Guest

    On Thu, 24 Jan 2008 01:22:57 -0800 (PST) wrote:

    n> until ($terminate)
    n> {
    ....
    n> # check for FAM events
    n> if ($fam->pending)
    n> {
    ....
    n> &process_fam_event($event);
    n> }
    n> }


    Looks like you are just running this in a busy loop, meaning you never
    pause to give the rest of the system a chance to catch up. Your process
    will be preempted by the kernel if necessary, but otherwise it will take
    up all the CPU (which is what you're seeing). Try a sleep(1) for a
    1-second pause at the end of the until() loop and see if your CPU
    utilization drops; it should.

    Ted
     
    Ted Zlatanov, Jan 24, 2008
    #9
  10. Guest

    wrote:


    >
    > # check for FAM events
    > if ($fam->pending)
    > {
    > my $event;
    > eval { $event = $fam->next_event; };
    >
    > if ($@)
    > {
    > print "FAM error: rebuilding cache...\n\n\n";
    > $fam = undef;
    > next;
    > }



    pending does not block, it returns immediately, and hence forms a tight
    loop. Since that isn't what you want to do, then don't do that. Skip
    the pending and just use the next_event. That is, replace the above with:

    # check for FAM events
    my $event = eval { $fam->next_event; };
    if ($@) {
    print "FAM error: rebuilding cache...\n\n\n";
    $fam = undef;
    next;
    };


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Jan 24, 2008
    #10
  11. Guest

    On Jan 24, 8:37 pm, wrote:
    > wrote:
    >
    > > # check for FAM events
    > > if ($fam->pending)
    > > {
    > > my $event;
    > > eval { $event = $fam->next_event; };

    >
    > > if ($@)
    > > {
    > > print "FAM error: rebuilding cache...\n\n\n";
    > > $fam = undef;
    > > next;
    > > }

    >
    > pending does not block, it returns immediately, and hence forms a tight
    > loop. Since that isn't what you want to do, then don't do that. Skip
    > the pending and just use the next_event. That is, replace the above with:
    >
    > # check for FAM events
    > my $event = eval { $fam->next_event; };
    > if ($@) {
    > print "FAM error: rebuilding cache...\n\n\n";
    > $fam = undef;
    > next;
    > };
    >
    > Xho
    >
    > --
    > --------------------http://NewsReader.Com/--------------------
    > The costs of publication of this article were defrayed in part by the
    > payment of page charges. This article must therefore be hereby marked
    > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    > this fact.



    Ya,,Thank U..
    As u said pending event is taking so much CPU usage more than(90%)...
    I removed the pending event so that the CPU usage came more or less
    equal to (5%).....

    Thank u

    Regards
    Nagandla
     
    , Jan 24, 2008
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Remy Cool
    Replies:
    1
    Views:
    448
    Remy Cool
    Aug 27, 2003
  2. Tobiah
    Replies:
    2
    Views:
    327
    Tobiah
    Sep 14, 2003
  3. Ben Weintraub

    Disabling modules using Modules/Setup

    Ben Weintraub, Sep 9, 2006, in forum: Python
    Replies:
    0
    Views:
    365
    Ben Weintraub
    Sep 9, 2006
  4. Peter Peyman Puk

    Importing v reloading modules modules

    Peter Peyman Puk, Mar 19, 2010, in forum: Python
    Replies:
    0
    Views:
    318
    Peter Peyman Puk
    Mar 19, 2010
  5. ImpalerCore
    Replies:
    0
    Views:
    890
    ImpalerCore
    Mar 10, 2011
Loading...

Share This Page