alternative for ctime

[GRP]

hi,

I would like to write a script to grep some pattern in few set of log
files and alert it...

In order to avoid duplicate alert i plan to store the "FILENAME:ERR-
LINE-NUMBER:FILE-CREATION-TIME" in a flat file (colon delimiter) , so
that next time if the same pattern found in same line it will check
against the flat file and won't alert. At the same time if it happens
in different line number it wud check in flat file & the 2nd filed
would be different so it will alert.

Since i can't find a way to get ctime in unix (i tried with perl as
well), would be there any other way to achieve this or better logic?

Your help will be greatly appreciated..

thks

 

[Mart van de Wege]

Since i can't find a way to get ctime in unix (i tried with perl as
well), would be there any other way to achieve this or better logic?

What's wrong with

($ctime) = (stat($filename))[10];

?

Mart

 

[John W. Krahn]

I would like to write a script to grep some pattern in few set of log
files and alert it...

In order to avoid duplicate alert i plan to store the "FILENAME:ERR-
LINE-NUMBER:FILE-CREATION-TIME" in a flat file (colon delimiter) , so
that next time if the same pattern found in same line it will check
against the flat file and won't alert. At the same time if it happens
in different line number it wud check in flat file & the 2nd filed
would be different so it will alert.

Since i can't find a way to get ctime in unix (i tried with perl as
well), would be there any other way to achieve this or better logic?

You can get the ctime, that is easy, but you can't get the
"FILE-CREATION-TIME" because unix doesn't store that anywhere.




John

 

[GRP]

You can get the ctime, that is easy, but you can't get the
"FILE-CREATION-TIME" because unix doesn't store that anywhere.

John

Mart,
Thks for the feedback.

I've tried
($ctime) = (stat($filename))[10];

where that will give only file changed time not creation time

Source : http://perldoc.perl.org/functions/stat.html
10 ctime inode change time in seconds since the epoch (*)

John,
After google it , found Unix wont be storing creation time, just
thought of checking with you all would there be any other way to
attain my requirement?
Basically better logic

Cheers

 

[Mart van de Wege]

Note: Don't quote sigs, it's considered impolite.

Mart,
Thks for the feedback.

I've tried
($ctime) = (stat($filename))[10];

where that will give only file changed time not creation time

Ah.

I *was* wondering about that. Your use of creation time was inconsistent
with your asking for ctime. On Unix, these *are* different things, and
as pointed out, creation time is not saved.

Regards,

Mart

 

[sln]

You can get the ctime, that is easy, but you can't get the
"FILE-CREATION-TIME" because unix doesn't store that anywhere.

John

Mart,
Thks for the feedback.

I've tried
($ctime) = (stat($filename))[10];

where that will give only file changed time not creation time

Source : http://perldoc.perl.org/functions/stat.html
10 ctime inode change time in seconds since the epoch (*)

John,
After google it , found Unix wont be storing creation time, just
thought of checking with you all would there be any other way to
attain my requirement?
Basically better logic

Cheers

You can get the current time I assume but why do you need
any time or timestamp at all? You do have modify time in unix
don't you? Thats the creation time the first time.

So, forget about the time. Post some algo that you want this to do.

-sln

 

[Seebs]

Some Unices do (all the BSDs, for instance),

??

I don't see anything in stat(2), where should I be looking?

-s

 

[Seebs]

In my stat(2) (FreeBSD 7.3-RELEASE) I have

| st_birthtime Time when the inode was created.

It's possible this is just a FreeBSD thing, but I thought it was a
4.4-ism.

Don't see it in NetBSD, so it's not an original 4.4ism. Could be that
newer *BSD have it, but it's not in 4.4. Wish the man page would give
history for that kind of thing.

-s

 

[Randal L. Schwartz]

Ben> In my stat(2) (FreeBSD 7.3-RELEASE) I have

Ben> | st_birthtime Time when the inode was created.

Ben> It's possible this is just a FreeBSD thing, but I thought it was a
Ben> 4.4-ism.

Definitely not in the One True Unix, therefore, not portable.

 

[John Bokma]

hi,

I would like to write a script to grep some pattern in few set of log
files and alert it...

In order to avoid duplicate alert i plan to store the "FILENAME:ERR-
LINE-NUMBER:FILE-CREATION-TIME" in a flat file (colon delimiter) , so
that next time if the same pattern found in same line it will check
against the flat file and won't alert. At the same time if it happens
in different line number it wud check in flat file & the 2nd filed
would be different so it will alert.

It might help if you describe when a clash can occur, i.e. why do you
need the inode creation time, and why is filename:err-line-no not unique
(I can think of reasons, but want to know the ones in your situation.)

 

[GRP]

It might help if you describe when a clash can occur, i.e. why do you
need the inode creation time, and why is filename:err-line-no not unique
(I can think of reasons, but want to know the ones in your situation.)

Hi John,

For example lets say i run my script at 10am where i found the error
pattern in the line number 100, i will store in a flat file in this
format "FILENAME:LINE-NUMBER" and the script will alert a msg to
tivoli. In the next run at 10.30am, assume there is no error found
other than line number 100, my script should not alert it since it's
already alerted. In order to achieve this along with "FILENAME:LINE-
NUMBER" i need to store some other value which is constant (ex. file
creation time , not change time), so that i can compare against it and
if found different i assume error happened in different line.

Since there is no way to capture file-creation-time i;m looking for
some other way to achieve this or probably you can suggest better
logic to achieve this...

thanks

 

[John Bokma]

For example lets say i run my script at 10am where i found the error
pattern in the line number 100, i will store in a flat file in this
format "FILENAME:LINE-NUMBER" and the script will alert a msg to
tivoli. In the next run at 10.30am, assume there is no error found
other than line number 100, my script should not alert it since it's
already alerted.

Clear, but FILENAME:LINE-NUMBER is already sufficient to stop that from
happening, unless you forgot to mention something.

In order to achieve this along with "FILENAME:LINE-
NUMBER" i need to store some other value which is constant (ex. file
creation time , not change time), so that i can compare against it and
if found different i assume error happened in different line.

Do the lines in the file have a time stamp?

Since there is no way to capture file-creation-time i;m looking for
some other way to achieve this or probably you can suggest better
logic to achieve this...

I am still missing important information. Give an example of a
possible clash.

 

[GRP]

Clear, but FILENAME:LINE-NUMBER is already sufficient to stop that from
happening, unless you forgot to mention something.


Do the lines in the file have a time stamp?


I am still missing important information. Give an example of a
possible clash.

here it goes....

Script will check for pattern in few log files (lets say its looking
for pattern "Out-of-Memory" in /u1/app/log/WL.log & /u2/app/log/WL.log
etc.. ).

These WL.log files from different directories will delete & overwrite
once it reach 10mb. It may happen in a day or in few days.

Since i need a maintain a flat file or Database like below
<directory-and-file-name>:<err-line-number>:<some-other-constant-value-
example.filecreation-time>

the reason being maintaining a flat/database file because if the same
pattern "Out-of-Memory" found in the same line, it should check in the
flat file & if the entry found it wont alert. Whereas the same pattern
found in a different line , it will not be available in the flat file
and alert will trigger.

Since the log files may rotate often i need to tag someother value
along with filename and linenumber in the flat file which is key
value. I can't simply store <directory-and-file-name>:<err-line-
number> in flat file , becos once the file recreated and if the error
comes in the same line, it wont get alerted , which is wrong. In order
to overcome i taught of storing file creation time, which is not
possible.

Hope i clarified now....

thks

 

[Dr.Ruud]

I would like to write a script to grep some pattern in few set of log
files and alert it...

In order to avoid duplicate alert i plan to store the "FILENAME:ERR-
LINE-NUMBER:FILE-CREATION-TIME" in a flat file (colon delimiter) , so
that next time if the same pattern found in same line it will check
against the flat file and won't alert. At the same time if it happens
in different line number it wud check in flat file & the 2nd filed
would be different so it will alert.

Since i can't find a way to get ctime in unix (i tried with perl as
well), would be there any other way to achieve this or better logic?

You can use git.

 

[sln]

here it goes....

Script will check for pattern in few log files (lets say its looking
for pattern "Out-of-Memory" in /u1/app/log/WL.log & /u2/app/log/WL.log
etc.. ).

These WL.log files from different directories will delete & overwrite
once it reach 10mb. It may happen in a day or in few days.

Since i need a maintain a flat file or Database like below
<directory-and-file-name>:<err-line-number>:<some-other-constant-value-
example.filecreation-time>

the reason being maintaining a flat/database file because if the same
pattern "Out-of-Memory" found in the same line, it should check in the
flat file & if the entry found it wont alert. Whereas the same pattern
found in a different line , it will not be available in the flat file
and alert will trigger.

Since the log files may rotate often i need to tag someother value
along with filename and linenumber in the flat file which is key
value. I can't simply store <directory-and-file-name>:<err-line-
number> in flat file , becos once the file recreated and if the error
comes in the same line, it wont get alerted , which is wrong. In order
to overcome i taught of storing file creation time, which is not
possible.

Hope i clarified now....

Are the log files continually appended to after and up until
the next 10 mb limit, where it is deleted?

If thats the case, then you need some other item for distinction.
The odds of this hapening should be very low though, unless the logs
are mechanical, and predictably repeatitive.

Its hard to believe there is no timestamp in the log. Why is that?

Finally, if you could timestamp the file in some way, how would
you age the entries in the flat file?
You either need the creation time of the log, or need to have whoever
creates the log to include a timestamp in the log of each entry or just
a single timestamp at the top.

-sln

 

[sln]

Are the log files continually appended to after and up until
the next 10 mb limit, where it is deleted?

If thats the case, then you need some other item for distinction.
The odds of this hapening should be very low though, unless the logs
are mechanical, and predictably repeatitive.

Its hard to believe there is no timestamp in the log. Why is that?

Finally, if you could timestamp the file in some way, how would
you age the entries in the flat file?
You either need the creation time of the log, or need to have whoever
creates the log to include a timestamp in the log of each entry or just
a single timestamp at the top.

One more thing. If the logs are being deleted, starting fresh,
how do you know when that happens?
If you have your alert script running on a timed interval,
how do you know that you didn't miss some information between the
last time it checked and when the log was deleted? How do you
get a last chance check?

Sounds like problems to me.

-sln

 

[Keith Thompson]

Script will check for pattern in few log files (lets say its looking
for pattern "Out-of-Memory" in /u1/app/log/WL.log & /u2/app/log/WL.log
etc.. ).

These WL.log files from different directories will delete & overwrite
once it reach 10mb. It may happen in a day or in few days.

Since i need a maintain a flat file or Database like below
<directory-and-file-name>:<err-line-number>:<some-other-constant-value-
example.filecreation-time>

the reason being maintaining a flat/database file because if the same
pattern "Out-of-Memory" found in the same line, it should check in the
flat file & if the entry found it wont alert. Whereas the same pattern
found in a different line , it will not be available in the flat file
and alert will trigger.

Since the log files may rotate often i need to tag someother value
along with filename and linenumber in the flat file which is key
value. I can't simply store <directory-and-file-name>:<err-line-
number> in flat file , becos once the file recreated and if the error
comes in the same line, it wont get alerted , which is wrong. In order
to overcome i taught of storing file creation time, which is not
possible.

Do you have any control over what's written to the files? If so, it
would be easy enough to write a timestamp and/or other information on
the first line.

If not, checking the inode number (stat("filename"))[1] should detect
when a new copy of the file is created. The combination of device and
inode number (elements 0 and 1 of the result of stat()) should uniquely
identify the file, but just inode number is probably enough for your
purposes.

Complex file system setups can mess this up, with the same file
appearing on different devices, but that's probably not going to be an
issue for you.

 

[Alan Curry]

[the question boiled down to "how do I know if a log file has been rotated
since the last time I looked at it?"]

|
|Do you have any control over what's written to the files? If so, it
|would be easy enough to write a timestamp and/or other information on
|the first line.

That's the best answer (other than "switch to FreeBSD so you can use
st_birthtime")

|
|If not, checking the inode number (stat("filename"))[1] should detect
|when a new copy of the file is created. The combination of device and
|inode number (elements 0 and 1 of the result of stat()) should uniquely
|identify the file, but just inode number is probably enough for your
|purposes.

device.inode is unique at a single point in time, but the same combination
can reappear later, after the first file is gone.

Rotated log files can get deleted eventually, and before that they can get
gzipped, which will result in a .gz file with a new inode number and the
original inode number becoming available for reuse.

It wouldn't be surprising to see /var/log/mail get the same inode number it
had 2 weeks ago, when it had the contents now present in /var/log/mail.2.gz