remove specific line from al ltext fiels in dir?

[Snail]

I've been trying to find a way to remove a line containing a specific
string.

In my case, I have a dir full of logs files and I want to remove all
lines containing "127.0.0.1" or "localhost", because those lines take up
more than 95% of the text in the logs making them rather big. Removing
those lines will make our logs much easier to read.

I found the opendir and readdir functions, but what if I want to
recurse. I suppose I could just while(<LOGFIE>) loop over each line and
check, but I'm not sure how to erase the line.

Or is there a better way to do this?

Thank you for any advise and help.

(PS: Running on a Linux platform, with Perl 5.6.1)

 

[Paul Lalli]

I've been trying to find a way to remove a line containing a specific
string.

Have you read
perldoc -q "delete a line"
?

In my case, I have a dir full of logs files and I want to remove all
lines containing "127.0.0.1" or "localhost", because those lines take up
more than 95% of the text in the logs making them rather big. Removing
those lines will make our logs much easier to read.

I found the opendir and readdir functions

I wouldn't even bother doing those explicitly. I would suggest just
using a one-liner using the -n and -i switches...

perl -ni -e'print unless /localhost|127\.0\.0\.1/' *.log

(Read about those two switches in
perldoc perlrun
)

, but what if I want to recurse.

When you think "recurse" in Perl, think "File::Find", which is a
standard module. There is also the non-standard File::Find::Rule
available on CPAN which offers a theoretically easier to understand
syntax.

perldoc File::Find

I suppose I could just while(<LOGFIE>) loop over each line and
check, but I'm not sure how to erase the line.

You can't. At least, not like you're thinking. You'd need to save a
copy of the original under a different name, open a new file for output
under the original name, and print all the lines from the saved copy
that you don't want to "delete". (This is precisely what -n and -i
help you with, from above).

Or is there a better way to do this?

Annoyingly, I'm not thinking of any particularly quick and easy ways of
combining the -ni approach with the File::Find aproach. I can think of
several ways, just no "good" ones:
* in File::Find's &wanted subroutine, spawn an external perl process
that uses -ni
* use BEGIN{} and END{} blocks in the "one-liner" to bring File::Find
into the -ni method.
* do the recursion external to perl, with the perl -ni process in the
body of a
for ${find . -name *.log -print}; do ... block
* set @ARGV to the list of files found from File::Find::find, set $^I,
and then start your while (<>) loop.

Any of the gurus know of any better way of recursing using the -ni
switches?

Paul Lalli

 

[Paul Lalli]

Annoyingly, I'm not thinking of any particularly quick and easy ways of
combining the -ni approach with the File::Find aproach. I can think of
several ways, just no "good" ones:
* in File::Find's &wanted subroutine, spawn an external perl process
that uses -ni
* use BEGIN{} and END{} blocks in the "one-liner" to bring File::Find
into the -ni method.
* do the recursion external to perl, with the perl -ni process in the
body of a
for ${find . -name *.log -print}; do ... block
* set @ARGV to the list of files found from File::Find::find, set $^I,
and then start your while (<>) loop.

Any of the gurus know of any better way of recursing using the -ni
switches?

Thinking more about this, I think my last option there is probably the
best I can come up with. (Although I was very unclear about how @ARGV
is set: find doesn't return anything. You have to populate @ARGV
within the &wanted function).

Here's my attempt to do what I think you're looking for (replace
'logdir' with whatever directory holds your logs):

#!/usr/bin/perl
use strict;
use warnings;
use File::Find;

local @ARGV;

sub wanted {
push @ARGV, $File::Find::name if /\.log$/ and -f;
}

find (\&wanted, 'logdir');

if (@ARGV){
print "Files to process: \n", join("\n", @ARGV), "\n";
local $^I = '.bkp';
while (<>){
print unless /localhost|127\.0\.0\.1/;
}
} else {
print "No files to process!\n";
}

__END__

 

[Tad McClellan]

I've been trying to find a way to remove a line containing a specific
string.

more than 95% of the text in the logs making them rather big.

^^^^

Are these "live" logs? That is, will logging writes be happening
when you run your cleanup program?

If so, then you better work file locking into the mix.

 

[Snail]

^^^^

Are these "live" logs? That is, will logging writes be happening
when you run your cleanup program?

If so, then you better work file locking into the mix.

Yes and no. They are live during the day, but at night, for 10 minutes,
its like a "maintence" time. My plan was to setup a cron job to strip
the uneeded lines from the logs. The server daemon provides no way (and
even worse, no source code :< ) to customize it's logs, thats why I
wanted to take matters into my own hands.

`perl -ni -e'print unless /localhost|127\.0\.0\.1/' *.log`

Works almsot perfectly. If I run it from my shell window I see annoying

"Can't do inplace edit: backup is not a regular file."

type messages when it encouters a directory, and I can no way to supress
them, even with 2>&1 appended to the end.

`perl -ni -e'print unless /localhost|127\.0\.0\.1/' *.log 2>&1`

I suppose it doesn't matter as this will be running from a cron job, but
I think it would run a little after if there was a way to just skip dirs
all together. Maybe it would be better to just write a full perl script
using readdir, and checking each one with -f to see if it's a file.

Unless there is a way to achieve this on the cmd line?

Thanks again.

 

[gimme_this_gimme_that]

Perl is the wrong tool for this task.

Try :

grep -v localhost logfile.log | grep -v 127.0.0.1 > new logfile.log

Note that grep has options to use a file so you can put
localhost, 127.0.0.1 etc into a file instead of doing a bunch of pipes.

 

[Tad McClellan]

`perl -ni -e'print unless /localhost|127\.0\.0\.1/' *.log`
Works almsot perfectly. If I run it from my shell window I see annoying

"Can't do inplace edit: backup is not a regular file."

type messages when it encouters a directory, and I can no way to supress
them, even with 2>&1 appended to the end.

`perl -ni -e'print unless /localhost|127\.0\.0\.1/' *.log 2>&1`

^ ^
^ ^

What's with the (shell) backticks?

Is that really how you are calling it?

Leave them off, or do the redirection outside of them, but you don't
what the annoying messages in your log file either, so make it 2>/dev/null
or some such.

I suppose it doesn't matter as this will be running from a cron job, but
I think it would run a little after if there was a way to just skip dirs
all together. Maybe it would be better to just write a full perl script
using readdir, and checking each one with -f to see if it's a file.

Unless there is a way to achieve this on the cmd line?

You can muck about with @ARGV before you let -n's while(<>)
loop look at it (line wrapped for posting):

perl -ni -e 'BEGIN{@ARGV = grep -f, @ARGV}
print unless /localhost|\Q127.0.0.1/' *.log


Or, even better, don't make subdirectories with silly names
that match *.log.

 

[Sherm Pendley]

You could.

I could what?

grep -v is what a sysadmin would use.

Use for what?

A perl one liner is what a perl enthusiast would use.

Use for what?

Have you read the posting guidelines for this group, and the Google Groups
guide to usenet netiquette? You've been asked many times now to quote the
messages you're replying to as appropriate. Why aren't you doing it?

sherm--

 

[Jürgen Exner]

Perl is the wrong tool for this task.

For what task?
Please provide enough context such that people have a chance to know what
you are talking about

Try :
grep -v localhost logfile.log | grep -v 127.0.0.1 > new logfile.log
Note that grep has options to use a file so you can put
localhost, 127.0.0.1 etc into a file instead of doing a bunch of
pipes.

And why shouldn't you be able to do this in Perl one-liner?

jue

 

[gimme_this_gimme_that]

You could.

grep -v is what a sysadmin would use.

A perl one liner is what a perl enthusiast would use.

 

[Tintin]

[Please quote appropriate context. Mind you, I know you are going to
deliberately ignore this request]

Perl is the wrong tool for this task.
Rubbish.


Try :

grep -v localhost logfile.log | grep -v 127.0.0.1 > new logfile.log

Note that grep has options to use a file so you can put
localhost, 127.0.0.1 etc into a file instead of doing a bunch of pipes.

Note that your "solution" doesn't do what the OP asked (although you didn't
quote any context, so it's hard to see what the OP did actually ask).

If you really did want to do a non Perl solution, it would be:

egrep -v "localhost|127\.0\.0\.1" logfile.log >/tmp/$$ && mv /tmp/$$
logfile.log

 

[Ian Wilson]

Perl is the wrong tool for this task.

Try :

grep -v localhost logfile.log | grep -v 127.0.0.1 > new logfile.log

I usually need to quote filenames with spaces in them.

Note that grep has options to use a file so you can put
localhost, 127.0.0.1 etc into a file instead of doing a bunch of pipes.

grep | grep is the wrong tool for this task.

Try :

egrep -v "(localhost|127.0.0.1)" logfile.log > newlogfile.log