How to extract some lines from a large report and then reverse every character?

[hzzc1012]

Hello, everyone:
I'm using quite big data file as large as 1 G bytes, and it contain
many lines, the longest of which will be about 1M byte.
And now I need to read the last 100 lines, for example, from the file
and reverse every character in the line adn then print it in another
file ( for example, 12345000 in the original file will be 00054321 in
the new file.)

Now I read the data file into memory, find the end , and then output
the character reversely to the output file, but I do find it very
memory and time consuming

I am a new comer to PERL, so I am not sure if there would be any other
more efficient solution. And I guess there would be.
Thank you in advance!



Jacobus

 

[Gunnar Hjalmarsson]

I'm using quite big data file as large as 1 G bytes, and it contain
many lines, the longest of which will be about 1M byte.
And now I need to read the last 100 lines, for example, from the file
and reverse every character in the line adn then print it in another
file ( for example, 12345000 in the original file will be 00054321 in
the new file.)

Now I read the data file into memory, find the end , and then output
the character reversely to the output file, but I do find it very
memory and time consuming

I am a new comer to PERL, so I am not sure if there would be any other
more efficient solution. And I guess there would be.

Check out the File::ReadBackwards module.

 

[Gunnar Hjalmarsson]

Check out the File::ReadBackwards module.

In addition to that:

perldoc -q reverse
perldoc -f reverse

 

[Paul Lalli]

I'm using quite big data file as large as 1 G bytes, and it contain
many lines, the longest of which will be about 1M byte.
And now I need to read the last 100 lines, for example, from the file

perldoc Tie::File
perldoc -q "line in a file"

and reverse every character in the line

perldoc -f reverse

adn then print it in another
file ( for example, 12345000 in the original file will be 00054321 in
the new file.)

perldoc -f open
perldoc -f print

Paul Lalli

 

[Xicheng]

I guess the key part of the OP's problem is how to track the last 100
lines which may contain upto 100MB spaces. File::ReadBackwards module
looks good, which read the huge file backward and line by line. then I
can reverse the contents of each line and "print" it into a new file,
the new file now has 100 lines and in completely reversed order(lines,
words, and characters).

So what if I dont want to reverse lines, I just want to reverse the
conents of each line. I have to do File::ReadBackwards again(no reverse
contents though)??... This looks weird...

Considering the huge size each line may contain, is there any better
solutions besides the File::ReadBackwards modules??

Many thanks
Xicheng

 

[J. Gleixner]

I guess the key part of the OP's problem is how to track the last 100
lines which may contain upto 100MB spaces. File::ReadBackwards module
looks good, which read the huge file backward and line by line. then I
can reverse the contents of each line and "print" it into a new file,
the new file now has 100 lines and in completely reversed order(lines,
words, and characters).

So what if I dont want to reverse lines, I just want to reverse the
conents of each line. I have to do File::ReadBackwards again(no reverse
contents though)??... This looks weird...

Considering the huge size each line may contain, is there any better
solutions besides the File::ReadBackwards modules??

Please, stop top-posting....


Non-perl answer, probably the fastest option:

rev < tail -100 file > file.reverse

For a more perlish answer, something similar to: (untested)

my $file = 'blah';
open( my $new_fh, '>', $file.new ) or die "Can't open $file.new: $!";
open( my $tail, "/usr/bin/tail -100 $file |" ) or die "can't open tail: $!";
while ( <$tail> ) { print $new_fh reverse; }
close $tail;
close $new_fh;

 

[xhoster]

I guess the key part of the OP's problem is how to track the last 100
lines which may contain upto 100MB spaces. File::ReadBackwards module
looks good, which read the huge file backward and line by line.

For values of huge being less than 2 Gig (on a 32 bit machine). Otherwise,
you have to hack the module by taking out the "use integer;" line. I've
done this, and it seems to work for truly huge files, but I can't gaurantee
I didn't introduce some subtle bug in the process.

then I
can reverse the contents of each line and "print" it into a new file,
the new file now has 100 lines and in completely reversed order(lines,
words, and characters).

So what if I dont want to reverse lines, I just want to reverse the
conents of each line. I have to do File::ReadBackwards again(no reverse
contents though)??... This looks weird...

Uh, wouldn't it be the other way around? You read normally (not with
File::ReadBackwards), and reverse the contents of each line?

Considering the huge size each line may contain,

He specified 1 MB. Are you calling that huge, or are you thinking of
something bigger?

is there any better
solutions besides the File::ReadBackwards modules??

File::ReadBackwards goes to some slight effort to reverse line order
without reversing the contents of the lines. If you want to reverse both,
then you could surely come up with something using seek and reverse that
would be more efficient at doing that exact thing. But considering that
you would have to go through the effort of actually coding it, maybe you
wouldn't consider that option "better".

Better is so context dependent, it makes little sense to discuss it in the
abstract.

Xho

 

[robic0]

Hello, everyone:
I'm using quite big data file as large as 1 G bytes, and it contain
many lines, the longest of which will be about 1M byte.
And now I need to read the last 100 lines, for example, from the file
and reverse every character in the line adn then print it in another
file ( for example, 12345000 in the original file will be 00054321 in
the new file.)

Now I read the data file into memory, find the end , and then output
the character reversely to the output file, but I do find it very
memory and time consuming

I am a new comer to PERL, so I am not sure if there would be any other
more efficient solution. And I guess there would be.
Thank you in advance!



Jacobus

huh?

olleH, enoyreve:
I'm gnisu etiuq gib atad elif sa egral sa 1 G setyb, dna ti niatnoc
ynam senil, eht tsegnol fo hcihw lliw eb tuoba M1 etyb.
dnA won I deen ot daer eht tsal 001 senil, rof elpmaxe, morf eht elif
dna esrever yreve retcarahc ni eht enil nda neht tnirp ti ni rehtona
elif ( rof elpmaxe, 00054321 ni eht lanigiro elif lliw eb 12345000 ni
eht wen elif.)

woN I daer eht atad elif otni yromem, dnif eht dne , dna neht tuptuo
eht retcarahc ylesrever ot eht tuptuo elif, tub I od dnif ti yrev
yromem dna emit gnimusnoc

I ma a wen remoc ot LREP, os I ma ton erus fi ereht dluow eb yna rehto
erom tneiciffe noitulos. dnA I sseug ereht dluow eb.
knahT uoy ni ecnavda!



subocaJ