How to extract some lines from a large report and then reverse every character?

Discussion in 'Perl Misc' started by hzzc1012@gmail.com, Jan 12, 2006.

  1. Guest

    Hello, everyone:
    I'm using quite big data file as large as 1 G bytes, and it contain
    many lines, the longest of which will be about 1M byte.
    And now I need to read the last 100 lines, for example, from the file
    and reverse every character in the line adn then print it in another
    file ( for example, 12345000 in the original file will be 00054321 in
    the new file.)

    Now I read the data file into memory, find the end , and then output
    the character reversely to the output file, but I do find it very
    memory and time consuming

    I am a new comer to PERL, so I am not sure if there would be any other
    more efficient solution. And I guess there would be.
    Thank you in advance!



    Jacobus
    , Jan 12, 2006
    #1
    1. Advertising

  2. Re: How to extract some lines from a large report and then reverseevery character?

    wrote:
    > I'm using quite big data file as large as 1 G bytes, and it contain
    > many lines, the longest of which will be about 1M byte.
    > And now I need to read the last 100 lines, for example, from the file
    > and reverse every character in the line adn then print it in another
    > file ( for example, 12345000 in the original file will be 00054321 in
    > the new file.)
    >
    > Now I read the data file into memory, find the end , and then output
    > the character reversely to the output file, but I do find it very
    > memory and time consuming
    >
    > I am a new comer to PERL, so I am not sure if there would be any other
    > more efficient solution. And I guess there would be.


    Check out the File::ReadBackwards module.

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Gunnar Hjalmarsson, Jan 12, 2006
    #2
    1. Advertising

  3. Re: How to extract some lines from a large report and then reverseevery character?

    Gunnar Hjalmarsson wrote:
    > wrote:
    >> And now I need to read the last 100 lines, for example, from the file
    >> and reverse every character in the line

    >
    > Check out the File::ReadBackwards module.


    In addition to that:

    perldoc -q reverse
    perldoc -f reverse

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Gunnar Hjalmarsson, Jan 12, 2006
    #3
  4. Paul Lalli Guest

    wrote:
    > I'm using quite big data file as large as 1 G bytes, and it contain
    > many lines, the longest of which will be about 1M byte.
    > And now I need to read the last 100 lines, for example, from the file


    perldoc Tie::File
    perldoc -q "line in a file"

    > and reverse every character in the line


    perldoc -f reverse

    > adn then print it in another
    > file ( for example, 12345000 in the original file will be 00054321 in
    > the new file.)


    perldoc -f open
    perldoc -f print

    Paul Lalli
    Paul Lalli, Jan 12, 2006
    #4
  5. Xicheng Guest

    I guess the key part of the OP's problem is how to track the last 100
    lines which may contain upto 100MB spaces. File::ReadBackwards module
    looks good, which read the huge file backward and line by line. then I
    can reverse the contents of each line and "print" it into a new file,
    the new file now has 100 lines and in completely reversed order(lines,
    words, and characters).

    So what if I dont want to reverse lines, I just want to reverse the
    conents of each line. I have to do File::ReadBackwards again(no reverse
    contents though)??... This looks weird...

    Considering the huge size each line may contain, is there any better
    solutions besides the File::ReadBackwards modules??

    Many thanks
    Xicheng

    Gunnar Hjalmarsson wrote:
    > Gunnar Hjalmarsson wrote:
    > > wrote:
    > >> And now I need to read the last 100 lines, for example, from the file
    > >> and reverse every character in the line

    > >
    > > Check out the File::ReadBackwards module.

    >
    > In addition to that:
    >
    > perldoc -q reverse
    > perldoc -f reverse
    >
    > --
    > Gunnar Hjalmarsson
    > Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Xicheng, Jan 12, 2006
    #5
  6. J. Gleixner Guest

    Re: How to extract some lines from a large report and then reverseevery character?

    Xicheng wrote:
    > I guess the key part of the OP's problem is how to track the last 100
    > lines which may contain upto 100MB spaces. File::ReadBackwards module
    > looks good, which read the huge file backward and line by line. then I
    > can reverse the contents of each line and "print" it into a new file,
    > the new file now has 100 lines and in completely reversed order(lines,
    > words, and characters).
    >
    > So what if I dont want to reverse lines, I just want to reverse the
    > conents of each line. I have to do File::ReadBackwards again(no reverse
    > contents though)??... This looks weird...
    >
    > Considering the huge size each line may contain, is there any better
    > solutions besides the File::ReadBackwards modules??


    Please, stop top-posting....


    Non-perl answer, probably the fastest option:

    rev < tail -100 file > file.reverse

    For a more perlish answer, something similar to: (untested)

    my $file = 'blah';
    open( my $new_fh, '>', $file.new ) or die "Can't open $file.new: $!";
    open( my $tail, "/usr/bin/tail -100 $file |" ) or die "can't open tail: $!";
    while ( <$tail> ) { print $new_fh reverse; }
    close $tail;
    close $new_fh;
    J. Gleixner, Jan 12, 2006
    #6
  7. Guest

    "Xicheng" <> wrote:
    > I guess the key part of the OP's problem is how to track the last 100
    > lines which may contain upto 100MB spaces. File::ReadBackwards module
    > looks good, which read the huge file backward and line by line.


    For values of huge being less than 2 Gig (on a 32 bit machine). Otherwise,
    you have to hack the module by taking out the "use integer;" line. I've
    done this, and it seems to work for truly huge files, but I can't gaurantee
    I didn't introduce some subtle bug in the process.

    > then I
    > can reverse the contents of each line and "print" it into a new file,
    > the new file now has 100 lines and in completely reversed order(lines,
    > words, and characters).
    >
    > So what if I dont want to reverse lines, I just want to reverse the
    > conents of each line. I have to do File::ReadBackwards again(no reverse
    > contents though)??... This looks weird...


    Uh, wouldn't it be the other way around? You read normally (not with
    File::ReadBackwards), and reverse the contents of each line?

    > Considering the huge size each line may contain,


    He specified 1 MB. Are you calling that huge, or are you thinking of
    something bigger?

    > is there any better
    > solutions besides the File::ReadBackwards modules??


    File::ReadBackwards goes to some slight effort to reverse line order
    without reversing the contents of the lines. If you want to reverse both,
    then you could surely come up with something using seek and reverse that
    would be more efficient at doing that exact thing. But considering that
    you would have to go through the effort of actually coding it, maybe you
    wouldn't consider that option "better".

    Better is so context dependent, it makes little sense to discuss it in the
    abstract.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Jan 12, 2006
    #7
  8. robic0 Guest

    On 12 Jan 2006 07:15:28 -0800, wrote:

    >Hello, everyone:
    >I'm using quite big data file as large as 1 G bytes, and it contain
    >many lines, the longest of which will be about 1M byte.
    >And now I need to read the last 100 lines, for example, from the file
    >and reverse every character in the line adn then print it in another
    >file ( for example, 12345000 in the original file will be 00054321 in
    >the new file.)
    >
    >Now I read the data file into memory, find the end , and then output
    >the character reversely to the output file, but I do find it very
    >memory and time consuming
    >
    >I am a new comer to PERL, so I am not sure if there would be any other
    >more efficient solution. And I guess there would be.
    >Thank you in advance!
    >
    >
    >
    > Jacobus

    huh?

    olleH, enoyreve:
    I'm gnisu etiuq gib atad elif sa egral sa 1 G setyb, dna ti niatnoc
    ynam senil, eht tsegnol fo hcihw lliw eb tuoba M1 etyb.
    dnA won I deen ot daer eht tsal 001 senil, rof elpmaxe, morf eht elif
    dna esrever yreve retcarahc ni eht enil nda neht tnirp ti ni rehtona
    elif ( rof elpmaxe, 00054321 ni eht lanigiro elif lliw eb 12345000 ni
    eht wen elif.)

    woN I daer eht atad elif otni yromem, dnif eht dne , dna neht tuptuo
    eht retcarahc ylesrever ot eht tuptuo elif, tub I od dnif ti yrev
    yromem dna emit gnimusnoc

    I ma a wen remoc ot LREP, os I ma ton erus fi ereht dluow eb yna rehto
    erom tneiciffe noitulos. dnA I sseug ereht dluow eb.
    knahT uoy ni ecnavda!



    subocaJ
    robic0, Jan 16, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. dogbite
    Replies:
    4
    Views:
    693
    osmium
    Oct 10, 2003
  2. =?Utf-8?B?SXJ3YW5zeWFo?=
    Replies:
    4
    Views:
    2,452
    =?Utf-8?B?SXJ3YW5zeWFo?=
    Oct 30, 2007
  3. Replies:
    0
    Views:
    276
  4. Tristan Knowles
    Replies:
    8
    Views:
    170
    Brian Candler
    Jul 19, 2005
  5. Replies:
    4
    Views:
    539
    cwdjrxyz
    Jan 17, 2006
Loading...

Share This Page