Search a Large files backwards

Discussion in 'Perl Misc' started by mud_saisem, Mar 2, 2010.

  1. mud_saisem

    mud_saisem Guest

    Hi there,

    Could anybody tell me if there is a fast and efficient way to scan/
    print a large files (1Gb+) backwards ?

    I am not sure that the reverse function is the best way of doing it,
    particularly for really large files.

    I was thinking of maybe finding the file size in bytes and then
    running through a for loop in a descending order printing the file
    position with seek, but as you all properly know, that will not work
    very well.

    i would really like to be able to do this with out using any modules.

    Any ideas ?

    eg:

    #!/usr/bin/perl

    $filesize = -s $ARGV[0];

    open(FILE, "$ARGV[0]") or die "$!\n";

    for ($filesize = -s $ARGV[0]; $filesize >= 0; $filesize--)
    {
    seek(FILE,$filesize,0);
    $line = <FILE>;
    print $line;
    }

    close (FILE);

    But my output is as follows

    ../file_byte_size.pl /tmp/test.txt

    ..
    s.
    ds.
    rds.
    ards.
    wards.
    kwards.
    ckwards.
    ackwards.
    backwards.


    e
    le
    ile
    file
    file
    e file
    he file
    the file
    the file
    t the file
    nt the file
    int the file
    rint the file
    print the file
    print the file
    n print the file
    an print the file
    can print the file
    can print the file
    I can print the file

    f
    if
    if
    e if
    ee if
    see if
    see if
    o see if
    to see if
    to see if
    e to see if
    le to see if
    ile to see if
    file to see if

    t
    st
    est
    test
    test
    a test
    a test
    s a test
    is a test
    is a test
    s is a test
    is is a test
    his is a test
    This is a test
     
    mud_saisem, Mar 2, 2010
    #1
    1. Advertising

  2. mud_saisem

    Uri Guttman Guest

    >>>>> "ms" == mud saisem <> writes:

    ms> Could anybody tell me if there is a fast and efficient way to scan/
    ms> print a large files (1Gb+) backwards ?

    ms> I am not sure that the reverse function is the best way of doing it,
    ms> particularly for really large files.

    ms> I was thinking of maybe finding the file size in bytes and then
    ms> running through a for loop in a descending order printing the file
    ms> position with seek, but as you all properly know, that will not work
    ms> very well.

    ms> i would really like to be able to do this with out using any modules.

    why no modules? that is always a bad excuse. you can always cut/paste
    the code in the worst case or use local::lib to install them locally.

    so the answer is to use File::ReadBackwards

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Mar 2, 2010
    #2
    1. Advertising

  3. mud_saisem

    mud_saisem Guest

    On Mar 2, 5:32 pm, "Uri Guttman" <> wrote:
    > >>>>> "ms" == mud saisem <> writes:

    >
    >   ms> Could anybody tell me if there is a fast and efficient way to scan/
    >   ms> print a large files (1Gb+) backwards ?
    >
    >   ms> I am not sure that the reverse function is the best way of doing it,
    >   ms> particularly for really large files.
    >
    >   ms> I was thinking of maybe finding the file size in bytes and then
    >   ms> running through a for loop in a descending order printing the file
    >   ms> position with seek, but as you all properly know, that will not work
    >   ms> very well.
    >
    >   ms> i would really like to be able to do this with out using any modules.
    >
    > why no modules? that is always a bad excuse. you can always cut/paste
    > the code in the worst case or use local::lib to install them locally.
    >
    > so the answer is to use File::ReadBackwards
    >
    > uri
    >
    > --
    > Uri Guttman  ------    --------  http://www.sysarch.com--
    > -----  Perl Code Review , Architecture, Development, Training, Support ------
    > ---------  Gourmet Hot Cocoa Mix  ----  http://bestfriendscocoa.com---------



    The point of not using modules is because I would like to learn and
    understand how to achieve this first. Simply using the module because
    it is easy is not going to teach me anything.

    I don't understand why you say not using the module is a bad excuse ?
     
    mud_saisem, Mar 2, 2010
    #3
  4. mud_saisem

    Uri Guttman Guest

    >>>>> "ms" == mud saisem <> writes:

    >> why no modules? that is always a bad excuse. you can always cut/paste
    >> the code in the worst case or use local::lib to install them locally.
    >>
    >> so the answer is to use File::ReadBackwards


    ms> The point of not using modules is because I would like to learn
    ms> and understand how to achieve this first. Simply using the module
    ms> because it is easy is not going to teach me anything.

    you didn't say that. it helps to know your motivation.

    and you can learn how to do it well by studying the code in the
    module. writing it for yourself may not show you all that you need to
    learn to do it well. it happens to be written by me and it needs to do
    more than a simple loop to handle all the cases and be fast and easy to
    use.

    ms> I don't understand why you say not using the module is a bad
    ms> excuse ?

    because we get people here who say they can't use modules and have many
    poor excuses for why they can't. it is a common whine.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Mar 2, 2010
    #4
  5. mud_saisem

    mud_saisem Guest

    On Mar 2, 6:34 pm, "Uri Guttman" <> wrote:
    > >>>>> "ms" == mud saisem <> writes:

    >
    >   >> why no modules? that is always a bad excuse. you can always cut/paste
    >   >> the code in the worst case or use local::lib to install them locally.
    >   >>
    >   >> so the answer is to use File::ReadBackwards
    >
    >   ms> The point of not using modules is because I would like to learn
    >   ms> and understand how to achieve this first. Simply using the module
    >   ms> because it is easy is not going to teach me anything.
    >
    > you didn't say that. it helps to know your motivation.
    >
    > and you can learn how to do it well by studying the code in the
    > module. writing it for yourself may not show you all that you need to
    > learn to do it well. it happens to be written by me and it needs to do
    > more than a simple loop to handle all the cases and be fast and easy to
    > use.
    >
    >   ms> I don't understand why you say not using the module is a bad
    >   ms> excuse ?
    >
    > because we get people here who say they can't use modules and have many
    > poor excuses for why they can't. it is a common whine.
    >
    > uri
    >
    > --
    > Uri Guttman  ------    --------  http://www.sysarch.com--
    > -----  Perl Code Review , Architecture, Development, Training, Support ------
    > ---------  Gourmet Hot Cocoa Mix  ----  http://bestfriendscocoa.com---------



    Yes, I have no problems with modules or using modules. I just wanted
    to find away to do it without a module.
    I don't want to be a perl programer that has to rely on other
    programmers work or solutions, even though they are developed from
    very good programmers such as yourself.

    I must admit that I am quite surprised that a developer with your
    calibre is in google groups offering support and help to others.

    I will take your advise onboard and download your module and study the
    code. I am not sure how much of it I will follow, but I will give it a
    shot anyway.
     
    mud_saisem, Mar 2, 2010
    #5
  6. mud_saisem

    sreservoir Guest

    On 3/2/2010 3:11 PM, mud_saisem wrote:
    > Yes, I have no problems with modules or using modules. I just wanted
    > to find away to do it without a module.
    > I don't want to be a perl programer that has to rely on other
    > programmers work or solutions, even though they are developed from
    > very good programmers such as yourself.


    problem: you can't. somebody else wrote and probably compiled your perl.

    > I must admit that I am quite surprised that a developer with your
    > calibre is in google groups offering support and help to others.


    it's usenet. and most of us are volunteers.

    > I will take your advise onboard and download your module and study the
    > code. I am not sure how much of it I will follow, but I will give it a
    > shot anyway.


    it is comment-a-line code; I find it easy to understand, but a bit
    difficult to read. matter of opinion, though.

    --

    "Six by nine. Forty two."
    "That's it. That's all there is."
    "I always thought something was fundamentally wrong with the universe"
     
    sreservoir, Mar 2, 2010
    #6
  7. mud_saisem

    Uri Guttman Guest

    >>>>> "ms" == mud saisem <> writes:

    ms> On Mar 2, 6:34 pm, "Uri Guttman" <> wrote:
    >>
    >> because we get people here who say they can't use modules and have many
    >> poor excuses for why they can't. it is a common whine.


    ms> Yes, I have no problems with modules or using modules. I just
    ms> wanted to find away to do it without a module. I don't want to be
    ms> a perl programer that has to rely on other programmers work or
    ms> solutions, even though they are developed from very good
    ms> programmers such as yourself.

    relying on other people's work is the mark of a good programmer. you do
    it all the time just by using perl itself or your OS, etc. there is no
    way anyone could write up all the code they would want to use today. i
    know http well enough but i wouldn't replace LWP with my own code. i
    might modify it or offer patches or feature requests, etc.

    ms> I must admit that I am quite surprised that a developer with your
    ms> calibre is in google groups offering support and help to others.

    why not? larry himself used to inhabit this group a decade ago. there
    are plenty of top perl coders who help out in one of the many possible
    forums (usenet, web, irc, mailing lists, etc.).

    ms> I will take your advise onboard and download your module and study
    ms> the code. I am not sure how much of it I will follow, but I will
    ms> give it a shot anyway.

    the code is very straightforward and does exactly what you want with all
    the bugs fixed and edge conditions handled. it is better to learn from a
    working module than to struggle to write your own. the whole concept of
    reading lines backwards requires some deeper understanding of files and
    i/o which would take you a good while to learn if you don't know it
    yet.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Mar 2, 2010
    #7
  8. mud_saisem

    Uri Guttman Guest

    >>>>> "s" == sreservoir <> writes:

    s> On 3/2/2010 3:11 PM, mud_saisem wrote:
    >> Yes, I have no problems with modules or using modules. I just wanted
    >> to find away to do it without a module.
    >> I don't want to be a perl programer that has to rely on other
    >> programmers work or solutions, even though they are developed from
    >> very good programmers such as yourself.


    s> problem: you can't. somebody else wrote and probably compiled your perl.

    >> I must admit that I am quite surprised that a developer with your
    >> calibre is in google groups offering support and help to others.


    s> it's usenet. and most of us are volunteers.

    >> I will take your advise onboard and download your module and study the
    >> code. I am not sure how much of it I will follow, but I will give it a
    >> shot anyway.


    s> it is comment-a-line code; I find it easy to understand, but a bit
    s> difficult to read. matter of opinion, though.

    and which is more important - reading or understanding code? you have to
    realize i have been coding for 36 years and doing perl for 17. my style
    (commenting included) has evolved to make sure anyone can understand
    what i wrote with the least amount of effort. it may be more verbose
    than some but the comments are written to explain WHY i did something as
    the code is WHAT i did.

    here is a tidbit to learn:

    code (all the text in a source file) is a record of all your logical
    decisions that you made when implementing a program. it is the one place
    where your ego should shine forth. you want readers of your code to be
    impressed with how much care you put into it thinking about them.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Mar 2, 2010
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. SK
    Replies:
    15
    Views:
    793
    Karl Heinz Buchegger
    Jun 17, 2004
  2. Peter Merz
    Replies:
    1
    Views:
    584
  3. Abby Lee
    Replies:
    5
    Views:
    422
    Abby Lee
    Aug 2, 2004
  4. Toby DiPasquale

    String search backwards?

    Toby DiPasquale, Jan 24, 2006, in forum: Ruby
    Replies:
    1
    Views:
    130
    Matthew Moss
    Jan 24, 2006
  5. Brad
    Replies:
    9
    Views:
    162
Loading...

Share This Page