Re: Can I beat perl at grep-like processing speed?

Discussion in 'Python' started by Tim Smith, Dec 29, 2006.

  1. Tim Smith

    Tim Smith Guest

    you may not be able to beat perl's regex speed, but you can take some steps to speed up your python program using map and filter

    here's a modified python program that will do your search faster

    #!/usr/bin/env python
    import re
    r = re.compile(r'destroy', re.IGNORECASE)

    def stripit(x):
    return x.rstrip("\r\n")

    print "\n".join( map(stripit, filter(r.search, file('bigfile'))) )

    #time comparison on my machine
    real 0m0.218s
    user 0m0.210s
    sys 0m0.010s

    real 0m0.464s
    user 0m0.450s
    sys 0m0.010s

    #original time comparison on my machine

    real 0m0.224s
    user 0m0.220s
    sys 0m0.010s

    real 0m0.508s
    user 0m0.510s
    sys 0m0.000s

    also, if you replace the regex with a test like lambda x: x.lower().find("destroy") != -1, you will get really close to the speed of perl's (its possible perl will even take this shortcut when getting such a simple regex

    #here's the times when doing the search this way
    real 0m0.221s
    user 0m0.210s
    sys 0m0.010s

    real 0m0.277s
    user 0m0.280s
    sys 0m0.000s

    -- Tim

    -- On 12/29/06 "js " <> wrote:

    > Just my curiosity.
    > Can python beats perl at speed of grep-like processing?
    >
    > $ wget http://www.gutenberg.org/files/7999/7999-h.zip
    > $ unzip 7999-h.zip
    > $ cd 7999-h
    > $ cat *.htm > bigfile
    > $ du -h bigfile
    > du -h bigfile
    > 8.2M bigfile
    >
    > ---------- grep.pl ----------
    > #!/usr/local/bin/perl
    > open(F, 'bigfile') or die;
    >
    > while(<F>) {
    > s/[\n\r]+$//;
    > print "$_\n" if m/destroy/oi;
    > }
    > ---------- END ----------
    > ---------- grep.py ----------
    > #!/usr/bin/env python
    > import re
    > r = re.compile(r'destroy', re.IGNORECASE)
    >
    > for s in file('bigfile'):
    > if r.search(s): print s.rstrip("\r\n")
    > ---------- END ----------
    >
    > $ time perl grep.pl > pl.out; time python grep.py > py.out
    > real 0m0.168s
    > user 0m0.149s
    > sys 0m0.015s
    >
    > real 0m0.450s
    > user 0m0.374s
    > sys 0m0.068s
    > # I used python2.5 and perl 5.8.6
    > --
    > http://mail.python.org/mailman/listinfo/python-list
     
    Tim Smith, Dec 29, 2006
    #1
    1. Advertising

  2. In <>, Tim Smith
    wrote:

    > also, if you replace the regex with a test like lambda x:
    > x.lower().find("destroy") != -1, you will get really close to the speed
    > of perl's


    Testing for ``'destroy' in x.lower()`` should even be a little bit faster.

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Dec 29, 2006
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Peter
    Replies:
    4
    Views:
    428
    Scott Ellsworth
    Oct 18, 2004
  2. Headless

    Re: IE6 has beat me up

    Headless, Aug 6, 2003, in forum: HTML
    Replies:
    2
    Views:
    411
  3. Headless

    Re: IE6 has beat me up

    Headless, Aug 6, 2003, in forum: HTML
    Replies:
    0
    Views:
    408
    Headless
    Aug 6, 2003
  4. Neil White

    Re: IE6 has beat me up

    Neil White, Aug 6, 2003, in forum: HTML
    Replies:
    23
    Views:
    783
    William Tasso
    Aug 10, 2003
  5. js
    Replies:
    4
    Views:
    386
    Fredrik Lundh
    Jan 3, 2007
Loading...

Share This Page