simple regex benchmark inaccuracies

Discussion in 'Perl Misc' started by it_says_BALLS_on_your forehead, Feb 24, 2006.

  1. these are the results:

    smmk94<WSP1>$ try_regex2.pl
    Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
    greedy: 0 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @
    1235595.10/s (n=1260307)
    stingy: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @
    1172432.11/s (n=1277951)
    Rate stingy greedy
    stingy 1172432/s -- -5%
    greedy 1235595/s 5% --
    smmk94<WSP1>$ try_regex2.pl
    Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
    greedy: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @
    1180321.70/s (n=1251141)
    stingy: 1 wallclock secs ( 1.07 usr + 0.01 sys = 1.08 CPU) @
    1261175.00/s (n=1362069)
    Rate greedy stingy
    greedy 1180322/s -- -6%
    stingy 1261175/s 7% --
    smmk94<WSP1>$ try_regex2.pl
    Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
    greedy: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @
    1180321.70/s (n=1251141)
    stingy: 2 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @
    1260306.67/s (n=1323322)
    Rate greedy stingy
    greedy 1180322/s -- -6%
    stingy 1260307/s 7% --
    smmk94<WSP1>$ try_regex2.pl
    Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
    greedy: 1 wallclock secs ( 1.16 usr + 0.00 sys = 1.16 CPU) @
    1248870.69/s (n=1448690)
    stingy: 2 wallclock secs ( 1.10 usr + 0.00 sys = 1.10 CPU) @
    1137400.91/s (n=1251141)
    Rate stingy greedy
    stingy 1137401/s -- -9%
    greedy 1248871/s 10% --



    and here is the code:
    =====
    #!/usr/local/bin/perl

    use strict; use warnings;
    use Benchmark qw(cmpthese);

    sub alt {
    my $input =
    'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    return 1 if $input =~ /campaign\w*_fulltop\.gif/;
    }

    sub alt2 {
    my $input =
    'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
    }

    cmpthese(-1, {
    greedy => \&alt,
    stingy => \&alt2,
    });



    ....any comments on why there's a flipflop?
    it_says_BALLS_on_your forehead, Feb 24, 2006
    #1
    1. Advertising

  2. it_says_BALLS_on_your forehead

    Guest

    "it_says_BALLS_on_your forehead" <> wrote:
    > these are the results:
    >
    > smmk94<WSP1>$ try_regex2.pl
    > Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
    > greedy: 0 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @
    > 1235595.10/s (n=1260307)
    > stingy: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @
    > 1172432.11/s (n=1277951)
    > Rate stingy greedy
    > stingy 1172432/s -- -5%
    > greedy 1235595/s 5% --


    What version of Benchmark are you running?

    ....

    >
    > ...any comments on why there's a flipflop?


    What is there to say? The difference between them is negligible, the
    results of Benchmark are subject to random error, systematic error, and
    correlation between errors--just like every other measurement is--and you
    are not running them long enough to average out even the error components
    that can be averaged out.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Feb 24, 2006
    #2
    1. Advertising

  3. wrote:

    >
    > What version of Benchmark are you running?



    $VERSION = 1.00;

    with Perl 5.6.1

    >
    > ...
    >
    > >
    > > ...any comments on why there's a flipflop?

    >
    > What is there to say? The difference between them is negligible, the
    > results of Benchmark are subject to random error, systematic error, and
    > correlation between errors--just like every other measurement is--and you
    > are not running them long enough to average out even the error components
    > that can be averaged out.


    perhaps, although the disparity seems too large to attribute to those
    errors. i could understand if the difference was within 5%, but in one
    case, greedy was 10% faster than stingy, and in the other, stingy was
    7% faster than greedy. i understand that there are many factors
    involved, but still i had hoped that the results from Benchmark would
    prove more reliable and consistent. FYI, i ran again with -5 as the
    first param to cmpthese, and got these results:

    smmk94<WSP1>$ try_regex2.pl
    Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
    greedy: 28 wallclock secs ( 5.16 usr + -0.02 sys = 5.14 CPU) @
    1219724.32/s (n=6269383)
    stingy: 19 wallclock secs ( 5.09 usr + 0.01 sys = 5.10 CPU) @
    1334927.45/s (n=6808130)
    Rate greedy stingy
    greedy 1219724/s -- -9%
    stingy 1334927/s 9% --
    it_says_BALLS_on_your forehead, Feb 24, 2006
    #3
  4. it_says_BALLS_on_your forehead

    Guest

    it_says_BALLS_on_your forehead wrote:

    > these are the results:


    > stingy 1172432/s -- -5%
    > greedy 1235595/s 5% --


    > greedy 1180322/s -- -6%
    > stingy 1261175/s 7% --


    > ...any comments on why there's a flipflop?


    Probably because a large portion of what you're comparing is my $input
    = 'blah'.

    Ed


    > sub alt {
    > my $input =
    > 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    > return 1 if $input =~ /campaign\w*_fulltop\.gif/;
    > }
    >
    > sub alt2 {
    > my $input =
    > 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    > return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
    > }
    , Feb 24, 2006
    #4
  5. wrote:
    > it_says_BALLS_on_your forehead wrote:
    >
    > > these are the results:

    >
    > > stingy 1172432/s -- -5%
    > > greedy 1235595/s 5% --

    >
    > > greedy 1180322/s -- -6%
    > > stingy 1261175/s 7% --

    >
    > > ...any comments on why there's a flipflop?

    >
    > Probably because a large portion of what you're comparing is my $input
    > = 'blah'.
    >
    > Ed


    the input is a real data record, and since that is constant, would not
    affect the results of the benchmark. Xho probably has the right of it,
    although i still have my doubts. i don't think that the Benchmark
    module can be completely absolved of responsibility for the
    discrepancies in the results...
    it_says_BALLS_on_your forehead, Feb 24, 2006
    #5
  6. it_says_BALLS_on_your forehead

    Guest

    it_says_BALLS_on_your forehead wrote:
    > wrote:
    > > it_says_BALLS_on_your forehead wrote:


    > > > ...any comments on why there's a flipflop?

    > >
    > > Probably because a large portion of what you're comparing is my $input
    > > = 'blah'.
    > >
    > > Ed

    >
    > the input is a real data record, and since that is constant, would not
    > affect the results of the benchmark. Xho probably has the right of it,
    > although i still have my doubts. i don't think that the Benchmark
    > module can be completely absolved of responsibility for the
    > discrepancies in the results...


    If you try it, you'll see. If you run a comparison of your tested
    function calls without the my declaration against those with the my
    declaration, the functions run 60% faster. You can see below that the
    RE resolutions are basically the same, whereas the exact same function
    (with a my declaration) yields the type of flip flop you're seeing.
    I'd assume the flip flop simply reflects the variances of memory
    management (for handling the my declarations) on the machine at the
    time of execution.

    Ed


    $ cat checkit.pl
    #!/usr/local/bin/perl

    use strict; use warnings;
    use Benchmark qw(cmpthese);

    sub my_call {
    my $input =
    'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    }

    cmpthese(-5, {
    once => \&my_call,
    twice => \&my_call,
    });

    my $input =
    'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';

    sub greedy {
    return 1 if $input =~ /campaign\w*_fulltop\.gif/;
    }

    sub stingy {
    return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
    }

    cmpthese(-5, {
    greedy => \&greedy,
    stingy => \&stingy,
    });
    $ ./checkit.pl
    Benchmark: running once, twice, each for at least 5 CPU seconds...
    once: 6 wallclock secs ( 5.55 usr + 0.00 sys = 5.55 CPU) @
    2945391.71/s (n=16346924)
    twice: 5 wallclock secs ( 5.24 usr + 0.00 sys = 5.24 CPU) @
    3180811.45/s (n=16667452)
    Rate once twice
    once 2945392/s -- -7%
    twice 3180811/s 8% --
    Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
    greedy: 5 wallclock secs ( 5.17 usr + 0.00 sys = 5.17 CPU) @
    3359500.00/s (n=17368615)
    stingy: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @
    3320958.89/s (n=17368615)
    Rate stingy greedy
    stingy 3320959/s -- -1%
    greedy 3359500/s 1% --
    $
    , Feb 25, 2006
    #6
  7. wrote:
    > it_says_BALLS_on_your forehead wrote:
    > > wrote:
    > > > it_says_BALLS_on_your forehead wrote:

    >
    > > > > ...any comments on why there's a flipflop?
    > > >
    > > > Probably because a large portion of what you're comparing is my $input
    > > > = 'blah'.
    > > >
    > > > Ed

    > >
    > > the input is a real data record, and since that is constant, would not
    > > affect the results of the benchmark. Xho probably has the right of it,
    > > although i still have my doubts. i don't think that the Benchmark
    > > module can be completely absolved of responsibility for the
    > > discrepancies in the results...

    >
    > If you try it, you'll see. If you run a comparison of your tested
    > function calls without the my declaration against those with the my
    > declaration, the functions run 60% faster. You can see below that the
    > RE resolutions are basically the same, whereas the exact same function
    > (with a my declaration) yields the type of flip flop you're seeing.
    > I'd assume the flip flop simply reflects the variances of memory
    > management (for handling the my declarations) on the machine at the
    > time of execution.
    >
    > Ed
    >
    >
    > $ cat checkit.pl
    > #!/usr/local/bin/perl
    >
    > use strict; use warnings;
    > use Benchmark qw(cmpthese);
    >
    > sub my_call {
    > my $input =
    > 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    > }
    >
    > cmpthese(-5, {
    > once => \&my_call,
    > twice => \&my_call,
    > });
    >
    > my $input =
    > 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
    >
    > sub greedy {
    > return 1 if $input =~ /campaign\w*_fulltop\.gif/;
    > }
    >
    > sub stingy {
    > return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
    > }
    >
    > cmpthese(-5, {
    > greedy => \&greedy,
    > stingy => \&stingy,
    > });
    > $ ./checkit.pl
    > Benchmark: running once, twice, each for at least 5 CPU seconds...
    > once: 6 wallclock secs ( 5.55 usr + 0.00 sys = 5.55 CPU) @
    > 2945391.71/s (n=16346924)
    > twice: 5 wallclock secs ( 5.24 usr + 0.00 sys = 5.24 CPU) @
    > 3180811.45/s (n=16667452)
    > Rate once twice
    > once 2945392/s -- -7%
    > twice 3180811/s 8% --
    > Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
    > greedy: 5 wallclock secs ( 5.17 usr + 0.00 sys = 5.17 CPU) @
    > 3359500.00/s (n=17368615)
    > stingy: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @
    > 3320958.89/s (n=17368615)
    > Rate stingy greedy
    > stingy 3320959/s -- -1%
    > greedy 3359500/s 1% --
    > $


    that is very interesting. i based my method of benchmarking on Randal
    Schwartz's, where he assigns the variable a value inside the
    subroutine, so i thought that would be good:

    http://www.stonehenge.com/merlyn/UnixReview/col49.html

    ....however, based on your results, maybe that's a bad idea?
    it_says_BALLS_on_your_forehead, Feb 25, 2006
    #7
  8. it_says_BALLS_on_your forehead

    Guest

    [ snipped discussion of how benchmarking functions with my declarations
    can cause varying benchmark results due to varying execution times for
    my() ]

    it_says_BALLS_on_your_forehead wrote:

    > that is very interesting. i based my method of benchmarking on Randal
    > Schwartz's, where he assigns the variable a value inside the
    > subroutine, so i thought that would be good:
    >
    > http://www.stonehenge.com/merlyn/UnixReview/col49.html
    >
    > ...however, based on your results, maybe that's a bad idea?


    My rule of thumb is to benchmark exactly what I'm trying to compare -
    nothing more, and nothing less. So if I wanted to compare regexes,
    then that's all I'd put into the comparison (as discussed previously in
    this thread). It's like if I want to compare driving times from
    Chicago to LA versus Chicago to Seattle. One way would be comparing
    driving times from Miami to Chicago to LA versus Miami to Chicago to
    Seattle - but why include the duplicated Miami to Chicago leg in there?
    Remove the common factors and let the desired comparison stand alone.

    So, yes, (in this one instance) I disagree with what Randal did there.
    Note, though, that his results are such that a five to ten percent
    swing wouldn't make any real difference. In your case, that five to
    ten percent swing was effectively the entire difference.

    As an aside, part of the issue in your code is that what you are
    comparing is so quick, the my declarations contribute a significant
    portion of the run time. Were you comparing something that took a good
    bit longer, then the my declarations would be a minimal fraction of the
    run time, and the run time variances in my() would be trivial compared
    to the rest of the compared code.

    Ed
    , Mar 4, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Bowen Chiu
    Replies:
    1
    Views:
    535
    Skip Montanaro
    Aug 9, 2003
  2. Replies:
    1
    Views:
    881
    Andrew Dalke
    Jan 12, 2004
  3. Replies:
    3
    Views:
    726
    Reedick, Andrew
    Jul 1, 2008
  4. cornelis van der bent

    inaccuracies in compression algorithm

    cornelis van der bent, Jul 23, 2009, in forum: C Programming
    Replies:
    18
    Views:
    529
    Keith Thompson
    Jul 24, 2009
  5. Csaba2000

    Inaccuracies in absolute positioning

    Csaba2000, Sep 1, 2003, in forum: Javascript
    Replies:
    2
    Views:
    84
    Csaba2000
    Sep 2, 2003
Loading...

Share This Page