question about speed of sequential string replacement vs regex or

Discussion in 'Perl Misc' started by Xah Lee, Sep 28, 2011.

  1. Xah Lee

    Xah Lee Guest

    curious question.

    suppose you have 300 different strings and they need all be replaced
    to say "aaa".

    is it faster to replace each one sequentially (i.e. replace first
    string to aaa, then do the 2nd, 3rd,...)
    , or is it faster to use a regex with “or” them all and do replace one
    shot? (i.e. "1ststr|2ndstr|3rdstr|..." -> aaa)

    let's say the sourceString this replacement to be done on is 500k
    chars.

    Anyone? i suppose the answer will be similar for perl, python, ruby.

    btw, the origin of this question is about writing a emacs lisp
    function that replace ~250 html named entities to unicode char.

    Xah
     
    Xah Lee, Sep 28, 2011
    #1
    1. Advertising

  2. >>>>> "Xah" == Xah Lee <> writes:

    Xah> curious question.
    Xah> suppose you have 300 different strings and they need all be replaced
    Xah> to say "aaa".

    And then suppose this isn't the *real* question, but one entirely of
    Fiction by Xah Lee.

    How helpful do you want to be?

    --
    Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
    <> <URL:http://www.stonehenge.com/merlyn/>
    Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
    See http://methodsandmessages.posterous.com/ for Smalltalk discussion
     
    Randal L. Schwartz, Sep 28, 2011
    #2
    1. Advertising

  3. Xah Lee

    Xah Lee Guest

    On Sep 28, 3:57 am, (Randal L. Schwartz) wrote:
    > >>>>> "Xah" == Xah Lee <> writes:

    >
    > Xah> curious question.
    > Xah> suppose you have 300 different strings and they need all be replaced
    > Xah> to say "aaa".
    >
    > And then suppose this isn't the *real* question, but one entirely of
    > Fiction by Xah Lee.
    >
    > How helpful do you want to be?


    it's a interesting question anyway.

    the question originally came from when i was coding elisp of a
    function that changes html entities to unicode char literal. The
    problem is slightly complicated, involving a few questions about speed
    in emacs. e.g. string vs buffer, and much more... i spent several
    hours on this but it's probably too boring to detail (but i'll do so
    if anyone wishes). But anyway, while digging these questions that's
    not clear in my mind, i thought of why not generate a regex or
    construct and do it in one shot, and wondered if that'd be faster. But
    afterwards, i realized this wouldn't be applicable to my problem
    because for my problem each string needs to be changed to a unique
    string, not all to the same string.

    Xah
     
    Xah Lee, Sep 28, 2011
    #3
  4. Xah Lee

    Xah Lee Guest

    here's more detail about the origin of this problem. Relevant to emacs
    lisp only.

    ------------------------------

    in the definition of “replace-regexp-in-string”, there's this comment:

    ;; To avoid excessive consing from multiple matches in long strings,
    ;; don't just call `replace-match' continually. Walk down the
    ;; string looking for matches of REGEXP and building up a (reversed)
    ;; list MATCHES. This comprises segments of STRING which weren't
    ;; matched interspersed with replacements for segments that were.
    ;; [For a `large' number of replacements it's more efficient to
    ;; operate in a temporary buffer; we can't tell from the function's
    ;; args whether to choose the buffer-based implementation, though it
    ;; might be reasonable to do so for long enough STRING.]

    note: «For a `large' number of replacements it's more efficient to
    operate in a temporary buffer».

    my question is, anyone got some idea, how “large” is large?

    currently, i have a function replace-pairs-in-string which is
    implemented by repeatedly calling “replace-pairs-in-string” like this:

    (while (< ii (length pairs))
    (setq mystr (replace-regexp-in-string
    (elt tempMapPoints ii)
    (elt (elt pairs ii) 1)
    mystr t t))
    (setq ii (1+ ii))
    )

    When there are 260 pairs of strings to be replaced on a file that's
    26k in size, my function takes about 3 seconds (which i think is too
    slow). I'm at pain deciding whether my function should be implemented
    like this or whether it should create a temp buffer. The problem with
    temp buffer is that, if you repeatedly call it, the overhead of
    creating buffer is going to make it much slower.

    i was actually surprised that replace-regexp-in-string isn't written
    in C, which i thought it was.

    is there technical reason the replace-regexp-in-string isn't C? (i
    suppose only because nobody every did it and the need for speed didn't
    suffice?) and also, shouldn't there also be a replace-in-string
    (literal, not regex)? because i thought implementing replacement for
    string should be much simpler and faster, because buffers comes with
    it a whole structure such as “point”, text properties, buffer names,
    buffier modifier, etc.

    Xah

    On Sep 28, 5:28 am, Xah Lee <> wrote:
    > On Sep 28, 3:57 am, (Randal L. Schwartz) wrote:
    >
    > > >>>>> "Xah" == Xah Lee <> writes:

    >
    > > Xah> curious question.
    > > Xah> suppose you have 300 different strings and they need all be replaced
    > > Xah> to say "aaa".

    >
    > > And then suppose this isn't the *real* question, but one entirely of
    > > Fiction by Xah Lee.

    >
    > > How helpful do you want to be?

    >
    > it's a interesting question anyway.
    >
    > the question originally came from when i was coding elisp of a
    > function that changes html entities to unicode char literal. The
    > problem is slightly complicated, involving a few questions about speed
    > in emacs. e.g. string vs buffer, and much more... i spent several
    > hours on this but it's probably too boring to detail (but i'll do so
    > if anyone wishes). But anyway, while digging these questions that's
    > not clear in my mind, i thought of why not generate a regex or
    > construct and do it in one shot, and wondered if that'd be faster. But
    > afterwards, i realized this wouldn't be applicable to my problem
    > because for my problem each string needs to be changed to a unique
    > string, not all to the same string.
    >
    >  Xah
     
    Xah Lee, Sep 28, 2011
    #4
  5. Xah Lee

    Willem Guest

    Xah Lee wrote:
    ) the question originally came from when i was coding elisp of a
    ) function that changes html entities to unicode char literal. The
    ) problem is slightly complicated, involving a few questions about speed
    ) in emacs. e.g. string vs buffer, and much more... i spent several
    ) hours on this but it's probably too boring to detail (but i'll do so
    ) if anyone wishes). But anyway, while digging these questions that's
    ) not clear in my mind, i thought of why not generate a regex or
    ) construct and do it in one shot, and wondered if that'd be faster. But
    ) afterwards, i realized this wouldn't be applicable to my problem
    ) because for my problem each string needs to be changed to a unique
    ) string, not all to the same string.

    In Perl, it would be applicable. You see, in Perl, you can call a function
    in the replacement of the regex substitution, which can then look up the
    html entity and return the wanted unicode literal.

    I think you can do that in some other languages as well.


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
     
    Willem, Sep 28, 2011
    #5
  6. Xah Lee

    Willem Guest

    Eli the Bearded wrote:
    ) In comp.lang.perl.misc, Willem <> wrote:
    )> In Perl, it would be applicable. You see, in Perl, you can call a function
    )> in the replacement of the regex substitution, which can then look up the
    )> html entity and return the wanted unicode literal.
    )
    ) A function? I'd use a hash.

    A function can return a sensible value for unknown substitutions.
    In the case where you prebuild a giant regex or-list, that is not an issue,
    but I would match html entities generically.


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
     
    Willem, Sep 28, 2011
    #6
  7. Xah Lee

    John Bokma Guest

    Willem <> writes:

    > Eli the Bearded wrote:
    > ) In comp.lang.perl.misc, Willem <> wrote:
    > )> In Perl, it would be applicable. You see, in Perl, you can call a function
    > )> in the replacement of the regex substitution, which can then look up the
    > )> html entity and return the wanted unicode literal.
    > )
    > ) A function? I'd use a hash.
    >
    > A function can return a sensible value for unknown substitutions.


    You can do that also in the RHS of the substitution and still keep it
    readable if you use something like

    s{..}{

    your
    code
    goes
    here
    }ge;

    However, a function can be easier on the eye:

    s{...}{ some_good_name( ... ) }ge;

    --
    John Bokma j3b

    Blog: http://johnbokma.com/ Perl Consultancy: http://castleamber.com/
    Perl for books: http://johnbokma.com/perl/help-in-exchange-for-books.html
     
    John Bokma, Sep 28, 2011
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mladen Adamovic
    Replies:
    0
    Views:
    740
    Mladen Adamovic
    Dec 4, 2003
  2. Mladen Adamovic
    Replies:
    3
    Views:
    14,615
    Mladen Adamovic
    Dec 5, 2003
  3. Chris Nevill

    Regex String Replacement

    Chris Nevill, Jan 31, 2004, in forum: Java
    Replies:
    5
    Views:
    476
    Chris Nevill
    Jan 31, 2004
  4. Avshi
    Replies:
    2
    Views:
    407
    Avshi
    May 18, 2005
  5. Xah Lee
    Replies:
    15
    Views:
    289
    Terry Reedy
    Sep 28, 2011
Loading...

Share This Page