embedding executable code in a regular expression in Python

Discussion in 'Python' started by Avi Kak, Jul 16, 2006.

  1. Avi Kak

    Avi Kak Guest

    Folks,

    Does regular expression processing in Python allow for executable
    code to be embedded inside a regular expression?

    For example, in Perl the following two statements

    $regex = qr/hello(?{print "saw hello\n"})mello(?{print "saw
    mello\n"})/;
    "jellohellomello" =~ /$regex/;

    will produce the output

    saw hello
    saw mello

    Is it possible to do the same in Python with any modules that come
    with the standard distribution, or with any other modules?

    Thanks in advance for any help,

    Avi Kak ()
    Avi Kak, Jul 16, 2006
    #1
    1. Advertising

  2. Avi Kak

    Ben C Guest

    On 2006-07-16, Avi Kak <> wrote:
    > Folks,
    >
    > Does regular expression processing in Python allow for executable
    > code to be embedded inside a regular expression?
    >
    > For example, in Perl the following two statements
    >
    > $regex = qr/hello(?{print "saw hello\n"})mello(?{print "saw
    > mello\n"})/;
    > "jellohellomello" =~ /$regex/;
    >
    > will produce the output
    >
    > saw hello
    > saw mello
    >
    > Is it possible to do the same in Python with any modules that come
    > with the standard distribution, or with any other modules?


    You can use sub and make the replacement pattern a function (or any
    "callable" thing) and it gets called back with the match object:

    import re

    def f(mo):
    if "hello" in mo.groups():
    print "saw hello"
    if "mello" in mo.groups():
    print "saw mello"

    re.sub(r'(hello)(mello)', f, "jellohellomello")

    Actually I didn't know you could do that in Perl. The time I've found
    this useful in Python is substitutions to convert e.g.
    "background-color" into "backgroundColor"; a function turns the c into
    C. I always assumed in Perl you would need to use eval for this, but
    perhaps there is another way.
    Ben C, Jul 16, 2006
    #2
    1. Advertising

  3. ----- Original Message -----
    From: "Avi Kak" <>
    Newsgroups: comp.lang.python
    To: <>
    Sent: Sunday, July 16, 2006 11:05 PM
    Subject: embedding executable code in a regular expression in Python


    > Folks,
    >
    > Does regular expression processing in Python allow for executable
    > code to be embedded inside a regular expression?
    >
    > For example, in Perl the following two statements
    >
    > $regex = qr/hello(?{print "saw hello\n"})mello(?{print "saw
    > mello\n"})/;
    > "jellohellomello" =~ /$regex/;
    >
    > will produce the output
    >
    > saw hello
    > saw mello
    >
    > Is it possible to do the same in Python with any modules that come
    > with the standard distribution, or with any other modules?
    >
    > Thanks in advance for any help,
    >
    > Avi Kak ()
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list


    There's a new module out that can do it like this:

    import SE
    Stream_Editor = SE.SE ('<EAT> "~[hm]ello~=print \'saw =\'\n" ')
    for line in Stream_Editor ('yellomellojellohello').split ('\n'):
    exec ("%s" % line)

    saw mello
    saw hello

    You'll find SE in the Cheese Shop. It has been commented favorably. I
    wouldn't comment it, because I wrote it.

    Regards

    Frederic
    Anthra Norell, Jul 16, 2006
    #3
  4. Avi Kak wrote:
    > Does regular expression processing in Python allow for executable
    > code to be embedded inside a regular expression?
    >
    > For example, in Perl the following two statements
    >
    > $regex = qr/hello(?{print "saw hello\n"})mello(?{print "saw
    > mello\n"})/;
    > "jellohellomello" =~ /$regex/;
    >
    > will produce the output
    >
    > saw hello
    > saw mello
    >
    > Is it possible to do the same in Python with any modules that come
    > with the standard distribution, or with any other modules?


    Just in case you were referring to security concerns: Sufficiently complex REs
    can take ages to compile and run and eat tons of memory, so there are always
    issues involved here.

    Stefan
    Stefan Behnel, Jul 17, 2006
    #4
  5. Avi Kak

    Paul McGuire Guest

    Avi Kak wrote:
    > Folks,
    >
    > Does regular expression processing in Python allow for executable
    > code to be embedded inside a regular expression?
    >
    > For example, in Perl the following two statements
    >
    > $regex = qr/hello(?{print "saw hello\n"})mello(?{print "saw
    > mello\n"})/;
    > "jellohellomello" =~ /$regex/;
    >
    > will produce the output
    >
    > saw hello
    > saw mello
    >


    Not nearly so terse, but perhaps easier to follow, here is a pyparsing
    version. Pyparsing parse actions are intended to do just what you ask.
    Parse actions may be defined to take no arguments, just one argument
    (which will be passed the list of matching token strings), 2 arguments
    (the match location and the matching tokens), or 3 arguments (the
    original source string, the match location, and the tokens). Parse
    actions are very good for transforming input text into modified output
    form, such as the "background-color" to "backgroundColor" transform -
    the BoaConstructor team used pyparsing to implement a version upgrade
    that transformed user source to a new version of wx (involving a
    variety of suh changes).

    Here is your jello/mello program, with two variations of parse actions.

    -- Paul

    from pyparsing import *

    instr = "jellorelohellomellofellowbellowmello"
    searchTerm = oneOf( ["jello","mello"] )

    # simple parse action, just echoes matched text
    def echoMatchedText(tokens):
    print "saw", tokens[0]

    searchTerm.setParseAction( echoMatchedText )
    searchTerm.searchString(instr)

    # modified parse action, prints location too
    def echoMatchedText(loc,tokens):
    print "saw", tokens[0], "at locn", loc

    searchTerm.setParseAction( echoMatchedText )
    searchTerm.searchString(instr)

    Prints out:
    saw jello
    saw mello
    saw mello
    saw jello at locn 0
    saw mello at locn 14
    saw mello at locn 31
    Paul McGuire, Jul 17, 2006
    #5
  6. Hi, see below ...

    ----- Original Message -----
    From: "Paul McGuire" <>
    Newsgroups: comp.lang.python
    To: <>
    Sent: Monday, July 17, 2006 10:09 AM
    Subject: Re: embedding executable code in a regular expression in Python


    > Avi Kak wrote:
    > > Folks,
    > >
    > > Does regular expression processing in Python allow for executable
    > > code to be embedded inside a regular expression?
    > >
    > > For example, in Perl the following two statements
    > >
    > > $regex = qr/hello(?{print "saw hello\n"})mello(?{print "saw
    > > mello\n"})/;
    > > "jellohellomello" =~ /$regex/;
    > >
    > > will produce the output
    > >
    > > saw hello
    > > saw mello
    > >

    >
    > Not nearly so terse, but perhaps easier to follow, here is a pyparsing
    > version. Pyparsing parse actions are intended to do just what you ask.
    > Parse actions may be defined to take no arguments, just one argument
    > (which will be passed the list of matching token strings), 2 arguments
    > (the match location and the matching tokens), or 3 arguments (the
    > original source string, the match location, and the tokens). Parse
    > actions are very good for transforming input text into modified output
    > form, such as the "background-color" to "backgroundColor" transform -
    > the BoaConstructor team used pyparsing to implement a version upgrade
    > that transformed user source to a new version of wx (involving a
    > variety of suh changes).
    >
    > Here is your jello/mello program, with two variations of parse actions.
    >
    > -- Paul
    >
    > from pyparsing import *
    >
    > instr = "jellorelohellomellofellowbellowmello"
    > searchTerm = oneOf( ["jello","mello"] )
    >
    > # simple parse action, just echoes matched text
    > def echoMatchedText(tokens):
    > print "saw", tokens[0]
    >
    > searchTerm.setParseAction( echoMatchedText )
    > searchTerm.searchString(instr)
    >
    > # modified parse action, prints location too
    > def echoMatchedText(loc,tokens):
    > print "saw", tokens[0], "at locn", loc
    >
    > searchTerm.setParseAction( echoMatchedText )
    > searchTerm.searchString(instr)
    >
    > Prints out:
    > saw jello
    > saw mello
    > saw mello
    > saw jello at locn 0
    > saw mello at locn 14
    > saw mello at locn 31
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list


    One fine example of a pyparse solution!

    On my part I had a second thought on the SE solution I proposed. It was
    needlessly complicated. Let me try again:

    >>> sentence = 'On each occurrence of the word "square" in this text the

    variable n will be squared. When it says "double" it will be doubled. At the
    end print n % 11.'
    >>> def square_n (): global n; n *= n
    >>> def double_n (): global n; n += n
    >>> def n_mod_11 (): global n; return n % 11
    >>> se_definitions = """<EAT>

    .... "square=print 'Calling square_n () - ',; square_n (); print 'n is now
    %d' % n \n" # A piece of code for 'square'
    ... "double=print 'Calling double_n () - ',; double_n (); print 'n is now
    %d' % n \n" # Another piece of code for 'double'
    ... ".=\nprint 'n %% 11 is: %d' % n_mod_11 ()\n" # Another piece of code
    for the final dot
    .... """
    >>> from SE import *
    >>> Stream_Editor = SE (definitions)
    >>> n = 9; exec (Stream_Editor (sentence))

    Calling square_n () - n is now 81
    Calling square_n () - n is now 6561
    n % 11 is: 5
    Calling double_n () - n is now 13122
    Calling double_n () - n is now 26244
    n % 11 is: 9
    n % 11 is: 9

    Suppose we now realize that the quoted words "square" and "double" should
    not trigger the respective function and neither should the ending dot of
    each sentence, except the last one. We fix the defintions in seconds, adding
    the exceptions as deletions. The order is immaterial. Targets may be regular
    expressions. (The last one is.)

    se_definitions = """<EAT> # Deletes everything except the defined
    substitutions
    .... "square=print 'Calling square_n () - ',; square_n (); print 'n is now
    %d' % n \n" # A piece of code for 'square'
    .... "double=print 'Calling double_n () - ',; double_n (); print 'n is now
    %d' % n \n" # Another piece of code for 'double'
    .... ".=\nprint 'n %% 11 is: %d' % n_mod_11 ()\n" # Another piece of code
    for the final dot
    .... \""square"=" \""double"=" # Deletes the quoted words
    .... "~\.\s~=" # Deletes dots followed by white space.
    .... """
    >>> n = 9; exec (SE (se_definitions)(sentence))

    Calling square_n () - n is now 81
    Calling double_n () - n is now 162
    n % 11 is: 8

    Note 1: SE is no match for pyparse on many accounts. Conversely SE has its
    own strengths in the realm of its specialty, in particular ease and speed of
    use, versatility and modularity.
    Most any problem can be solved in a variety of different ways. Finding
    good approaches is one of the appeals of programming.

    Note 2: The SE solution rests entirely on Python's 'exec' functionality and
    so here we have another testimony to Python's excellence.

    Note 3: I shall include the example in the manual. I hadn't thought of it.
    My thanks to the OP for the inspiration. Perhaps he might want to explain
    his purpose. I don't quite see the solution's problem.

    Frederic
    Anthra Norell, Jul 18, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. VSK
    Replies:
    2
    Views:
    2,261
  2. nico
    Replies:
    1
    Views:
    298
    Dennis Reinhardt
    Jul 1, 2004
  3. Replies:
    15
    Views:
    465
    Dave Thompson
    Aug 21, 2006
  4. pratap
    Replies:
    20
    Views:
    739
    Flash Gordon
    Mar 7, 2007
  5. Shankarjee Krishnamoorthi

    Embedding a executable inside Python Script

    Shankarjee Krishnamoorthi, Sep 19, 2007, in forum: Python
    Replies:
    2
    Views:
    375
    Shankarjee Krishnamoorthi
    Sep 19, 2007
Loading...

Share This Page