pyparsing and 'keywords'

Discussion in 'Python' started by Berteun Damman, Dec 14, 2004.

  1. Hello,

    I'm having some problems with pyparsing, I could not find how to tell
    it to view certain words as keywords, i.e. not as a possible variable
    name (in an elegant way),
    for example, I have this little grammar:

    terminator = Literal(";")
    expr = Word(alphas)
    body = Forward();
    ifstat = "if" + body + "fi"
    stat = expr | ifstat
    body << OneOrMore(stat + terminator)
    program = body

    I.e. some program which contains statements separated by semicolons. A
    statement is either an if [....] fi statement or simply a word.

    If I try however to parse the String "if test; testagain; fi;", it does
    not work, because the fi is interpreted as an expr, not as the end of
    the if statement, and of course, adding another fi doesn't solve this
    either.

    How to fix this?

    Thank you,

    Berteun
    Berteun Damman, Dec 14, 2004
    #1
    1. Advertising

  2. Berteun Damman

    Paul McGuire Guest

    "Berteun Damman" <> wrote in message
    news:...
    > Hello,
    >
    > I'm having some problems with pyparsing, I could not find how to tell
    > it to view certain words as keywords, i.e. not as a possible variable
    > name (in an elegant way),
    > for example, I have this little grammar:
    >
    > terminator = Literal(";")
    > expr = Word(alphas)
    > body = Forward();
    > ifstat = "if" + body + "fi"
    > stat = expr | ifstat
    > body << OneOrMore(stat + terminator)
    > program = body
    >
    > I.e. some program which contains statements separated by semicolons. A
    > statement is either an if [....] fi statement or simply a word.
    >
    > If I try however to parse the String "if test; testagain; fi;", it does
    > not work, because the fi is interpreted as an expr, not as the end of
    > the if statement, and of course, adding another fi doesn't solve this
    > either.
    >
    > How to fix this?
    >
    > Thank you,
    >
    > Berteun
    >

    Berteun -

    The simplest way I can think of for this grammar off the top of my head is
    to use a parse action to reject keywords.

    keywords = [ "if", "fi", "else", "return" ]
    def rejectKeywords(string,loc,tokens):
    if tokens[0] in keywords:
    raise ParseException(string,loc,"found keyword %s" % tokens[0])
    expr.setParseAction( rejectKeywords )

    I took a different tack in the idl parser that is included in the pyparsing
    examples directory, but it is more extensive.

    -- Paul
    Paul McGuire, Dec 14, 2004
    #2
    1. Advertising

  3. On Tue, 14 Dec 2004 18:39:19 GMT, Paul McGuire
    <._bogus_.com> wrote:
    >> If I try however to parse the String "if test; testagain; fi;", it does
    >> not work, because the fi is interpreted as an expr, not as the end of
    >> the if statement, and of course, adding another fi doesn't solve this
    >> either.

    > The simplest way I can think of for this grammar off the top of my head is
    > to use a parse action to reject keywords.


    Thank you for your quick and good solution, that did the trick indeed.

    But I'm wondering, isn't it possible to have some sort of lexing phase
    which already identifies keywords as such? Or to provide a table with
    keywords, which pyparsing is able to automatically recognize?

    So you would be able to do IF = Keyword("if"), just as a Literal now is
    created, but now the parser knows this word shouldn't be interpreted any
    other way than as a keyword. Or would that be a bad idea?

    Berteun
    Berteun Damman, Dec 14, 2004
    #3
  4. Berteun Damman

    Paul McGuire Guest

    "Berteun Damman" <berteun@NO_SPAMdds.nl> wrote in message
    news:...
    > On Tue, 14 Dec 2004 18:39:19 GMT, Paul McGuire
    > <._bogus_.com> wrote:
    > >> If I try however to parse the String "if test; testagain; fi;", it does
    > >> not work, because the fi is interpreted as an expr, not as the end of
    > >> the if statement, and of course, adding another fi doesn't solve this
    > >> either.

    > > The simplest way I can think of for this grammar off the top of my head

    is
    > > to use a parse action to reject keywords.

    >
    > Thank you for your quick and good solution, that did the trick indeed.
    >
    > But I'm wondering, isn't it possible to have some sort of lexing phase
    > which already identifies keywords as such? Or to provide a table with
    > keywords, which pyparsing is able to automatically recognize?
    >
    > So you would be able to do IF = Keyword("if"), just as a Literal now is
    > created, but now the parser knows this word shouldn't be interpreted any
    > other way than as a keyword. Or would that be a bad idea?
    >
    > Berteun

    Berteun -

    This is not a bad idea in and of itself, as I mentioned earlier, I had
    similar problems with the idl parser. For example, if in the body of your
    if block you had a statement:

    ifThisWorksItsAMiracle;

    The leading "if" would match the Literal("if"), although the statement is
    actually the complete 'ifThisWorksItsAMiracle' method call. That is,
    Literals are not automatically assumed to be whitespace-terminated. This is
    part of pyparsing's philosophy of being whitespace-blind.

    It's been a while since v 1.2.2 was released, I think adding a Keyword class
    much as you describe (on top of the other bug-fixes and additions since
    then) would be sufficient material for coming up with a 1.3 release. I'll
    work on it over the holiday "break," assuming I get one!

    -- Paul
    Paul McGuire, Dec 14, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Inyeol Lee
    Replies:
    2
    Views:
    316
    Inyeol Lee
    Nov 28, 2005
  2. Tim Arnold

    pyparsing and LaTeX?

    Tim Arnold, Nov 30, 2005, in forum: Python
    Replies:
    0
    Views:
    285
    Tim Arnold
    Nov 30, 2005
  3. Ezequiel, Justin

    pyparsing and LaTeX?

    Ezequiel, Justin, Dec 9, 2005, in forum: Python
    Replies:
    2
    Views:
    335
    Mike Meyer
    Dec 9, 2005
  4. Bytter

    PyParsing and Headaches

    Bytter, Nov 22, 2006, in forum: Python
    Replies:
    4
    Views:
    581
    Bytter
    Nov 23, 2006
  5. avidfan
    Replies:
    2
    Views:
    613
    avidfan
    Oct 31, 2007
Loading...

Share This Page