TPG error when using 't' as the first letter of a token

[Andrew James]

Gentlemen,

I'm running into a problem whilst testing the parsing of a language I've
created with TPG . It seems that for some reason, TPG balks when I try
to parse an expression whose first letter is 't' (or, in fact, at any
time when 't' is at the beginning of a token). This doesn't happen with
any other letter (as far as I know), nor if the 'T' is capitalised.

My grammar looks like this:

# Tokens
separator space '\s+';
token Num '\d+(.\d+)?';
token Ident '[a-zA-Z]\w*';
token CharList '\'.*\'';
token CatUnOp '~';
token CatOp '[/\^~]';
token MetaOp '[=\+\-!]';
token Date '\d\d-\d\d-\d\d\d\d';
token FileID '(\w+\.\w+)'
;
# Rules
START -> CatExpr '\?' '[' MetaExpr ']'
| CatExpr
| FileID
;
CatExpr -> CatUnOp CatName
| CatName (CatOp CatName)*
| CatName
;
CatName -> Ident
#| '(' CatExpr ')'
;
MetaExpr -> MetaCrit (',' MetaCrit)*
;
MetaCrit -> Ident MetaOp Value
;
Value -> CharList | Num | Date
;

My test script like this:

if __name__ == '__main__':
""" For testing purposes only """
parseTests = ('This/is/a/simple/test', 'another/simple/test',
"a/test/with/[author='drew']")
for line in parseTests:
try:
print "\nParsing: %s \n%s\n" % (line,"="*(len(line)+9))
qp = MFQueryParser()
print qp(line)
except Exception, inst:
print "EXCEPTION: " + str(inst)

The output when using a letter which is not 't':

Parsing: another/simple/test
============================

[ 1][ 2]START.CatExpr: (1,1) Ident another != CatUnOp
[ 2][ 3]START.CatExpr.CatName: (1,1) Ident another == Ident
[ 3][ 2]START.CatExpr: (1,8) CatOp / == CatOp
[ 4][ 3]START.CatExpr.CatName: (1,9) Ident simple == Ident
[ 5][ 2]START.CatExpr: (1,15) CatOp / == CatOp
[ 6][ 3]START.CatExpr.CatName: (1,16) _tok_2 t != Ident
[ 7][ 1]START: (1,15) CatOp / != _tok_1
[ 8][ 2]START.CatExpr: (1,1) Ident another != CatUnOp
[ 9][ 3]START.CatExpr.CatName: (1,1) Ident another == Ident
[ 10][ 2]START.CatExpr: (1,8) CatOp / == CatOp
[ 11][ 3]START.CatExpr.CatName: (1,9) Ident simple == Ident
[ 12][ 2]START.CatExpr: (1,15) CatOp / == CatOp
[ 13][ 3]START.CatExpr.CatName: (1,16) _tok_2 t != Ident
EXCEPTION: SyntacticError at line 1, row 16: Syntax error near t

The output when using 't' as the first letter:

Parsing: tanother/simple/test
=============================

[ 1][ 2]START.CatExpr: (1,1) _tok_2 t != CatUnOp
[ 2][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 3][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 4][ 2]START.CatExpr: (1,1) _tok_2 t != CatUnOp
[ 5][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 6][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 7][ 1]START: (1,1) _tok_2 t != FileID
EXCEPTION: SyntacticError at line 1, row 1: Syntax error near t


I'm not sure whether this is something I'm doing wrong in my regular
expressions or whether something is being escaped in the TPG code, or
whether.... I just don't know!

I'm going through the TPG code at the moment but am not very hopeful of
finding the problem. Could someone possibly just let me know if I've
made an obvious mistake somewhere?

Many thanks,
Andrew

 

[Greg Krohn]

Gentlemen,

I'm running into a problem whilst testing the parsing of a language I've
created with TPG . It seems that for some reason, TPG balks when I try
to parse an expression whose first letter is 't' (or, in fact, at any
time when 't' is at the beginning of a token). This doesn't happen with
any other letter (as far as I know), nor if the 'T' is capitalised.

I have no idea what TPG is, but if it's only having trouble with a lower
case t, could it be that the t is somehow getting escaped and TPG thinks
it's a tab character?

Just a thought.

greg

 

[Andrew James]

Greg,
Thanks for your response. Thought about this and tried with a lowercase
'n' and a few other letters to check it out. It ran fine, so I'm back to
square one ;-/

TPG is a parser generator for Python (which is really quite cool) and
can be found at: http://christophe.delord.free.fr/en/tpg/

Can anyone else offer any insight?

Regards,
Andrew

 

[Andrew James]

Gentlemen,
Fixed what turned out to be my appalling knowledge of regexps. It turns
out that inline tokens are parsed as regular expressions by TPG and I
hadn't properly escaped the '[' characters. This means the START rule
has been changed to this:

START -> CatExpr ('\[' MetaExpr '\]')?

Now things work as expected, but I'm still unsure as to why this symptom
was only occurring with the uncapitalised character 't'.... can anyone
enlighten me?

Regards,
Andrew

 

[Paul McGuire]

Gentlemen,

I'm running into a problem whilst testing the parsing of a language I've
created with TPG . It seems that for some reason, TPG balks when I try
to parse an expression whose first letter is 't' (or, in fact, at any
time when 't' is at the beginning of a token). This doesn't happen with
any other letter (as far as I know), nor if the 'T' is capitalised.

My grammar looks like this:

# Tokens
separator space '\s+';
token Num '\d+(.\d+)?';
token Ident '[a-zA-Z]\w*';
token CharList '\'.*\'';
token CatUnOp '~';
token CatOp '[/\^~]';
token MetaOp '[=\+\-!]';
token Date '\d\d-\d\d-\d\d\d\d';
token FileID '(\w+\.\w+)'
;
# Rules
START -> CatExpr '\?' '[' MetaExpr ']'
| CatExpr
| FileID
;
CatExpr -> CatUnOp CatName
| CatName (CatOp CatName)*
| CatName
;
CatName -> Ident
#| '(' CatExpr ')'
;
MetaExpr -> MetaCrit (',' MetaCrit)*
;
MetaCrit -> Ident MetaOp Value
;
Value -> CharList | Num | Date
;

My test script like this:

if __name__ == '__main__':
""" For testing purposes only """
parseTests = ('This/is/a/simple/test', 'another/simple/test',
"a/test/with/[author='drew']")
for line in parseTests:
try:
print "\nParsing: %s \n%s\n" % (line,"="*(len(line)+9))
qp = MFQueryParser()
print qp(line)
except Exception, inst:
print "EXCEPTION: " + str(inst)

<snip>

FYI, as a comparative data point, here is your parser implemented using
pyparsing. I had to change your last test case because it didn't seem to
match your grammar.

-- Paul
(Download pyparsing at http://pyparsing.sourceforge.net.)

from pyparsing import alphas, nums, alphanums, Word, Optional, oneOf, Group,
\
Literal, Combine, sglQuotedString, Forward, delimitedList, ZeroOrMore,
OneOrMore

integer = Word(nums)
num = Combine(integer + Optional("." + integer))
identChars = alphanums + "_$"
ident = Word(alphas, identChars)
charList = sglQuotedString
unop = Literal("~")
binop = oneOf("/ ^ ~")
metaOp = oneOf("= + - !")
date = Combine( Word(nums,exact=2) + "-" + Word(nums,exact=2) + "-" +
Word(nums,exact=4) )
fileId = Combine( Word(identChars) + "." + Word(identChars) )

value = charList | date | num
metaCrit = ident + metaOp + value
metaExpr = Group(delimitedList( metaCrit ))
expr = Forward()
name = ident | Group( "(" + expr + ")" )
expr << Group( ( unop + name ) | ( name + ZeroOrMore(binop + name) ) )

start = Group( expr + "?" + "[" + metaExpr + "]" ) | expr | fileId

parseTests = (
'This/is/a/simple/test',
'tanother/simple/test',
"a/test/with?[author='drew']"
)
for t in parseTests:
print start.parseString(t)

Output:
=====
[['This', '/', 'is', '/', 'a', '/', 'simple', '/', 'test']]
[['tanother', '/', 'simple', '/', 'test']]
[[['a', '/', 'test', '/', 'with'], '?', '[', ['author', '=', "'drew'"],
']']]

 

[Andrew James]

Paul,
Thanks for your detailed response. I've read through the PyParsing site
and I'm considering using it for further development of my project -
wish I'd found it before!

I noted your comments about my last test case; the syntax of the
language I'm implementing is closely related to XPath and the [] at the
end of the query are intended to contain metadata criteria. I have since
revised my parsing rules (see below) and they now parse both the
original test queries and arbitrarily complex boolean expressions.

I'm just about to post a new message with a description of what the
language is aimed at doing and asking for any improvements - I would
value your opinion if you have the time.

# Tokens
separator space '\s+';
token Num '\d+(.\d+)?';
token Ident '[a-zA-Z]\w*';
token CharList '\'.*\'';
token CatUnOp '~';
token CatOp '[/\^]';
token MetaOp '[=\+\-!]';
token Date '\d\d-\d\d-\d\d\d\d';
token FileID '(\w+\.\w+)';
token EmptyLine '^$';

# Rules
START -> CatExpr ('\[' MetaExpr '\]')?
| FileID
| EmptyLine
;
CatExpr -> CatUnOp CatName
| CatName (CatOp CatExpr)*
;
CatName -> Ident
| '\(' CatExpr '\)'
;
MetaExpr -> MetaCrit (',' MetaCrit)*
;
MetaCrit -> Ident MetaOp Value
;
Value -> CharList | Num | Date
;

parseTests = (
"simple",
"this/is/a/simple/test",
"a/test/with/metadata[author='drew',date=10]",
"music/mp3/~jackson/michael",
"docs/latex/~(computer^science)",
"media/video/((comedy/action)^thriller)"
)

Regards,
Andrew