Perl pattern 5.6+ bug causing Parse::Yapp to fail

[Clint Olsen]

I'm using Perl 5.8.2, and I'm seeing some strange behavior from the pattern
matching engine.

Parse::Yapp is a module written to generate Perl in a yacc-like fashion.
However, I can't get the debugging version to work because it's not
correctly parsing between the markers in the file.

In debug mode, the module calls and modifies itself, uncommenting debug
blocks and then 'eval'-ing the string to produce a debug version of a
particular subroutine.

Here's the relevant section of code. The problem is the binary flip-flop
operator and the search pattern. For some reason, the search patterns are
not picking up the 'sub _Parse {' pattern seen in the next function.
What's more strange is that if I create a separate program with the while
loop shown in _DBLoad, I do in fact pick up the beginning and end of the
function.

Any suggestions?

Thanks,

-Clint

sub _DBLoad {
{
no strict 'refs';

exists(${__PACKAGE__.'::'}{_DBParse})#Already loaded ?
and return;
}
my($fname)=__FILE__;
my(@drv);
open(DRV,"<$fname") or die "Report this as a BUG: Cannot open $fname";
while(<DRV>) {
/^\s*sub\s+_Parse\s*{\s*$/ .. /^\s*}\s*#\s*_Parse\s*$/
and do {
s/^#DBG>//;
push(@drv,$_);
}
}
close(DRV);

$drv[0]=~s/_P/_DBP/;
eval join('',@drv);
}

#Note that for loading debugging version of the driver,
#this file will be parsed from 'sub _Parse' up to '}#_Parse' inclusive.
#So, DO NOT remove comment at end of sub !!!
sub _Parse {
my($self)=shift;

my($rules,$states,$lex,$error)
= @$self{ 'RULES', 'STATES', 'LEX', 'ERROR' };
my($errstatus,$nberror,$token,$value,$stack,$check,$dotpos)
= @$self{ 'ERRST', 'NBERR', 'TOKEN', 'VALUE', 'STACK', 'CHECK', 'DOTPOS' };

#DBG> my($debug)=$$self{DEBUG};
#DBG> my($dbgerror)=0;

#DBG> my($ShowCurToken) = sub {
#DBG> my($tok)='>';
#DBG> for (split('',$$token)) {
#DBG> $tok.= (ord($_) < 32 or ord($_) > 126)
#DBG> ? sprintf('<%02X>',ord($_))
#DBG> : $_;
#DBG> }
#DBG> $tok.='<';
#DBG> };

$$errstatus=0;
....

 

[Anno Siegel]

I'm using Perl 5.8.2, and I'm seeing some strange behavior from the pattern
matching engine.

Parse::Yapp is a module written to generate Perl in a yacc-like fashion.
However, I can't get the debugging version to work because it's not
correctly parsing between the markers in the file.

In debug mode, the module calls and modifies itself, uncommenting debug
blocks and then 'eval'-ing the string to produce a debug version of a
particular subroutine.

Here's the relevant section of code. The problem is the binary flip-flop
operator and the search pattern. For some reason, the search patterns are
not picking up the 'sub _Parse {' pattern seen in the next function.

This expression

/^\s*sub\s+_Parse\s*{\s*$/ .. /^\s*}\s*#\s*_Parse\s*$/

doesn't look for "sub _Parse {" in the next function, and it would never
find it unless the source defined _Parse twice. It looks for the
final "}#_Parse" that by convention ends that routine.

Or have I entirely misunderstood your question?

What's more strange is that if I create a separate program with the while
loop shown in _DBLoad, I do in fact pick up the beginning and end of the
function.

Is your code below an example of where it fails, or is it the self-
created program that works as expected? In any case, it is incomplete
because the crucial end of the _Parse routine isn't there. I've left
it in for reference, but as an aid to answering your question it doesn't
seem to help much.

Anno

sub _DBLoad {
{
no strict 'refs';

exists(${__PACKAGE__.'::'}{_DBParse})#Already loaded ?
and return;
}
my($fname)=__FILE__;
my(@drv);
open(DRV,"<$fname") or die "Report this as a BUG: Cannot open $fname";
while(<DRV>) {
/^\s*sub\s+_Parse\s*{\s*$/ .. /^\s*}\s*#\s*_Parse\s*$/
and do {
s/^#DBG>//;
push(@drv,$_);
}
}
close(DRV);

$drv[0]=~s/_P/_DBP/;
eval join('',@drv);
}

#Note that for loading debugging version of the driver,
#this file will be parsed from 'sub _Parse' up to '}#_Parse' inclusive.
#So, DO NOT remove comment at end of sub !!!
sub _Parse {
my($self)=shift;

my($rules,$states,$lex,$error)
= @$self{ 'RULES', 'STATES', 'LEX', 'ERROR' };
my($errstatus,$nberror,$token,$value,$stack,$check,$dotpos)
= @$self{ 'ERRST', 'NBERR', 'TOKEN', 'VALUE', 'STACK', 'CHECK', 'DOTPOS' };

#DBG> my($debug)=$$self{DEBUG};
#DBG> my($dbgerror)=0;

#DBG> my($ShowCurToken) = sub {
#DBG> my($tok)='>';
#DBG> for (split('',$$token)) {
#DBG> $tok.= (ord($_) < 32 or ord($_) > 126)
#DBG> ? sprintf('<%02X>',ord($_))
#DBG> : $_;
#DBG> }
#DBG> $tok.='<';
#DBG> };

$$errstatus=0;
...

 

[Clint Olsen]

This expression

/^\s*sub\s+_Parse\s*{\s*$/ .. /^\s*}\s*#\s*_Parse\s*$/

doesn't look for "sub _Parse {" in the next function, and it would never
find it unless the source defined _Parse twice. It looks for the
final "}#_Parse" that by convention ends that routine.

My understanding of this operator is that it is true once the left hand
side is true and remains true until it finds the left hand pattern. So, in
essence it grabs the entire contents between these two markers. Yes, I did
leave off the trailing marker for brevity reasons. Otherwise the sub was
too long for posting.

The problem is, the first pattern is never matched. But, when I create a
separate program and run it on this file, it _does_ match. So, I think
there's likely a bug lurking around somewhere in Perl.

Is your code below an example of where it fails, or is it the self-
created program that works as expected? In any case, it is incomplete
because the crucial end of the _Parse routine isn't there. I've left it
in for reference, but as an aid to answering your question it doesn't
seem to help much.

As it stands, DBLoad is a sub which is contained within the package that
will be parsed by that pattern shown above. It grabs the sub in between
the two markers and strips out the #DBG> comments and then 'evals' the code
on the fly to create a new routine.

-Clint

 

[Anno Siegel]

My understanding of this operator is that it is true once the left hand
side is true and remains true until it finds the left hand pattern. So, in ^^^^
right

essence it grabs the entire contents between these two markers.

Yes, after s/left/right/, that's what it does.

Yes, I did
leave off the trailing marker for brevity reasons. Otherwise the sub was
too long for posting.

You should have left the final "}#_parse" in place, omitting other parts.

The problem is, the first pattern is never matched. But, when I create a
separate program and run it on this file, it _does_ match. So, I think
there's likely a bug lurking around somewhere in Perl.

So how can we reproduce the bug, if there is one? Give us a recipe
how to use Parse::Yapp so that the error occurs.

Anno

 

[Clint Olsen]

So how can we reproduce the bug, if there is one? Give us a recipe how
to use Parse::Yapp so that the error occurs.

That's a good question.

I just tried the example Calc.yp that came with it, and debug mode appears
to be working. So, it's specific to my module which has sensitive data in
it (can't be distributed). However, it compiles cleanly under 'strict' and
'warning', so I'm not sure how I could isolate the problem. As usual with
Perl, I guess I'll have to do the traditional 'binary search' approach
where I gut halves of my code until I can see what's causing it.

Having said that, can you envision why a simple parse like that would fail?

Thanks,

-Clint

 

[Clint Olsen]

Try peppering your code/tokenizer/whatever with some debugging print()
statements, and printing out the location and any relevant variable
values. That might help, as does the debugging mode. Also are you using
the latest versions of perl, Parse::Yapp, etc?

The problem has something to do with my code, but since the sub is just
trying to parse its own file (the generated .pm file), it's not as simple
as embedding print statements.

I am using Perl 5.8.X and the latest Yapp.

Try putting all this so-sensitive data at the top of the module rather
than scattered throughout the code, if it is. That way you can easily
change the values to something inoffensive if you do need to show other
people the code if you still need help.

Yeah, I need to tear out my stuff and put them in separate package files.

Thanks,

-Clint