Hash in inherited object fields not properly accessible (5.8.6, RH 7.3)

[PeterSShenkin]

Here's a code excerpt that isn't sufficient to duplicate the problem as
a standalone, but does illustrate what the problem is. For the bug to
appear, $self->compute_base() has to be called from a class that
inherits from the one where the subroutine is defined. Then, on the
call to $self->compute_base(), the "while" loop behaves as if the
type_stats hash is empty; that is, the body is not executed. The hash
actually does have elements, so I expected the body to be executed for
each one. (The hash elements both exist and are defined.)

In the course of inserting printouts to determine what was going on, I
discovered that computing ntype_stats as shown below causes the "while"
to behave properly. Presumably, the call to "keys" makes the hash
contents visible.

Following the code excerpt is the output of "perl -V".

Apologies if this is a known bug; if so, please refer me to where it is
documented.

sub compute_base {
# compute() each child and return hash of results:
my $self = shift;
if( not ref $self ) {
confess
"$self" . "::compute_base: '$self' is not an object";
}
my $nsample = $self->{nsample};
my $results;
# the while finds no k,v pairs on the 1st call to compute_base
# without the following line; this seems to be a bug in Perl:
my $ntype_stats = scalar keys %{$self->{type_stats}};
while( my ($k,$v) = each %{$self->{type_stats}} ) {
$results->{$k} = $v->compute($nsample);
}
return $results;
}




Summary of my perl5 (revision 5 version 8 subversion 6) configuration:
Platform:
osname=linux, osvers=2.4.20-28.7, archname=i686-linux-thread-multi
uname='linux rosaleen 2.4.20-28.7 #1 thu dec 18 11:31:59 est 2003
i686 unknown '
config_args='-Dprefix=/utils -Dusethreads
-Accflags=-DPERL_REENTRANT_MAXSIZE=65536'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DPERL_REENTRANT_MAXSIZE=65536 -fno-strict-aliasing -pipe
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-I/usr/include/gdbm',
optimize='-O2',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DPERL_REENTRANT_MAXSIZE=65536 -fno-strict-aliasing -pipe
-I/usr/local/include -I/usr/include/gdbm'
ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.3
2.96-113)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.2.5.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.2.5'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
Built under linux
Compiled at Mar 12 2005 15:27:25
@INC:
/utils/lib/perl5/5.8.6/i686-linux-thread-multi
/utils/lib/perl5/5.8.6
/utils/lib/perl5/site_perl/5.8.6/i686-linux-thread-multi
/utils/lib/perl5/site_perl/5.8.6
/utils/lib/perl5/site_perl/5.8.1
/utils/lib/perl5/site_perl

 

[Uri Guttman]

P> In the course of inserting printouts to determine what was going on, I
P> discovered that computing ntype_stats as shown below causes the "while"
P> to behave properly. Presumably, the call to "keys" makes the hash
P> contents visible.

P> Apologies if this is a known bug; if so, please refer me to where it is
P> documented.

P> sub compute_base {
P> # compute() each child and return hash of results:
P> my $self = shift;

P> # the while finds no k,v pairs on the 1st call to compute_base
P> # without the following line; this seems to be a bug in Perl:
P> my $ntype_stats = scalar keys %{$self->{type_stats}};

P> while( my ($k,$v) = each %{$self->{type_stats}} ) {
P> $results->{$k} = $v->compute($nsample);
P> }

did you iterate over that object hash in some other code before that
code is called? if so, that could have exhausted the iterator and you
are getting back the end of iteration empty list. the call to scalar
keys resets the hash iterator. your data is there (try printing it with
data::dumper) but your hash's iterator is at the end (somehow you left
it there). see perldoc -f each for more on hash iterators. and you don't
need to assign to $ntype_stats, the keys call in void or scalar context
will reset the iterator.

uri

 

[PeterSShenkin]

did you iterate over that object hash in some other code before that
code is called? if so, that could have exhausted the iterator and you
are getting back the end of iteration empty list. the call to scalar
keys resets the hash iterator. your data is there ...

Yes, I knew the data were there -- just the fact that adding the call
to "keys" allowed the iteration showed that. And in fact I had also
printed the object with Data:umper.

Your suggestion is interesting -- even likely, though I'm not aware of
having done such iteration. I'll have to do some careful examination
to determine where/whether this is occurring. It's not impossible.

Is there by any chance some trick that one can use to obtain the
current state of the iterator without actually iterating?

Thanks,
-P.

 

[Uri Guttman]

P> Yes, I knew the data were there -- just the fact that adding the call
P> to "keys" allowed the iteration showed that. And in fact I had also
P> printed the object with Data:umper.

P> Your suggestion is interesting -- even likely, though I'm not aware of
P> having done such iteration. I'll have to do some careful examination
P> to determine where/whether this is occurring. It's not impossible.

i trust perl more than i trust your eyes . i would find it odd that
your code did that but who knows? the fact that resetting the iterator
fixes it tells me that you must have done some iterating before. i would
have to examine all the earlier code to find out but you will be doing
that. when you find it and smack your head, let us know! if you can
post all the other related code, maybe more eyes could find the
problem. can you reduce the problem to a short runnable script? maybe
just the constructor and/or some called methods might show the problem.

P> Is there by any chance some trick that one can use to obtain the
P> current state of the iterator without actually iterating?

not from the level of perl code AFAIK. all you can do is reset it and
then iterate until you get an end of hash marker from each (empty list
or undefined key). note that values also resets the iterator and you
will always get a fresh iteration whenever you call values or keys or
have the hash in a list context. besides, what kind of state would an
iterator return but the next key or the end of keys marker? internally
the iterator is just a pointer to the next key in the hash buckets.

uri

 

[anno4000]

P> Uri Guttman wrote:
[...]

P> Is there by any chance some trick that one can use to obtain the
P> current state of the iterator without actually iterating?

not from the level of perl code AFAIK.

Devel:eek:ump shows the state of a hash iterator.

To track this down you could tie the hash to a derivative of Tie::StdHash
that shouts (eg Carp::Confess) on FIRSTKEY and NEXTKEY. That should
catch any unexpected (and expected) iterations.

Anno

 

[Uri Guttman]

P> Uri Guttman wrote:

a> [...]

P> Is there by any chance some trick that one can use to obtain the
P> current state of the iterator without actually iterating?
a> Devel:eek:ump shows the state of a hash iterator.

i am sure that module does xs stuff. so that doesn't contradict what i
said which was from the perl level. i should have said pure perl
level.

a> To track this down you could tie the hash to a derivative of
a> Tie::StdHash that shouts (eg Carp::Confess) on FIRSTKEY and
a> NEXTKEY. That should catch any unexpected (and expected)
a> iterations.

that is a good idea.

uri

 

[PeterSShenkin]

OK, boys and girls. I found the problem. I also trust Perl more than
I trust my eyes; but how much do you trust Data:umper?

Here is a complete example that illustrates the problem. In my test
program, I had called Data:umper:ump just before the call that
generated the bad results. I had set Data:umper::Sortkeys to True.
Apparently, when a dump is done with this option, a hash traversal is
incompletely done within Data:umper, causing a subsequent call to
"each" to return 0 unless the iterator is manually reset between the
two calls.

In the following example, commenting out the offending line causes the
second "while" to print out the hash table. With the offending line
left in, the second "while" is silent.

-P.

########## snip #######################
use strict;
use Data:umper;

my %hash= (
a => 1,
b => 2,
c => 3,
d => 4,
);

while( my ($key,$val) = each %hash ) {
my $val = $hash{ $key };
print "while each: key,val= '$key', '$val'\n";
}

my $dumper = Data:umper->new( [\%hash] );
$dumper->Sortkeys( 1 ); #### offending line
my $dump_string = $dumper->Dump();

# the following block is silent
# unless the Sortkeys line is removed:
while( my ($key,$val) = each %hash ) {
my $val = $hash{ $key };
print "2nd while each: key,val= '$key', '$val'\n";
}
########## snip #######################

 

[PeterSShenkin]

Here are two more data points:

1. If I specify Data:umper::Useperl(1), the problem goes away.

2. I also see the problem unless I specify Useperl(1) with the
following version of Perl. Thus, the problem does not appear to be
specific to the build of Perl I was using earlier or to the OS I was
running on (though it could still be specific to 5.8.6).

Summary of my perl5 (revision 5 version 8 subversion 6) configuration:
Platform:
osname=darwin, osvers=8.0, archname=darwin-thread-multi-2level
uname='darwin b28.apple.com 8.0 darwin kernel version 7.5.0: thu
mar 3 18:48:46 pst 2005; root:xnuxnu-517.99.13.obj~1release_ppc power
macintosh powerpc '
config_args='-ds -e -Dprefix=/usr -Dccflags=-g -pipe
-Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-g -pipe -fno-common -DPERL_DARWIN
-no-cpp-precomp -fno-strict-aliasing -I/usr/local/include',
optimize='-Os',
cppflags='-no-cpp-precomp -g -pipe -fno-common -DPERL_DARWIN
-no-cpp-precomp -fno-strict-aliasing -I/usr/local/include'
ccversion='', gccversion='3.3 20030304 (Apple Computer, Inc. build
1809)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags
='-L/usr/local/lib'
libpth=/usr/local/lib /usr/lib
libs=-ldbm -ldl -lm -lc
perllibs=-ldl -lm -lc
libc=/usr/lib/libc.dylib, so=dylib, useshrplib=true,
libperl=libperl.dylib
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-bundle -undefined dynamic_lookup
-L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
Locally applied patches:
23953 - fix for File:ath::rmtree CAN-2004-0452 security issue
33990 - fix for setuid perl security issues
Built under darwin
Compiled at Mar 20 2005 16:34:19
@INC:
/System/Library/Perl/5.8.6/darwin-thread-multi-2level
/System/Library/Perl/5.8.6
/Library/Perl/5.8.6/darwin-thread-multi-2level
/Library/Perl/5.8.6
/Library/Perl
/Network/Library/Perl/5.8.6/darwin-thread-multi-2level
/Network/Library/Perl/5.8.6
/Network/Library/Perl
/System/Library/Perl/Extras/5.8.6/darwin-thread-multi-2level
/System/Library/Perl/Extras/5.8.6
/Library/Perl/5.8.1

 

[Uri Guttman]

P> Here is a complete example that illustrates the problem. In my test
P> program, I had called Data:umper:ump just before the call that
P> generated the bad results. I had set Data:umper::Sortkeys to True.
P> Apparently, when a dump is done with this option, a hash traversal is
P> incompletely done within Data:umper, causing a subsequent call to
P> "each" to return 0 unless the iterator is manually reset between the
P> two calls.

P> In the following example, commenting out the offending line causes the
P> second "while" to print out the hash table. With the offending line
P> left in, the second "while" is silent.

P> while( my ($key,$val) = each %hash ) {
P> my $val = $hash{ $key };
P> print "while each: key,val= '$key', '$val'\n";
P> }

P> my $dumper = Data:umper->new( [\%hash] );
P> $dumper->Sortkeys( 1 ); #### offending line
P> my $dump_string = $dumper->Dump();

P> # the following block is silent
P> # unless the Sortkeys line is removed:
P> while( my ($key,$val) = each %hash ) {
P> my $val = $hash{ $key };
P> print "2nd while each: key,val= '$key', '$val'\n";
P> }

i would call that a bug in data::dumper and you should post that to p5p
(use perlbug). glad you figured it out.

uri

 

[PeterSShenkin]

i would call that a bug in data::dumper and you should post that to p5p
(use perlbug).

Done, with Subject: "Data:umper with Sortkeys performs incomplete
hash traversal".

glad you figured it out.

Thanks for your help -- without which I don't think I would have.

-P.