Help with tied/nested data structures

C

Clint Olsen

Hi:

I have a script included in the post that behaves strangely. Originally I
had a plain hash that had a few scalars inside of it that were tied.
However, for reasons I won't bore you with here, I needed to also tie the
containing hash. However, once I did this, I broke the tied scalars.
After I tie the scalar, it nearly gets immediately culled by the Perl
garbage collector (DESTROY is called), and I'm a bit puzzled why it's
firing. I'm sure some of you Perl blackbelts have seen something like this
before and can recite chapter/line/verse in the bible why this isn't
kosher.

Thanks,

-Clint

#!/usr/bin/perl

use strict;
use warnings;

package Foo;

my $pos;
my @keys;

sub TIEHASH {
my ($class,@args) = @_;

print "TIEHASH($class)\n"; bless {}
}

# Fetch a value from a tied hash
#
sub FETCH {
my $self = $_[0];
my $key = $_[1];
print "FETCH(", ref $self, ") ", "$key => ", ${$self}{$key} ? ${$self}{$key} : "undef", "\n"; ${$self}{$key};
}

# Get the first key from a tied hash
#
sub FIRSTKEY {
my $self = $_[0];
$pos = 0;
@keys = keys %{$self};

$keys[$pos++];
}

# Get the next key from a tied hash
#
sub NEXTKEY {
print "NEXTKEY => ", $keys[$pos] ? $keys[$pos] : "undef", "\n";
$keys[$pos++]
}

# check if a key exists in a tied hash
#
sub EXISTS {
my ($self,$key) = @_;

print "EXISTS $key => ", exists ${$self}{$key}, "\n";
exists ${$self}{$key};
}

#
# Place the value into the hash
#
sub STORE {
my ($self,$key,$val) = @_;
print "STORE(", ref $self, ") $key => $val\n"; ${$self}{$key} = $val;
}

sub DELETE {
my ($self,$key) = @_;

print "DELETE(", ref $self, ") $key\n";
delete ${$self}{$key};
}

sub CLEAR {
my ($self) = @_;

print "CLEAR\n";
%{$self} = ();
}

package Bar;

sub TIESCALAR {
my $class = $_[0];
my $tool = $_[1];

print "TIESCALAR($class) => $tool\n";

return bless \$tool, $class;
}

sub FETCH {
my $self = $_[0];

print "FETCH(", ref $self, ") fetching $$self\n";

return $$self;
}

sub STORE {
my ($self,$val) = @_;

print ref $self, ": STORE => $val\n";
}

sub DESTROY {
my $self = $_[0];

print "DESTROY(", ref $self, ") => $$self\n";
}

package main;

my %hash;

tie %hash, 'Foo';

tie $hash{bar}, 'Bar', 'blah';

$hash{booga} = 'baz';

print "Tied scalar value is $hash{bar}\n";
 
M

Mumia W.

Hi:

I have a script included in the post that behaves strangely. Originally I
had a plain hash that had a few scalars inside of it that were tied.
However, for reasons I won't bore you with here, I needed to also tie the
containing hash. However, once I did this, I broke the tied scalars.
After I tie the scalar, it nearly gets immediately culled by the Perl
garbage collector (DESTROY is called),
[...]
package main;

my %hash;

tie %hash, 'Foo';

tie $hash{bar}, 'Bar', 'blah';

$hash{booga} = 'baz';

print "Tied scalar value is $hash{bar}\n";

One part of the problem is that you're not saving the values
returned by tie. Those values are the object references that
underly the tied hashes and scalars. Since the ref counts for
those objects is zero right after you've created them, the GC
cleans them up almost immediately, so save the values:

my $object1 = tie %hash, 'Foo';

my $object2 = tie $hash{bar}, 'Bar', 'blah';


HTH
 
A

anno4000

Mumia W. said:
Hi:

I have a script included in the post that behaves strangely. Originally I
had a plain hash that had a few scalars inside of it that were tied.
However, for reasons I won't bore you with here, I needed to also tie the
containing hash. However, once I did this, I broke the tied scalars.
After I tie the scalar, it nearly gets immediately culled by the Perl
garbage collector (DESTROY is called),
[...]
package main;

my %hash;

tie %hash, 'Foo';

tie $hash{bar}, 'Bar', 'blah';

$hash{booga} = 'baz';

print "Tied scalar value is $hash{bar}\n";

One part of the problem is that you're not saving the values
returned by tie.

That is normally not necessary.
Those values are the object references that
underly the tied hashes and scalars. Since the ref counts for
those objects is zero right after you've created them, the GC
cleans them up almost immediately, so save the values:

No. Normally the object is stored in the magic structure that is
attached to the tied variable, refcounted and all. As long as that
magic survives, the object survives. The question is why it doesn't
in this case.

Actually another question needs to be asked first. What are the
expected semantics of

tie $hash{ key}, 'TieScalar';

when %hash itself it tied. Scalar tie() only works on actual scalar
variables (SVs), but there is no such thing associated with $hash{ key}
with a tied hash. So there really is no place to attach the scalar tie
magic to. Perl should probably warn about that. At the moment it
appears to silently drop it, as witnessed by the early call to DESTROY.

If you want to tie the values of a tied hash, you'd have to look into
the hash-tying mechanism itself. If there is a definite scalar for
every key where the value is stored, you'd have to tie that. If there
is no such scalar, you're out of luck.

Then again, it seems to me that anything you could to in an additional
scalar tie of the value you could also to in the hash-tie in the first
place.

Anno
 
X

xhoster

Mumia W. said:
Hi:

I have a script included in the post that behaves strangely.
Originally I had a plain hash that had a few scalars inside of it that
were tied. However, for reasons I won't bore you with here, I needed to
also tie the containing hash. However, once I did this, I broke the
tied scalars. After I tie the scalar, it nearly gets immediately culled
by the Perl garbage collector (DESTROY is called),
[...]
package main;

my %hash;

tie %hash, 'Foo';

tie $hash{bar}, 'Bar', 'blah';

$hash{booga} = 'baz';

print "Tied scalar value is $hash{bar}\n";

One part of the problem is that you're not saving the values
returned by tie.

That is not the problem.
Those values are the object references that
underly the tied hashes and scalars. Since the ref counts for
those objects is zero right after you've created them,

Another reference is also stored with the thing tied. Well, except
for some bug is preventing that in this case.
the GC
cleans them up almost immediately, so save the values:

my $object1 = tie %hash, 'Foo';

my $object2 = tie $hash{bar}, 'Bar', 'blah';

That delays the DESTROY call, but it doesn't solve the problem. The thing
that is tied now exists, but it is still not associated with the hash
element, it is just free standing.

I thought the problem was that you were tying a copy returned by the tied
FETCH (rather than the the actual thing stored in the underlying structure)
but it turns out there is no FETCH in the first place. Seems like some
kind of bug.

Xho
 
C

Clint Olsen

No. Normally the object is stored in the magic structure that is
attached to the tied variable, refcounted and all. As long as that magic
survives, the object survives. The question is why it doesn't in this
case.

Actually another question needs to be asked first. What are the expected
semantics of

tie $hash{ key}, 'TieScalar';

when %hash itself it tied. Scalar tie() only works on actual scalar
variables (SVs), but there is no such thing associated with $hash{ key}
with a tied hash. So there really is no place to attach the scalar tie
magic to. Perl should probably warn about that. At the moment it
appears to silently drop it, as witnessed by the early call to DESTROY.

If you want to tie the values of a tied hash, you'd have to look into the
hash-tying mechanism itself. If there is a definite scalar for every key
where the value is stored, you'd have to tie that. If there is no such
scalar, you're out of luck.

Then again, it seems to me that anything you could to in an additional
scalar tie of the value you could also to in the hash-tie in the first
place.

In my case, I only want to tie a few scalars (keys), not all of them. So,
adding this code to the FETCH method of the hash is kludgy at best. We
elected to tie the scalars to avoid a performance problem. Before we were
executing an external program for these and when we included the package
the external (slow) program was executed a bunch of times, so by tying the
scalars we only run it when the value is actually read.

I should point out that I have in the past been able to do nested tied
structures and I didn't have problems with the Perl GC culling the objects.
I think this was around 5.6.X though...

-Clint
 
X

xhoster

Clint Olsen said:
On 2006-07-19, (e-mail address removed)-berlin.de

I should point out that I have in the past been able to do nested tied
structures and I didn't have problems with the Perl GC culling the
objects.

You aren't having a problem with the GC culling them now. If the GC simply
stopped culling them, then you would have a memory leak in addition to the
problem you are currently having. It is merely a symptom, not the problem.
I think this was around 5.6.X though...

I tested your program on 5.6.1, and it displayed the same problem as you
are seeing. Maybe you did it in a different way back then? You can do it
this way:

{
my $x; tie $x, 'Bar', 'blah';
$hash{bar}=\$x;
}

But then you need to dereference it upon use, which you might not like:

print "Tied scalar value is ${$hash{bar}}\n";

Maybe you could make an object with an overloaded stringifier?


Xho
 
C

Clint Olsen

You aren't having a problem with the GC culling them now. If the GC
simply stopped culling them, then you would have a memory leak in
addition to the problem you are currently having. It is merely a
symptom, not the problem.

Yes, you are right.
I tested your program on 5.6.1, and it displayed the same problem as you
are seeing. Maybe you did it in a different way back then? You can do
it this way:

Yes, I was doing something entirely different. I was chaining ties and
using Inline::C to link hashes of hashes of flubledorgs of arrays of... (ad
nauseum). But the concept was the same. HOWEVER, I never did ever try
tied scalars in the mix. That could be where things are different.
{
my $x; tie $x, 'Bar', 'blah';
$hash{bar}=\$x;
}

But then you need to dereference it upon use, which you might not like:

print "Tied scalar value is ${$hash{bar}}\n";

Yeah, unfortunately we have a large and unwieldy codebase that's requiring
(or use'ing) this package, and we'd have to go update a lot of code to
derefence the scalar reference. I like that you have found some
workaround, though.
Maybe you could make an object with an overloaded stringifier?

Can you elaborate on this scheme? I'm curious what this involves.

Thanks,

-Clint
 
A

anno4000

You aren't having a problem with the GC culling them now. If the GC simply
stopped culling them, then you would have a memory leak in addition to the
problem you are currently having. It is merely a symptom, not the problem.


I tested your program on 5.6.1, and it displayed the same problem as you
are seeing. Maybe you did it in a different way back then? You can do it
this way:

{
my $x; tie $x, 'Bar', 'blah';
$hash{bar}=\$x;
}

But then you need to dereference it upon use, which you might not like:

print "Tied scalar value is ${$hash{bar}}\n";

Maybe you could make an object with an overloaded stringifier?

Sounds like a possibility.

Here is another: I'm assuming that the TieHash class actually
uses a normal hash to store values. If it doesn't, your more
abstract approach may still work. It still assumes that the
scalar is stored as such and not, for instance, stringified.
Some tie classes do that.

Say the "real" hash behind the tie is $obj->hashref. Then add to
the methods in Tiehash one that returns a scalar ref to the relevant
hash location:

sub location {
my ( $obj, $key) = @_;
\ $obj->hashref->{ $key};
}

Then, after

tie my %hash, 'TieHash';

instead of (vainly) trying to tie $hash{ $key}, do

tie ${ tied( %hash)->location( $key) }, 'TieScalar';

This ties the place where $hash{ $key} is actually stored. It
only makes sense when such a place exists.

Anno
 
X

xhoster

Yeah, unfortunately we have a large and unwieldy codebase that's
requiring (or use'ing) this package, and we'd have to go update a lot of
code to derefence the scalar reference. I like that you have found some
workaround, though.

This may be getting into the "Cure is worse than disease" territory, but
you could change the tied hash so that it will automatically dereference
the thing when it gets fetched. Of course, you need to do this automatic
derefernce only if the thing being fetched is actually a reference to a
tied variable.

# Fetch a value from a tied hash
#
sub FETCH {
my $self = $_[0];
my $key = $_[1];
print "FETCH(", ref $self, ") ", "$key => ",
${$self}{$key} ? ${$self}{$key} : "undef", "\n";
return ${${$self}{$key}} if ref ${$self}{$key} eq 'SCALAR'
and tied ${${$self}{$key}};
return ${$self}{$key};
}


Now you still need to store the reference:

{
my $x; tie $x, 'Bar', 'blah';
$hash{bar}=\$x;
}

But you no longer need to derefernce upon hash access:

print "Tied scalar value is $hash{bar}\n";

If the store is done in only a few tighly controlled places but the fetch
is done from many and/or uncontrolled places, this could work well.

Can you elaborate on this scheme? I'm curious what this involves.

I haven't used overload, I just know about its existence, so I can't really
elaborate. see perldoc overload.

The reason you have to store a reference to the tied scalar is that if you
try to store the tied scalar itself, it will invoke FETCH on it, rather
than storing the underlying SV. By using overload rather than tie, you can
probably distinguish between "I'm fetching just to store it somewhere else"
vs "I'm fetching so that I can print out the contents".

However, if the codebase is unwieldy, then you probably can't gaurantee
that all of the "fetch to store" will be in a non-stringy context and all
of the "fetch to print" will be in the stringy context.

Xho
 
M

Mumia W.

Mumia W. said:
One part of the problem is that you're not saving the values
returned by tie.

That is not the problem.
[...]

You're right, it's not. Anno's description seems to be
correct. I can't find a way to tie a hash value to a class
when the hash itself is also tied.

Unless Clint Olsen (the OP) wants to re-write Perl's
hash-tying code, it's probably better for him to re-write the
class he's tying the hash to.

Clint, I'm thinking that you can modify the hash-tying class
(Foo) to accept a reference to an underlying hash and operate
on the values in that hash. Then you can tie the scalars of
the underlying hash to the scalar-tying class (Bar).
 
M

Mumia W.

[...]
Clint, I'm thinking that you can modify the hash-tying class (Foo) to
accept a reference to an underlying hash and operate on the values in
that hash. Then you can tie the scalars of the underlying hash to the
scalar-tying class (Bar).

I hope this demonstrates the technique:

#!/usr/bin/perl

use strict 'vars','subs', 'refs';
use warnings;

my %hash;
my @planets = qw(mercury venus earth);

tie %hash, 'HashTest';

for my $plan (@planets) {
$plan =~ m/^./;
$hash{$plan} = ' ...' . (ucfirst $plan);
}

print " Planet: $hash{$_}\n" for (@planets);
exit;

#######################################

package ScalarTest;
use Tie::Scalar;
use base 'Tie::StdScalar';

sub FETCH {
my ($self) = shift;
print "FETCH on $self in @{[ __PACKAGE__ ]}\n";
$self->SUPER::FETCH(@_);
}


package HashTest;
use Tie::Hash;
use base 'Tie::ExtraHash';
use Data::Dumper;

sub FETCH {
my ($self) = shift;
print "FETCH on $self in @{[ __PACKAGE__ ]}\n";
$self->SUPER::FETCH(@_);
}

sub STORE {
my $self = shift;
my ($name, $value) = @_;
# print "STORE on $self in @{[ __PACKAGE__ ]}\n";

$self->[0]{$name} = '';
my $ref = \$self->[0]{$name};
if (! tied($ref)) {
tie $$ref, 'ScalarTest', $value;
}
$self;
}

__END__

I used Tie::ExtraHash and Tie::StdScalar to cut down on the
code. That's not too relevant. The important part is where the
scalar hash value is tested to see if it's already tied. If
not, it's tied to 'ScalarTest.'

HTH
 
M

Mumia W.

Mumia W. schreef:
my ($self) = shift;
[...]
my ($self) = shift;
[...]
my $self = shift;

Any special reason for the difference?

No reason other than the fact that the program went through
several changes. Heck, I even realized I left some debugging
cruft in it after I'd posted :)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,898
Latest member
BlairH7607

Latest Threads

Top