Variable destruction threads

[Jeff]

Hello,

I'm seeing a strange issue w/ perl in the code below.

The code below dynamically creates executable objects and passes them
along to a thread pool (written in c++, swig'd to Perl) to execute.
When execution of the object is complete, the new thread calls the perl
callback function that was provided at the time of the object's
construction.

The callback function prints the output of the executable object and
then destroys the object. The problem is that the callback function is
receiving an object that has already been destroyed.

I believe the destruction is taking place in the main creation loop,
but I'm not sure why. I think it's because I'm assigning the new
objects to the same variable over and over, and some point perl tries
to clean up this mess, and destroys the objects. And, unfortunately,
Perl is destroying the object before the callback function for one of
the objects has been processed.

Is there a way to manually increase the reference count for these
objects or a better way to go about this?



use EPEE;

#callback function
sub func()
{
my($cmd) = @_ ;
$output=$cmd->GetFormattedOutput();
printf("%s\n",$output);

$cmd->DESTROY();
}

#create threadPool
$threadPool=new EPEE::CThreadPool(32,16);

#main dynamic object creation loop
for($i=0;$i<1000000;$i++)
{
$testcmd=new EPEE::CCommand(\&func);
$threadPool->Execute($testcmd);
}

$threadPool->WaitForCompletion();

Thanks,
Jeff

 

[Fabian Pilkowski]

The callback function prints the output of the executable object and
then destroys the object. The problem is that the callback function is
receiving an object that has already been destroyed.

I believe the destruction is taking place in the main creation loop,
but I'm not sure why. I think it's because I'm assigning the new
objects to the same variable over and over,

Since you're guessing this is your problem, fix this by using lexical
scoped vars (those declared with "my").

and some point perl tries
to clean up this mess, and destroys the objects. And, unfortunately,
Perl is destroying the object before the callback function for one of
the objects has been processed.

Is there a way to manually increase the reference count for these
objects or a better way to go about this?

use EPEE;

use strict;
use warnings;

#callback function
sub func()
{
my($cmd) = @_ ;

Are you really sure that this works? You declare "func" with prototypes
and define that there are no parameters for this function. But then you
want to get the first one. I would remove the prototype definition.

$output=$cmd->GetFormattedOutput();
printf("%s\n",$output);

$cmd->DESTROY();
}

sub func {
my $cmd = shift;
printf "%s\n", $cmd->GetFormattedOutput();
$cmd->DESTROY();
}

#create threadPool
$threadPool=new EPEE::CThreadPool(32,16);

my $threadPool = new EPEE::CThreadPool( 32, 16 );

#main dynamic object creation loop
for($i=0;$i<1000000;$i++)
{
$testcmd=new EPEE::CCommand(\&func);
$threadPool->Execute($testcmd);
}

for my $i ( 1 .. 1_000_000 ) {
my $testcmd = new EPEE::CCommand( \&func );
$threadPool->Execute( $testcmd );
}

$threadPool->WaitForCompletion();

regards,
fabian

 

[Jeff]

By changing testcmd to a lexical scope variable, it will be destroyed
even earlier than it was before -- well before the other thread tries
to execute the testcmd object. This will only make the problem worse.

I understand the problem a little more now and it comes down to this:

There's one testcmd object and two perl references to this object. But
the problem is that each reference to the testcmd object is a seperate
perl interpreter (one cloned interpreter). This normally is not a
problem, but the reference in the cloned interpreter is created from
the raw c++ object.

The bottom line is that the reference count of both perl objects is
only 1. So if the main interpreter, the one that created the object,
is destroyed first, the c++ object is also destroyed. This leaves a
perl reference in the cloned interpreter to a null c++ object.

I can think of only two solutions:
1. Reference count the c++ object in a thread safe manner.
2. Duplicate the testcmd object (sv_dup) in the cloned interpreter
before the thread executes.

I still don't understand how perl decides when to destroy the $testcmd
object when it's created in global variable scope. Is it assumed to be
destroyed as soon as the testcmd variable is reassigned? This was my
initial thought, but if I print $testcmd after each iteration it grows
w/ an extra hash each time. The first time it has one hash value, the
two, etc...

Any insight?

Thanks,
Jeff