threads, XSUB allocated memory, destructors, destruction

Discussion in 'Perl Misc' started by Andrew Torda, Sep 30, 2005.

  1. Andrew Torda

    Andrew Torda Guest

    I have a perl module built out of XSUBs.
    The functions malloc() space, build structures and return
    pointers to perl. Perl calls their destructor routines with no
    problems. The C structures go back to the perl interpreter, as
    T_PTROBJ things, specified in a typemap file.

    Now, if I use threads, I
    make lots of new data
    break my arrays into 2 pieces
    threads->new( myfunc, @array_to_read_from);
    threads->new( myfunc, @array_to_read_from);
    thread[0]->join
    thread[1]->join

    Unfortunately, at the join() stage, each thread decides to
    cleanup, and call the destructors for my @array_to_read_from.
    Obviously free() gets called multiple times, terrible things
    happen to the heap and everything dies.

    What should I be doing in my XSUBs when I create objects (really
    just pointers to malloc()'d space) to ask perl not to do this ?
    I am sure I am missing something obvious, but I cannot find it in
    perlxs or perlxstut man pages.
    Many thanks for any advice.
     
    Andrew Torda, Sep 30, 2005
    #1
    1. Advertising

  2. Andrew Torda

    Sisyphus Guest

    "Andrew Torda" <-hamburg.de> wrote in message
    news:dhjp8d$a4l$-hamburg.de...
    > I have a perl module built out of XSUBs.
    > The functions malloc() space, build structures and return
    > pointers to perl. Perl calls their destructor routines with no
    > problems. The C structures go back to the perl interpreter, as
    > T_PTROBJ things, specified in a typemap file.
    >
    > Now, if I use threads, I
    > make lots of new data
    > break my arrays into 2 pieces
    > threads->new( myfunc, @array_to_read_from);
    > threads->new( myfunc, @array_to_read_from);
    > thread[0]->join
    > thread[1]->join
    >
    > Unfortunately, at the join() stage, each thread decides to
    > cleanup, and call the destructors for my @array_to_read_from.
    > Obviously free() gets called multiple times, terrible things
    > happen to the heap and everything dies.
    >


    Can't quite reproduce the exact problem. I gather that '@array_to_read_from'
    is an array of these "pointer" objects. Below my sig is an Inline::C script
    that (I think) does pretty much as described (and not much else). It doesn't
    use a typemap, but I don't think there's any relevance in that.

    I find that there's no problem at the join() stage. The cleanup of the
    pointer objects does not take place (afaict) until the script is about to
    exit (which is as it should be). At that time, however, a problem usually
    (but not always) does arise - after the majority of the pointer objects have
    been freed, I get (on Win32) the "Free to wrong pool...during global
    destruction" error. (That's a problem - but it may be a Win32-specific
    issue, so I won't go delving into that just yet. Which OS are you on ? I was
    going to give the script a run on Linux .... but my perl on Linux wasn't
    built with threads support.)

    Anyway ... the output I get looks like this:

    Thread started
    Thread started
    Thread started
    200 400 650
    All joined
    destroying int object
    int object destroyed
    destroying int object
    int object destroyed
    ..
    ..
    ..
    destroying int object
    int object destroyed
    destroying int object
    Free to wrong pool b39078 not c5d040 during global destruction.

    If you already have (or can be bothered installing) Inline::C then you might
    run the script and check that the problem really does occur when you think
    it does. Otoh, I might have been way off-beam with my interpretation of what
    you've said - and/or my script may be irrelevant to the problem you're
    facing - in which case feel free to modify it to better demonstrate the
    issue at hand.

    (Btw, with that script, I've established that the cleanup problem *is*
    associated with the using of threads. If I remove the threads stuff , then
    the cleanup proceeds smoothly every time. I also tried replacing malloc/free
    with New/Safefree, but it made no difference - which is not surprising.)

    Hth - but don't stress too much if it doesn't :)

    Cheers,
    Rob

    use warnings;
    use threads;

    package Experimental;

    use Inline C => Config =>
    BUILD_NOISY => 1;

    use Inline C => <<'EOC';

    SV * create_int_obj(SV * x) {
    int * int_obj, i, s;
    SV * obj_ref, * obj;

    s = (int)SvUV(x);

    /* Allocate space for s ints */
    /* New(1, int_obj, s, int); */
    int_obj = malloc(sizeof(int) * s);
    if(int_obj == NULL) croak("Failed to allocate memory in create_int_obj
    function");
    obj_ref = newSViv(0);
    obj = newSVrv(obj_ref, "Experimental");

    sv_setiv(obj, (IV)int_obj);
    SvREADONLY_on(obj);
    return obj_ref;
    }

    void DESTROY(SV * m) {
    printf("destroying int object\n");
    /* Safefree((int *) SvIV(SvRV(m))); */
    free((int *) SvIV(SvRV(m)));
    printf("int object destroyed\n");
    }


    EOC

    # Create an array of 200 pointer objects
    @array_to_read_from_1 = create_em(200);

    # Create an array of 400 pointer objects.
    @array_to_read_from_2 = create_em(400);

    # Create an array of 650 pointers.
    @array_to_read_from_3 = create_em(650);

    $thread1 = threads->new("start_thread", @array_to_read_from_1);
    $thread2 = threads->new("start_thread", @array_to_read_from_2);
    $thread3 = threads->new("start_thread", @array_to_read_from_3);

    $s1 = $thread1->join;
    $s2 = $thread2->join;
    $s3 = $thread3->join;

    print "$s1 $s2 $s3\n";
    print "All joined\n";

    sleep(2);

    # Then usually crashes at some time during the destruction of
    # the pointer objects with the error "Free to wrong pool...during global
    destruction"

    sub start_thread {
    print "Thread started\n";
    return scalar(@_);
    }

    sub create_em {
    # Return an array of pointer objects.
    my @ret = ();
    for(1..$_[0]) { push(@ret, create_int_obj(10 + int(rand(10))))}
    return @ret;
    }
     
    Sisyphus, Oct 2, 2005
    #2
    1. Advertising

  3. Andrew Torda

    Andrew Torda Guest

    "Sisyphus" <> writes:

    > "Andrew Torda" <-hamburg.de> wrote in message

    [.... original code deleted ...]
    > > Unfortunately, at the join() stage, each thread decides to
    > > cleanup, and call the destructors for my @array_to_read_from.
    > > Obviously free() gets called multiple times, terrible things
    > > happen to the heap and everything dies.
    > >

    >
    > Can't quite reproduce the exact problem. I gather that '@array_to_read_from'


    But your example does reproduce the problem. The exact point of
    cleanup is not specified (different under windows and linux).

    > is an array of these "pointer" objects. Below my sig is an Inline::C script
    > that (I think) does pretty much as described (and not much else). It doesn't
    > use a typemap, but I don't think there's any relevance in that.


    In order to see the problem with threads cleaning up malloc'd
    memory, I think you should take your nice example, and put a loop
    like
    for (my $i = 0; $i < 10; $i++) {
    around your code below. Under linux, at least, I think you will
    find you will not reach the second iteration of the loop.
    > $thread1 = threads->new("start_thread", @array_to_read_from_1);
    > $thread2 = threads->new("start_thread", @array_to_read_from_2);
    > $thread3 = threads->new("start_thread", @array_to_read_from_3);
    >
    > $s1 = $thread1->join;
    > $s2 = $thread2->join;
    > $s3 = $thread3->join;
    >
    > print "$s1 $s2 $s3\n";
    > print "All joined\n";


    Thanks for the example.
     
    Andrew Torda, Oct 2, 2005
    #3
  4. Andrew Torda

    Sisyphus Guest

    "Andrew Torda" <> wrote in message
    ..
    ..
    > In order to see the problem with threads cleaning up malloc'd
    > memory, I think you should take your nice example, and put a loop
    > like
    > for (my $i = 0; $i < 10; $i++) {
    > around your code below. Under linux, at least, I think you will
    > find you will not reach the second iteration of the loop.
    > > $thread1 = threads->new("start_thread", @array_to_read_from_1);
    > > $thread2 = threads->new("start_thread", @array_to_read_from_2);
    > > $thread3 = threads->new("start_thread", @array_to_read_from_3);
    > >
    > > $s1 = $thread1->join;
    > > $s2 = $thread2->join;
    > > $s3 = $thread3->join;
    > >
    > > print "$s1 $s2 $s3\n";
    > > print "All joined\n";

    >


    So I did :

    for($i = 0; $i < 10; $i++){

    $thread1 = threads->new("start_thread", @array_to_read_from_1);
    $thread2 = threads->new("start_thread", @array_to_read_from_2);
    $thread3 = threads->new("start_thread", @array_to_read_from_3);

    $s1 = $thread1->join;
    $s2 = $thread2->join;
    $s3 = $thread3->join;

    print "$s1 $s2 $s3\n";
    print "All joined\n";

    sleep(2);
    }

    And there was no problem on Win32 with that .... until the global
    destruction phase after the 10 loops had run (just prior to exit). ie, I
    got:

    Thread started
    Thread started
    Thread started
    200 400 650
    All joined
    Thread started
    Thread started
    Thread started
    200 400 650
    All joined
    ..
    ..
    Thread started
    Thread started
    Thread started
    200 400 650
    All joined
    Thread started
    Thread started
    Thread started
    200 400 650
    All joined
    destroying int object
    int object destroyed
    destroying int object
    int object destroyed
    destroying int object
    int object destroyed
    ..
    ..
    destroying int object
    int object destroyed
    destroying int object
    Free to wrong pool 336c878 not abababab during global destruction.

    Looks like the error is different on linux, then. Now wishing I'd built my
    perl on linux with threads support ... just so I can see for myself.

    Afaik, if you built your pointer objects as I built mine, then you've done
    it right - and what you're up against is simply a bug. I guess that would
    call for a bug report (see 'perldoc perlbug').

    If you haven't already, you could check that everything functions ok when
    threads are removed from the scene (just to prove that it's threads that are
    stuffing things up, and not something else).

    You could also try replacing malloc/free with New/Safefree (though I'm not
    expecting that to fix anything) ..... plus any other straws you care to
    clutch at :)

    Cheers,
    Rob
     
    Sisyphus, Oct 3, 2005
    #4
  5. Andrew Torda

    Guest

    "Sisyphus" <> wrote:
    >
    > Anyway ... the output I get looks like this:
    >
    > Thread started
    > Thread started
    > Thread started
    > 200 400 650
    > All joined
    > destroying int object
    > int object destroyed
    > destroying int object
    > int object destroyed
    > .
    > .
    > .
    > destroying int object
    > int object destroyed
    > destroying int object
    > Free to wrong pool b39078 not c5d040 during global destruction.


    On linux, I just get a segfault.

    ....
    > void DESTROY(SV * m) {
    > printf("destroying int object\n");


    printf("destroying int object %d\n", SvIV(SvRV(m)));

    > /* Safefree((int *) SvIV(SvRV(m))); */
    > free((int *) SvIV(SvRV(m)));
    > printf("int object destroyed\n");
    > }


    For reasons I don't understand, it is trying to call DESTROY twice for each
    object. It goes through calling DESTROY once for each object, then starts
    running through again, seg faulting at some random point on the second
    pass.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Oct 3, 2005
    #5
  6. Andrew Torda

    Sisyphus Guest

    <> wrote in message

    >
    > On linux, I just get a segfault.
    >


    On Win32 I get a segfault, too - produced, I presume, by freeing "to wrong
    pool".

    ...

    >
    > For reasons I don't understand, it is trying to call DESTROY twice for

    each
    > object. It goes through calling DESTROY once for each object, then starts
    > running through again, seg faulting at some random point on the second
    > pass.
    >


    On Win32, it looks like it's only being called once for each object - and it
    seems to be the freeing of the very last object that needs to be freed that
    produces the error (now that I look more closely).

    On further investigation, I found that if the pointer objects are not
    blessed into a package, then there seems to be no problem at all - see the
    (modified) script below my sig. Does it also work ok on linux ? If so then
    perhaps the op can get around his problem by using unblessed objects - which
    means that it's then his responsibility to call _destroy() explicitly at the
    appropriate time(s).

    Cheers,
    Rob

    use warnings;
    use threads;

    #package Experimental;

    use Inline C => Config =>
    BUILD_NOISY => 1;

    use Inline C => <<'EOC';

    SV * create_int_obj(SV * x) {
    int * int_obj, i, s;
    SV * obj_ref, * obj;

    s = (int)SvUV(x);

    /* Allocate space for s ints */
    /* New(1, int_obj, s, int); */
    int_obj = malloc(sizeof(int) * s);
    if(int_obj == NULL) croak("Failed to allocate memory in create_int_obj
    function");
    obj_ref = newSViv(0);
    obj = newSVrv(obj_ref, NULL);

    sv_setiv(obj, (IV)int_obj);
    SvREADONLY_on(obj);
    return obj_ref;
    }

    void _destroy(SV * m) {
    printf("destroying int object\n");
    /* Safefree((int *) SvIV(SvRV(m))); */
    free((int *) SvIV(SvRV(m)));
    printf("int object destroyed\n");
    }


    EOC

    # Create an array of 200 pointer objects
    @array_to_read_from_1 = create_em(200);

    # Create an array of 400 pointer objects.
    @array_to_read_from_2 = create_em(400);

    # Create an array of 650 pointers.
    @array_to_read_from_3 = create_em(650);

    $thread1 = threads->new("start_thread", @array_to_read_from_1);
    $thread2 = threads->new("start_thread", @array_to_read_from_2);
    $thread3 = threads->new("start_thread", @array_to_read_from_3);

    $s1 = $thread1->join;
    $s2 = $thread2->join;
    $s3 = $thread3->join;

    print "$s1 $s2 $s3\n";
    print "All joined\n";

    sleep(2);

    # Let's now re-assign to @array_to_read_from_1.
    # First we free up the existing pointer objects:
    for(@array_to_read_from_1) {_destroy($_)}

    # Then assign some new pointer objects:
    @array_to_read_from_1 = create_em(310);


    $thread4 = threads->new("start_thread", @array_to_read_from_1);
    $s4 = $thread4->join;

    print "$s4\nStill ok\n";
    sleep(2);

    for(@array_to_read_from_1) {_destroy($_)}
    for(@array_to_read_from_2) {_destroy($_)}
    for(@array_to_read_from_3) {_destroy($_)}

    sub start_thread {
    print "Thread started\n";
    return scalar(@_);
    }

    sub create_em {
    # Return an array of pointer objects.
    my @ret = ();
    for(1..$_[0]) { push(@ret, create_int_obj(10 + int(rand(10))))}
    return @ret;
    }
     
    Sisyphus, Oct 4, 2005
    #6
  7. Andrew Torda

    Guest

    "Sisyphus" <> wrote:
    > <> wrote in message
    >
    > >
    > > On linux, I just get a segfault.
    > >

    >
    > On Win32 I get a segfault, too - produced, I presume, by freeing "to
    > wrong pool".
    >
    > ...
    >
    > >
    > > For reasons I don't understand, it is trying to call DESTROY twice for

    > each
    > > object. It goes through calling DESTROY once for each object, then
    > > starts running through again, seg faulting at some random point on the
    > > second pass.
    > >

    >
    > On Win32, it looks like it's only being called once for each object - and
    > it seems to be the freeing of the very last object that needs to be freed
    > that produces the error (now that I look more closely).


    Could it be that Win32 is also trying to destroy everything twice, and it
    is the destroying of the very first object, but on the second pass, which
    produces the error? (doing so before the message on that one can be
    printed)


    > On further investigation, I found that if the pointer objects are not
    > blessed into a package, then there seems to be no problem at all - see
    > the (modified) script below my sig. Does it also work ok on linux ?


    Yes, it seems to work on Linux.

    > If so
    > then perhaps the op can get around his problem by using unblessed objects
    > - which means that it's then his responsibility to call _destroy()
    > explicitly at the appropriate time(s).


    I tried moving the DESTROY from XSUB to Perl, and have the Perl DESTROY
    call _destroy. But now DESTROY is getting called twice again, and we are
    back to seg-faulting.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Oct 4, 2005
    #7
  8. Andrew Torda

    Sisyphus Guest

    <> wrote in message
    >
    > Could it be that Win32 is also trying to destroy everything twice, and it
    > is the destroying of the very first object, but on the second pass, which
    > produces the error? (doing so before the message on that one can be
    > printed)
    >


    Perhaps something along those lines is occurring. I've now found that
    whenever the script runs without error, the number of times that DESTROY()
    gets called is exactly double the number of times it ought to get called. So
    the doubling up is definitely happening.

    It's just that whenever it segfaults, the printout on the screen indicates
    that it's happening on the last object of the array. ie - if there's only
    one array involved (simplest scenario), containing x pointer objects, the
    segfault apparently occurs the xth time that DESTROY() is called. If it
    doesn't segfault at that point, then DESTROY() will be successfully called
    2x times.

    > I tried moving the DESTROY from XSUB to Perl, and have the Perl DESTROY
    > call _destroy. But now DESTROY is getting called twice again, and we are
    > back to seg-faulting.
    >


    Heh - I'd tried that, too. Didn't help *me*, either :)

    Cheers,
    Rob
     
    Sisyphus, Oct 4, 2005
    #8
  9. Also sprach Sisyphus:

    ><> wrote in message
    >>
    >> Could it be that Win32 is also trying to destroy everything twice, and it
    >> is the destroying of the very first object, but on the second pass, which
    >> produces the error? (doing so before the message on that one can be
    >> printed)
    >>

    >
    > Perhaps something along those lines is occurring. I've now found that
    > whenever the script runs without error, the number of times that DESTROY()
    > gets called is exactly double the number of times it ought to get called. So
    > the doubling up is definitely happening.
    >
    > It's just that whenever it segfaults, the printout on the screen indicates
    > that it's happening on the last object of the array. ie - if there's only
    > one array involved (simplest scenario), containing x pointer objects, the
    > segfault apparently occurs the xth time that DESTROY() is called. If it
    > doesn't segfault at that point, then DESTROY() will be successfully called
    > 2x times.
    >
    >> I tried moving the DESTROY from XSUB to Perl, and have the Perl DESTROY
    >> call _destroy. But now DESTROY is getting called twice again, and we are
    >> back to seg-faulting.
    >>

    >
    > Heh - I'd tried that, too. Didn't help *me*, either :)


    Admiring all your attempts to understand and fix the problem, the
    explanation for that behaviour is a different one.

    When creating a new thread, data aren't shared by default. Instead, each
    thread gets a clone. Perl can easily create clones of pure-Perl types
    however it's out of its wits when a Perl type actually refers to an
    outside resource (such as those allocated integers you were dealing
    with).

    So what happens is that the pure-Perl part is still cloned, but not the
    underlying integer objects. Each of these cloned objects refer to the
    same memory allocations. So for each clone DESTROY is called, trying to
    deallocate the integer-objects. No wonder this results in double-,
    triple-, etc.-frees, depending on how many threads you have.

    So in essence: the XS portion of your code isn't thread-safe. In order
    to make it thread-safe, you have to take control over the cloning
    process. See 'perldoc perlmod' and search for "CLONE". For recent perls
    (>= 5.8.7) all you have to do is provide a CLONE_SKIP method for the
    package 'Experimental' (in your case). If it returns a true value, no
    cloning of these object happens and therefore no multiple frees either.
    Older perls only have a CLONE method that is called once for each
    thread. It's not clear to me how it can be used to prevent cloning. I
    think it can't and that's why CLONE_SKIP was introduced.

    Tassilo
    --
    use bigint;
    $n=71423350343770280161397026330337371139054411854220053437565440;
    $m=-8,;;$_=$n&(0xff)<<$m,,$_>>=$m,,print+chr,,while(($m+=8)<=200);
     
    Tassilo v. Parseval, Oct 5, 2005
    #9
  10. Andrew Torda

    Sisyphus Guest

    "Tassilo v. Parseval" <> wrote in message
    ..
    ..
    >
    > So in essence: the XS portion of your code isn't thread-safe. In order
    > to make it thread-safe, you have to take control over the cloning
    > process. See 'perldoc perlmod' and search for "CLONE". For recent perls
    > (>= 5.8.7) all you have to do is provide a CLONE_SKIP method for the
    > package 'Experimental' (in your case).


    Yep - CLONE_SKIP seems to be the missing ingredient - works for me, anyway.

    I don't understand why the problem arises only with *blessed* objects. Does
    cloning not occur wrt unblessed objects ? Or is it just that problems are
    avoided simply because perl doesn't (automatically) clean up unblessed
    objects ?

    Cheers,
    Rob
     
    Sisyphus, Oct 7, 2005
    #10
  11. Also sprach Sisyphus:

    > "Tassilo v. Parseval" <> wrote in message
    > .
    > .
    >>
    >> So in essence: the XS portion of your code isn't thread-safe. In order
    >> to make it thread-safe, you have to take control over the cloning
    >> process. See 'perldoc perlmod' and search for "CLONE". For recent perls
    >> (>= 5.8.7) all you have to do is provide a CLONE_SKIP method for the
    >> package 'Experimental' (in your case).

    >
    > Yep - CLONE_SKIP seems to be the missing ingredient - works for me, anyway.
    >
    > I don't understand why the problem arises only with *blessed* objects. Does
    > cloning not occur wrt unblessed objects ? Or is it just that problems are
    > avoided simply because perl doesn't (automatically) clean up unblessed
    > objects ?


    Cloning happens with every variable showing up in a threaded program
    (unless it's shared).

    Unblessed objects are different in that there is no custom DESTROY
    method called when they are garbage-collected. This of course means that
    memory manually allocated within an XSUB is lost as DESTROY is usually
    the place to claim it back.

    Tassilo
    --
    use bigint;
    $n=71423350343770280161397026330337371139054411854220053437565440;
    $m=-8,;;$_=$n&(0xff)<<$m,,$_>>=$m,,print+chr,,while(($m+=8)<=200);
     
    Tassilo v. Parseval, Oct 7, 2005
    #11
  12. Andrew Torda

    Guest

    "Sisyphus" <> wrote:
    > "Tassilo v. Parseval" <> wrote in
    > message .
    > .
    > >
    > > So in essence: the XS portion of your code isn't thread-safe. In order
    > > to make it thread-safe, you have to take control over the cloning
    > > process. See 'perldoc perlmod' and search for "CLONE". For recent perls
    > > (>= 5.8.7) all you have to do is provide a CLONE_SKIP method for the
    > > package 'Experimental' (in your case).

    >
    > Yep - CLONE_SKIP seems to be the missing ingredient - works for me,
    > anyway.
    >
    > I don't understand why the problem arises only with *blessed* objects.
    > Does cloning not occur wrt unblessed objects ? Or is it just that
    > problems are avoided simply because perl doesn't (automatically) clean up
    > unblessed objects ?


    For the unblessed objects, you were calling _destroy yourself, and doing
    it from within just one of the threads.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Oct 7, 2005
    #12
  13. Andrew Torda

    Sisyphus Guest

    "Tassilo v. Parseval"
    ..
    ..
    > Cloning happens with every variable showing up in a threaded program
    > (unless it's shared).
    >


    Or unless CLONE_SKIP() returns 1.
    I haven't explained my confusion very well. Setting CLONE_SKIP fixes the
    problem by preventing the cloning. But looks to me that the cloning is not
    the real problem - it's the DESTROY() that gets called that does the
    damage - which makes CLONE_SKIP a workaround, rather than a fix (... I know
    .... it's a moot distinction). Perhaps the correct way to fix the problem is
    to do something appropriate within the 'CLONE' subroutine - though I don't
    know what that "something appropriate" actually is. The perlmod docs seems
    to be saying that is the purpose of the CLONE subroutine:

    "In "CLONE" you can do whatever you need to do, like for example handle the
    cloning of non-Perl data, if necessary."

    Looking at it another way:
    If it's ok that unblessed objects be cloned, then it ought to be ok that
    blessed objects also be cloned.

    Is it possible to prevent cloning of unblessed objects ? Does CLONE_SKIP
    work in a script that doesn't declare a package name ? (ie does it work in
    package main ? How would one verify ?)

    Cheers,
    Rob
     
    Sisyphus, Oct 7, 2005
    #13
  14. Also sprach Sisyphus:

    > "Tassilo v. Parseval"
    > .
    > .
    >> Cloning happens with every variable showing up in a threaded program
    >> (unless it's shared).
    >>

    >
    > Or unless CLONE_SKIP() returns 1.
    > I haven't explained my confusion very well. Setting CLONE_SKIP fixes the
    > problem by preventing the cloning. But looks to me that the cloning is not
    > the real problem - it's the DESTROY() that gets called that does the
    > damage - which makes CLONE_SKIP a workaround, rather than a fix (... I know
    > ... it's a moot distinction). Perhaps the correct way to fix the problem is
    > to do something appropriate within the 'CLONE' subroutine - though I don't
    > know what that "something appropriate" actually is. The perlmod docs seems
    > to be saying that is the purpose of the CLONE subroutine:
    >
    > "In "CLONE" you can do whatever you need to do, like for example handle the
    > cloning of non-Perl data, if necessary."


    The tricky part about that is that CLONE is not an instance method so
    it's not called per object that is about to be cloned.

    > Looking at it another way:
    > If it's ok that unblessed objects be cloned, then it ought to be ok that
    > blessed objects also be cloned.
    >
    > Is it possible to prevent cloning of unblessed objects ? Does CLONE_SKIP
    > work in a script that doesn't declare a package name ? (ie does it work in
    > package main ? How would one verify ?)


    CLONE and CLONE_SKIP are run for each stash, and that should include
    main::. The problem with having main::CLONE_SKIP is that then no
    variable is cloned any longer, not even regular Perl types with no
    attached XSUB-logic.

    I don't have a recent enough threaded perl right now to test what
    happens in such a case. I assume that then all variables are essentially
    shared but without any mechanism of mutual exclusive access to them. In
    essence, your program should have a high likelihood of blowing up
    terribly.

    As for CLONE and how to use that, here's one example. CLONE is run in
    the context of the newly created thread. The idea of the code below is
    now to have CLONE mark the cloned variable as non-freeable with respect
    to that thread. I abuse the SVf_FAKE flag (renamed to SVf_DONT_FREE) for
    the mark since it's not going to be set by perl for the variable to be
    cloned:

    #!/usr/bin/perl -w

    use threads;

    package Experimental;

    use Inline C => Config => BUILD_NOISY => 1;

    use Inline C => <<'EOC';

    #define SVf_DONT_FREE SVf_FAKE

    SV * create_obj (int x) {
    int *int_obj;
    SV *obj;

    New(0, int_obj, x, int);
    obj = sv_setref_pv(newSV(0), "Experimental", (void*)int_obj);

    return obj;
    }

    void dont_free (SV *obj) {
    SvFLAGS(SvRV(obj)) |= SVf_DONT_FREE;
    }

    void DESTROY (SV *obj) {
    if (SvFLAGS(SvRV(obj)) & SVf_DONT_FREE) {
    printf("not freeing\n");
    return;
    }
    printf("freeing\n");
    Safefree((int*)SvIV(SvRV(obj)));
    }
    EOC

    my @array = map create_obj(rand 10), 1 .. 5;

    sub CLONE {
    dont_free($_) for @array;
    }

    my @t = map threads->new("run", @array), 1 .. 2;

    $_->join for @t;

    sub run {
    for (@_) {
    printf "%i / %i\n", threads->tid, $_;
    }
    }
    __END__
    1 / 137794056
    1 / 137801572
    1 / 137801596
    1 / 137801620
    1 / 137801644
    2 / 138506624
    2 / 138514140
    2 / 138514164
    2 / 138514188
    2 / 138514212
    not freeing
    not freeing
    not freeing
    not freeing
    not freeing
    not freeing
    not freeing
    not freeing
    not freeing
    not freeing
    freeing
    freeing
    freeing
    freeing
    freeing

    This solution is not entirely satisfactory because the cloneable
    variables in question need to be visible for CLONE. But by and large
    this should be adaptable enough to be used in threaded programs that
    make use of classes written in XS.

    Tassilo
    --
    use bigint;
    $n=71423350343770280161397026330337371139054411854220053437565440;
    $m=-8,;;$_=$n&(0xff)<<$m,,$_>>=$m,,print+chr,,while(($m+=8)<=200);
     
    Tassilo v. Parseval, Oct 8, 2005
    #14
  15. Andrew Torda

    Sisyphus Guest

    "Tassilo v. Parseval" <> wrote in message
    ..
    ..
    >
    > As for CLONE and how to use that, here's one example. CLONE is run in
    > the context of the newly created thread. The idea of the code below is
    > now to have CLONE mark the cloned variable as non-freeable with respect
    > to that thread. I abuse the SVf_FAKE flag (renamed to SVf_DONT_FREE) for
    > the mark since it's not going to be set by perl for the variable to be
    > cloned:
    >


    Thanks for taking the time to provide that example. That about wraps it up
    for me ..... wonder how the op is getting on with this :)

    Cheers,
    Rob
     
    Sisyphus, Oct 8, 2005
    #15
  16. Andrew Torda

    Andrew Torda Guest

    Greetings from the original poster
    > Thanks for taking the time to provide that example. That about wraps it up
    > for me ..... wonder how the op is getting on with this :)


    I can tell you... Not very happy.
    Using CLONE_SKIP() does what it should. It stops you being
    destroy()ed. Unfortunately, "man perlmod" says your objects
    "will be copied as unblessed, undef values" and "if the
    child thread needs to make use of the objects, then a more
    sophisticated approach is needed.
    OK.
    Maybe I should write my own CLONE(), although this is a bad
    idea. Perldoc Threads::shared says that "Currently CLONE is
    called with no parameters other than the invocant package
    name". This means you can't get to the address you need to do the
    copying. It is a catastrophically ugly approach to a large amount
    of data which should be shared since it is passed to threads for
    use in some calculations.

    We are dealing with arrays of blessed objects. They should
    be shared between threads, but then one hits the problem
    that ""bless" is not supported on shared references. In other
    words, if one has a reference to a 2D array of blessed objects
    and says p $$coords[0][0], you get something like
    MyPtr=SCALAR(0x85efe8c). If you then mark it as shared,
    "share $coords", and try the print statement, you are bluntly
    told, "Invalid value for shared scalar at ...".

    So, that is the answer to the question, "how is the op getting
    on with this".
    Andrew
     
    Andrew Torda, Oct 10, 2005
    #16
  17. Andrew Torda

    Guest

    Andrew Torda <please@no_mail.com> wrote:
    > Greetings from the original poster
    > > Thanks for taking the time to provide that example. That about wraps it
    > > up for me ..... wonder how the op is getting on with this :)

    >
    > I can tell you... Not very happy.
    > Using CLONE_SKIP() does what it should. It stops you being
    > destroy()ed. Unfortunately, "man perlmod" says your objects
    > "will be copied as unblessed, undef values" and "if the
    > child thread needs to make use of the objects, then a more
    > sophisticated approach is needed.


    Another possible way to prevent the children threads from trying to
    free the data is to have the threads bail out with a POSIX::_exit.
    I do this in forked processes all the time (where the goal is to prevent
    the costs of cleaning up, which can be very great for large amounts of
    data, especially with cow), but don't know how it will work in threads.


    > OK.
    > Maybe I should write my own CLONE(), although this is a bad
    > idea. Perldoc Threads::shared says that "Currently CLONE is
    > called with no parameters other than the invocant package
    > name". This means you can't get to the address you need to do the
    > copying.


    You can, but you need to store that address (those addresses) somewhere
    other than in the objects themselves (or you need to store the objects
    themselves somewhere in the package).

    > It is a catastrophically ugly approach to a large amount
    > of data which should be shared since it is passed to threads for
    > use in some calculations.


    Exactly what kind of sharing are you doing? Is most the sharing read
    only or does each thread need to be able to write such that the other
    threads can see the changes?

    If only the "master" thread need to write to this large amount of data,
    and if you are using a unix-like system, then you could try forking
    rather than threads. Perl may not know how to clone XSUB data, but the
    kernel does know how to. And because of cow (copy on write), forked data
    can live in the same RAM as the original until it starts getting changed.


    >
    > We are dealing with arrays of blessed objects. They should
    > be shared between threads, but then one hits the problem
    > that ""bless" is not supported on shared references. In other
    > words, if one has a reference to a 2D array of blessed objects
    > and says p $$coords[0][0], you get something like
    > MyPtr=SCALAR(0x85efe8c). If you then mark it as shared,
    > "share $coords", and try the print statement, you are bluntly
    > told, "Invalid value for shared scalar at ...".


    You should not get that from a print statement. That error occurs either
    when you assign to a shared variable, or when you share a variable. Not
    when you read from a variable.


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Oct 10, 2005
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    662
    Matt Wharton
    Dec 9, 2004
  2. Pablo
    Replies:
    6
    Views:
    345
    Howard
    Aug 18, 2006
  3. Jeff

    XS/XSUB FAQs? Tutorials?

    Jeff, Sep 25, 2003, in forum: Perl Misc
    Replies:
    2
    Views:
    93
    Anno Siegel
    Sep 26, 2003
  4. thorsten kracht

    xsub and gcc 4.0.2, static variables

    thorsten kracht, Mar 6, 2006, in forum: Perl Misc
    Replies:
    1
    Views:
    95
    Anno Siegel
    Mar 6, 2006
  5. Chris
    Replies:
    7
    Views:
    180
    sisyphus
    Nov 18, 2010
Loading...

Share This Page