Memory Mgmt Using C and Perl

Discussion in 'Perl Misc' started by Mark Shelor, Jan 21, 2004.

  1. Mark Shelor

    Mark Shelor Guest

    I'm the author of a CPAN module (Digest::SHA) that's implemented mostly
    in C. A few of the underlying C routines use "malloc()" and "free()" to
    create and destroy various data structures at runtime. So far, the CPAN
    test results haven't indicated any big problems.

    However, a user contacted me the other day with reports of "pool" errors
    when attempting to build and run the Digest::SHA test scripts on MSWin.
    He didn't have any specifics about the Perl implementation he was
    using, but it wasn't ActiveState. He also stated that this particular
    Perl implementation was reported to have problems, but again, he wasn't
    specific.

    So, with little else to go on, I began to suspect that there might be a
    conflict between the memory manager in his Perl implementation, and the
    memory management that's being done by the C library's malloc() and
    free() functions.

    So, my question: is it generally required (or recommended) to avoid
    using malloc's within the C code of Perl extensions, and to instead use
    Perl's memory management functions (e.g. New) within the XS code? Doing
    this would cost me quite a bit of effort, so I'm reluctant to go ahead
    with this approach, especially if it turns out not to have been the
    problem in the first place.

    Mark
    Mark Shelor, Jan 21, 2004
    #1
    1. Advertising

  2. Also sprach Mark Shelor:

    > I'm the author of a CPAN module (Digest::SHA) that's implemented mostly
    > in C. A few of the underlying C routines use "malloc()" and "free()" to
    > create and destroy various data structures at runtime. So far, the CPAN
    > test results haven't indicated any big problems.
    >
    > However, a user contacted me the other day with reports of "pool" errors
    > when attempting to build and run the Digest::SHA test scripts on MSWin.
    > He didn't have any specifics about the Perl implementation he was
    > using, but it wasn't ActiveState. He also stated that this particular
    > Perl implementation was reported to have problems, but again, he wasn't
    > specific.
    >
    > So, with little else to go on, I began to suspect that there might be a
    > conflict between the memory manager in his Perl implementation, and the
    > memory management that's being done by the C library's malloc() and
    > free() functions.


    Very rightly so.

    > So, my question: is it generally required (or recommended) to avoid
    > using malloc's within the C code of Perl extensions, and to instead use
    > Perl's memory management functions (e.g. New) within the XS code? Doing
    > this would cost me quite a bit of effort, so I'm reluctant to go ahead
    > with this approach, especially if it turns out not to have been the
    > problem in the first place.


    The rule for XS is easy: Always use the provided macros New(), Newz(),
    Renew() and Safefree() which are wrappers for malloc(), calloc(),
    realloc() and free(). See perlclib.pod for details.

    Just yesterday on , we had this issue coming up. Here's
    the explanation from Nick Ing-Simmons. He's talking about the PV slot of
    a SV but that can be generalized:

    >
    >> DO NOT malloc() and put in PV slot.

    >
    >Hmmh, why not?


    Because it will segfault on Win32 and anywhere else where
    perl does not use system malloc() as its allocator.
    Perl assumes it owns SvPVX and will free it - to perl's allocator
    when it is no longer required. It also assumes
    that it can

    New(xxx,foo,char,SvCUR(sv)+1);
    memcpy(foo,SvPVX(sv),SvCUR(sv)+1);



    SvPVX(sv) = malloc(42);
    ...

    When SV's REFCNT==0 perl does:
    Perl_Free(SvPVX(sv))

    If Perl_Free != free this segfaults.

    Notice that on Win32 in particular perl does NOT use system malloc
    (or even anything remotely like it), and many platforms
    use perl's malloc for speed.

    >For me, PV has never really been for strings only. It's a
    >pointer value so storing pointers in it seems natural to me.

    I am not saying it has to be a string. I am saying it MUST
    be allocated with the allocator perl is using - malloc() is WRONG -
    It must be New() allocated.

    Tassilo
    --
    $_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
    pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus})!JAPH!qq(rehtona{tsuJbus#;
    $_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexiixesixeseg;y~\n~~dddd;eval
    Tassilo v. Parseval, Jan 21, 2004
    #2
    1. Advertising

  3. Mark Shelor

    Sisyphus Guest

    Mark Shelor wrote:
    > I'm the author of a CPAN module (Digest::SHA) that's implemented mostly
    > in C. A few of the underlying C routines use "malloc()" and "free()" to
    > create and destroy various data structures at runtime. So far, the CPAN
    > test results haven't indicated any big problems.
    >
    > However, a user contacted me the other day with reports of "pool" errors
    > when attempting to build and run the Digest::SHA test scripts on MSWin.
    > He didn't have any specifics about the Perl implementation he was
    > using, but it wasn't ActiveState. He also stated that this particular
    > Perl implementation was reported to have problems, but again, he wasn't
    > specific.
    >
    > So, with little else to go on, I began to suspect that there might be a
    > conflict between the memory manager in his Perl implementation, and the
    > memory management that's being done by the C library's malloc() and
    > free() functions.
    >
    > So, my question: is it generally required (or recommended) to avoid
    > using malloc's within the C code of Perl extensions, and to instead use
    > Perl's memory management functions (e.g. New) within the XS code? Doing
    > this would cost me quite a bit of effort, so I'm reluctant to go ahead
    > with this approach, especially if it turns out not to have been the
    > problem in the first place.
    >


    I believe this is recommended, but I don't think it's the cause of the
    problem - though I hasten to add that I'm not an expert on such matters
    and I've yet to investigate the problem thoroughly.

    What I have found is that the module (version 4.2.0) builds and tests
    fine on my MinGW/dmake-built perl 5.6.1 on Win2k (built from ActiveState
    build 626 source).

    However, on my MinGW/dmake-built perl 5.8.0 on Win2k (built from
    ActiveState build 802 source), the build process fails with the
    following error:
    src\sha.c:730: called object is not a function
    ..
    ..
    src\sha.c:751: called object is not a function

    There's a dozen or so warnings as well, but I think those are the fatal
    errors. With perl 5.6.1, instead of getting fatal errors at those lines
    I get the warnings:
    src\sha.c:730: warning: assignment from incompatible pointer type
    ..
    ..
    src\sha.c:751: warning: comparison of distinct pointer types lacks a cast

    Next I built the module on my perl 5.8.2 (built using Visual Studio
    ..NET, aka MSVC++ 7, built from ActiveState build 807 source) on the same
    Win2k.
    It builds fine, but all of the tests except the first produce the 'Free
    to wrong pool 15d2bb0 not C0100 t\blah-blah.t line x' error.

    Afaik there's no memory management conflict here.

    I'll have some time to look more closely at this tomorrow night if need
    be. All the better if, in the meantime, someone can present some ideas
    that might save me some time and brainstrain :)

    (No problems on mandrake-9.1, perl 5.8.0, btw.)

    Cheers,
    Rob


    --
    To reply by email u have to take out the u in kalinaubears.
    Sisyphus, Jan 21, 2004
    #3
  4. Mark Shelor

    Sisyphus Guest

    Sisyphus wrote:
    > Mark Shelor wrote:


    >> So, my question: is it generally required (or recommended) to avoid
    >> using malloc's within the C code of Perl extensions, and to instead
    >> use Perl's memory management functions (e.g. New) within the XS code?
    >> Doing this would cost me quite a bit of effort, so I'm reluctant to go
    >> ahead with this approach, especially if it turns out not to have been
    >> the problem in the first place.
    >>

    >
    > I believe this is recommended, but I don't think it's the cause of the
    > problem - though I hasten to add that I'm not an expert on such matters
    > and I've yet to investigate the problem thoroughly.
    >


    Just looking a little further and I find those same 'Free to wrong pool
    ....' errors occur on perl 5.8.0 (AS source 802) built using MSVC++6.0 on
    win2k. Again there's no problem with perl 5.6.1.

    I've verified that the 'shaclose()' call creates that error (no surprise
    about that I guess) - which I think implies that it's 'free()' and/or
    'memset()' that's causing the problem (since they're the only things
    that 'shaclose()' does). I deleted the 'memset()' call from 'shaclose()'
    in src/sha.c, and rebuilt the module - but 'shaclose()' still produces
    the same error. Alternatively, if I leave the 'memset()' call in there
    but remove the 'free()' call then I can get some of the tests to pass.
    Other tests still fail with the same 'wrong pool' error .... I guess
    they might be using something other than 'shaclose()', but I haven't
    traced it back.

    That's a pretty rough way of trying to eliminate the problem function -
    and I don't know if that's a valid way of making that elimination. But
    if it *is*, then it looks that, for whatever reason, you *do* need to
    replace 'malloc()' and 'free()' with 'New()' and 'Safefree()'. I found a
    couple of instances of 'calloc()' in your code, too. I've definitely
    experienced weird errors with that in the past. ('Realloc()' is
    perlapi's equivalent, I think.)

    While you're at it you might want to replace 'memset()' with its perl
    API equivalent. See the list in 'perldoc perlclib' - there might also be
    some other functions you could rewrite with perl API equivalents (eg
    'strlen()', which has also posed problems for me on win32 in the past).

    Hth.

    Cheers,
    Rob

    --
    To reply by email u have to take out the u in kalinaubears.
    Sisyphus, Jan 22, 2004
    #4
  5. Mark Shelor

    Mark Shelor Guest

    Sisyphus wrote:

    > What I have found is that the module (version 4.2.0) builds and tests
    > fine on my MinGW/dmake-built perl 5.6.1 on Win2k (built from ActiveState
    > build 626 source).
    >
    > However, on my MinGW/dmake-built perl 5.8.0 on Win2k (built from
    > ActiveState build 802 source), the build process fails with the
    > following error:
    > src\sha.c:730: called object is not a function
    > .
    > .
    > src\sha.c:751: called object is not a function



    Thanks very much, Rob, for testing this out on different configurations.

    The above-cited errors apparently result from the compiler's not
    recognizing the pre-defined "stdin" file handle. It's supposed to be
    declared in <stdio.h>, so these errors are odd indeed. I would suspect
    that the MinGW definition of "stdin" is the source of the problem.


    > There's a dozen or so warnings as well, but I think those are the fatal
    > errors. With perl 5.6.1, instead of getting fatal errors at those lines
    > I get the warnings:
    > src\sha.c:730: warning: assignment from incompatible pointer type
    > .
    > .
    > src\sha.c:751: warning: comparison of distinct pointer types lacks a cast



    Same thing again, i.e., I suspect a problem with the compiler's
    <stdio.h> include file regarding the definition of "stdin".


    > Next I built the module on my perl 5.8.2 (built using Visual Studio
    > .NET, aka MSVC++ 7, built from ActiveState build 807 source) on the same
    > Win2k.
    > It builds fine, but all of the tests except the first produce the 'Free
    > to wrong pool 15d2bb0 not C0100 t\blah-blah.t line x' error.



    Aha! This is the same problem that the user got. I suspected the cause
    might be that the C library "free()" function is in conflict with the
    Perl memory manager.


    > Afaik there's no memory management conflict here.



    That could be, but "free to wrong pool" may indicate that C's "free()"
    is attempting to de-allocate memory to an area that Perl's memory
    manager doesn't know about.


    > I'll have some time to look more closely at this tomorrow night if need
    > be. All the better if, in the meantime, someone can present some ideas
    > that might save me some time and brainstrain :)
    >
    > (No problems on mandrake-9.1, perl 5.8.0, btw.)



    Rob, you've already been a HUGE help in narrowing down (and re-creating)
    the problem. I really appreciate your looking into this.

    One interesting thing: the CPAN tester for the MSWin32 platform used
    ActiveState and VC++ and encountered no problems at all. His exact
    configuration can be seen at:

    http://www.nntp.perl.org/group/perl.cpan.testers/116990

    Kind Regards, Mark
    Mark Shelor, Jan 22, 2004
    #5
  6. Mark Shelor

    Sisyphus Guest

    Mark Shelor wrote:

    >
    > One interesting thing: the CPAN tester for the MSWin32 platform used
    > ActiveState and VC++ and encountered no problems at all. His exact
    > configuration can be seen at:
    >
    > http://www.nntp.perl.org/group/perl.cpan.testers/116990
    >


    Yes, that cpan tester was running perl 5.6.1. I've found no problems
    with that build of perl. It only occurs with 5.8.*.

    My original assertion that it's *not* a memory management is starting to
    look a little flaky :)

    Cheers,
    Rob


    --
    To reply by email u have to take out the u in kalinaubears.
    Sisyphus, Jan 22, 2004
    #6
  7. Also sprach Mark Shelor:

    > Sisyphus wrote:


    >> Afaik there's no memory management conflict here.

    >
    >
    > That could be, but "free to wrong pool" may indicate that C's "free()"
    > is attempting to de-allocate memory to an area that Perl's memory
    > manager doesn't know about.


    I suspect that it's an maybe issue with threads. The ActiveState perls
    are usually threaded ones. Each thread has its own heap of memory. You
    get this error when one threads allocates some memory and another thread
    tries to free() it.

    Also, this thread on the p5porters-list could be some help:

    <http://www.nntp.perl.org/group/perl.perl5.porters/87622>

    Tassilo
    --
    $_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
    pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus})!JAPH!qq(rehtona{tsuJbus#;
    $_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexiixesixeseg;y~\n~~dddd;eval
    Tassilo v. Parseval, Jan 22, 2004
    #7
  8. Mark Shelor

    Sisyphus Guest

    Mark Shelor wrote:

    >
    > The above-cited errors apparently result from the compiler's not
    > recognizing the pre-defined "stdin" file handle. It's supposed to be
    > declared in <stdio.h>, so these errors are odd indeed. I would suspect
    > that the MinGW definition of "stdin" is the source of the problem.
    >


    Those compiler warnings are not simply a mingw thing. The very same
    lines produce warnings with the MSVC compilers, too.

    In src/sha.c (following recomendations in 'perlclib' docs) I changed:

    1) Every occurrence of 'FILE*' to 'PerlIO*'
    2) Every occurrence of 'stdout' to 'PerlIO_stdout()'
    3) Every occurrence of 'stdin' to 'PerlIO_stdin()'
    4) Every occurrence of 'fprintf' to 'PerlIO_printf'

    I now get only one compiler warning when building the module (as opposed
    to the 11 warnings I used to get - some of which became fatal errors on
    mingw-built perl 5.8). The one remaining warning is in relation to the
    'fgets()' function at line 669 of src/sha.c. 'perldoc perlclib' has some
    advice regarding this, too - but I'm a little unsure as to how to put
    that advice into practice.
    Anyway, with those changes in place, I've now managed to build the
    module with my mingw-built perl 5.8.0. Of course, having achieved that,
    I now find that it, too, suffers precisely the same problems wrt freeing
    to "wrong pool" :)

    Cheers,
    Rob

    --
    To reply by email u have to take out the u in kalinaubears.
    Sisyphus, Jan 22, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Krishnan

    ASP .Net pages - instance mgmt

    Krishnan, Apr 5, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    274
    Krishnan
    Apr 5, 2004
  2. =?Utf-8?B?QVZM?=

    state mgmt issue

    =?Utf-8?B?QVZM?=, Feb 13, 2006, in forum: ASP .Net
    Replies:
    1
    Views:
    317
    Steve C. Orr [MVP, MCSD]
    Feb 13, 2006
  3. Karthik D
    Replies:
    3
    Views:
    706
    Karthik D
    Jul 2, 2003
  4. richard

    memory mgmt problem

    richard, Sep 1, 2003, in forum: C Programming
    Replies:
    12
    Views:
    546
    richard
    Sep 2, 2003
  5. RS Wood

    [CM] OStatic on Package Mgmt & Perl

    RS Wood, Mar 6, 2014, in forum: Perl Misc
    Replies:
    1
    Views:
    89
    George Mpouras
    Mar 7, 2014
Loading...

Share This Page