Memory Mgmt Using C and Perl

[Mark Shelor]

I'm the author of a CPAN module (Digest::SHA) that's implemented mostly
in C. A few of the underlying C routines use "malloc()" and "free()" to
create and destroy various data structures at runtime. So far, the CPAN
test results haven't indicated any big problems.

However, a user contacted me the other day with reports of "pool" errors
when attempting to build and run the Digest::SHA test scripts on MSWin.
He didn't have any specifics about the Perl implementation he was
using, but it wasn't ActiveState. He also stated that this particular
Perl implementation was reported to have problems, but again, he wasn't
specific.

So, with little else to go on, I began to suspect that there might be a
conflict between the memory manager in his Perl implementation, and the
memory management that's being done by the C library's malloc() and
free() functions.

So, my question: is it generally required (or recommended) to avoid
using malloc's within the C code of Perl extensions, and to instead use
Perl's memory management functions (e.g. New) within the XS code? Doing
this would cost me quite a bit of effort, so I'm reluctant to go ahead
with this approach, especially if it turns out not to have been the
problem in the first place.

Mark

 

[Tassilo v. Parseval]

Also sprach Mark Shelor:

I'm the author of a CPAN module (Digest::SHA) that's implemented mostly
in C. A few of the underlying C routines use "malloc()" and "free()" to
create and destroy various data structures at runtime. So far, the CPAN
test results haven't indicated any big problems.

However, a user contacted me the other day with reports of "pool" errors
when attempting to build and run the Digest::SHA test scripts on MSWin.
He didn't have any specifics about the Perl implementation he was
using, but it wasn't ActiveState. He also stated that this particular
Perl implementation was reported to have problems, but again, he wasn't
specific.

So, with little else to go on, I began to suspect that there might be a
conflict between the memory manager in his Perl implementation, and the
memory management that's being done by the C library's malloc() and
free() functions.

Very rightly so.

So, my question: is it generally required (or recommended) to avoid
using malloc's within the C code of Perl extensions, and to instead use
Perl's memory management functions (e.g. New) within the XS code? Doing
this would cost me quite a bit of effort, so I'm reluctant to go ahead
with this approach, especially if it turns out not to have been the
problem in the first place.

The rule for XS is easy: Always use the provided macros New(), Newz(),
Renew() and Safefree() which are wrappers for malloc(), calloc(),
realloc() and free(). See perlclib.pod for details.

Just yesterday on (e-mail address removed), we had this issue coming up. Here's
the explanation from Nick Ing-Simmons. He's talking about the PV slot of
a SV but that can be generalized:

Hmmh, why not?

Because it will segfault on Win32 and anywhere else where
perl does not use system malloc() as its allocator.
Perl assumes it owns SvPVX and will free it - to perl's allocator
when it is no longer required. It also assumes
that it can

New(xxx,foo,char,SvCUR(sv)+1);
memcpy(foo,SvPVX(sv),SvCUR(sv)+1);



SvPVX(sv) = malloc(42);
...

When SV's REFCNT==0 perl does:
Perl_Free(SvPVX(sv))

If Perl_Free != free this segfaults.

Notice that on Win32 in particular perl does NOT use system malloc
(or even anything remotely like it), and many platforms
use perl's malloc for speed.

For me, PV has never really been for strings only. It's a
pointer value so storing pointers in it seems natural to me.

I am not saying it has to be a string. I am saying it MUST
be allocated with the allocator perl is using - malloc() is WRONG -
It must be New() allocated.

Tassilo

 

[Sisyphus]

I'm the author of a CPAN module (Digest::SHA) that's implemented mostly
in C. A few of the underlying C routines use "malloc()" and "free()" to
create and destroy various data structures at runtime. So far, the CPAN
test results haven't indicated any big problems.

However, a user contacted me the other day with reports of "pool" errors
when attempting to build and run the Digest::SHA test scripts on MSWin.
He didn't have any specifics about the Perl implementation he was
using, but it wasn't ActiveState. He also stated that this particular
Perl implementation was reported to have problems, but again, he wasn't
specific.

So, with little else to go on, I began to suspect that there might be a
conflict between the memory manager in his Perl implementation, and the
memory management that's being done by the C library's malloc() and
free() functions.

So, my question: is it generally required (or recommended) to avoid
using malloc's within the C code of Perl extensions, and to instead use
Perl's memory management functions (e.g. New) within the XS code? Doing
this would cost me quite a bit of effort, so I'm reluctant to go ahead
with this approach, especially if it turns out not to have been the
problem in the first place.

I believe this is recommended, but I don't think it's the cause of the
problem - though I hasten to add that I'm not an expert on such matters
and I've yet to investigate the problem thoroughly.

What I have found is that the module (version 4.2.0) builds and tests
fine on my MinGW/dmake-built perl 5.6.1 on Win2k (built from ActiveState
build 626 source).

However, on my MinGW/dmake-built perl 5.8.0 on Win2k (built from
ActiveState build 802 source), the build process fails with the
following error:
src\sha.c:730: called object is not a function
..
..
src\sha.c:751: called object is not a function

There's a dozen or so warnings as well, but I think those are the fatal
errors. With perl 5.6.1, instead of getting fatal errors at those lines
I get the warnings:
src\sha.c:730: warning: assignment from incompatible pointer type
..
..
src\sha.c:751: warning: comparison of distinct pointer types lacks a cast

Next I built the module on my perl 5.8.2 (built using Visual Studio
..NET, aka MSVC++ 7, built from ActiveState build 807 source) on the same
Win2k.
It builds fine, but all of the tests except the first produce the 'Free
to wrong pool 15d2bb0 not C0100 t\blah-blah.t line x' error.

Afaik there's no memory management conflict here.

I'll have some time to look more closely at this tomorrow night if need
be. All the better if, in the meantime, someone can present some ideas
that might save me some time and brainstrain

(No problems on mandrake-9.1, perl 5.8.0, btw.)

Cheers,
Rob

 

[Sisyphus]

Mark Shelor wrote:

I believe this is recommended, but I don't think it's the cause of the
problem - though I hasten to add that I'm not an expert on such matters
and I've yet to investigate the problem thoroughly.

Just looking a little further and I find those same 'Free to wrong pool
....' errors occur on perl 5.8.0 (AS source 802) built using MSVC++6.0 on
win2k. Again there's no problem with perl 5.6.1.

I've verified that the 'shaclose()' call creates that error (no surprise
about that I guess) - which I think implies that it's 'free()' and/or
'memset()' that's causing the problem (since they're the only things
that 'shaclose()' does). I deleted the 'memset()' call from 'shaclose()'
in src/sha.c, and rebuilt the module - but 'shaclose()' still produces
the same error. Alternatively, if I leave the 'memset()' call in there
but remove the 'free()' call then I can get some of the tests to pass.
Other tests still fail with the same 'wrong pool' error .... I guess
they might be using something other than 'shaclose()', but I haven't
traced it back.

That's a pretty rough way of trying to eliminate the problem function -
and I don't know if that's a valid way of making that elimination. But
if it *is*, then it looks that, for whatever reason, you *do* need to
replace 'malloc()' and 'free()' with 'New()' and 'Safefree()'. I found a
couple of instances of 'calloc()' in your code, too. I've definitely
experienced weird errors with that in the past. ('Realloc()' is
perlapi's equivalent, I think.)

While you're at it you might want to replace 'memset()' with its perl
API equivalent. See the list in 'perldoc perlclib' - there might also be
some other functions you could rewrite with perl API equivalents (eg
'strlen()', which has also posed problems for me on win32 in the past).

Hth.

Cheers,
Rob

 

[Mark Shelor]

What I have found is that the module (version 4.2.0) builds and tests
fine on my MinGW/dmake-built perl 5.6.1 on Win2k (built from ActiveState
build 626 source).

However, on my MinGW/dmake-built perl 5.8.0 on Win2k (built from
ActiveState build 802 source), the build process fails with the
following error:
src\sha.c:730: called object is not a function
.
.
src\sha.c:751: called object is not a function

Thanks very much, Rob, for testing this out on different configurations.

The above-cited errors apparently result from the compiler's not
recognizing the pre-defined "stdin" file handle. It's supposed to be
declared in <stdio.h>, so these errors are odd indeed. I would suspect
that the MinGW definition of "stdin" is the source of the problem.

There's a dozen or so warnings as well, but I think those are the fatal
errors. With perl 5.6.1, instead of getting fatal errors at those lines
I get the warnings:
src\sha.c:730: warning: assignment from incompatible pointer type
.
.
src\sha.c:751: warning: comparison of distinct pointer types lacks a cast

Same thing again, i.e., I suspect a problem with the compiler's

Next I built the module on my perl 5.8.2 (built using Visual Studio
.NET, aka MSVC++ 7, built from ActiveState build 807 source) on the same
Win2k.
It builds fine, but all of the tests except the first produce the 'Free
to wrong pool 15d2bb0 not C0100 t\blah-blah.t line x' error.

Aha! This is the same problem that the user got. I suspected the cause
might be that the C library "free()" function is in conflict with the
Perl memory manager.

Afaik there's no memory management conflict here.

That could be, but "free to wrong pool" may indicate that C's "free()"
is attempting to de-allocate memory to an area that Perl's memory
manager doesn't know about.

I'll have some time to look more closely at this tomorrow night if need
be. All the better if, in the meantime, someone can present some ideas
that might save me some time and brainstrain

(No problems on mandrake-9.1, perl 5.8.0, btw.)

Rob, you've already been a HUGE help in narrowing down (and re-creating)
the problem. I really appreciate your looking into this.

One interesting thing: the CPAN tester for the MSWin32 platform used
ActiveState and VC++ and encountered no problems at all. His exact
configuration can be seen at:

http://www.nntp.perl.org/group/perl.cpan.testers/116990

Kind Regards, Mark

 

[Sisyphus]

One interesting thing: the CPAN tester for the MSWin32 platform used
ActiveState and VC++ and encountered no problems at all. His exact
configuration can be seen at:

http://www.nntp.perl.org/group/perl.cpan.testers/116990

Yes, that cpan tester was running perl 5.6.1. I've found no problems
with that build of perl. It only occurs with 5.8.*.

My original assertion that it's *not* a memory management is starting to
look a little flaky

Cheers,
Rob

 

[Tassilo v. Parseval]

Also sprach Mark Shelor:

Sisyphus wrote:


That could be, but "free to wrong pool" may indicate that C's "free()"
is attempting to de-allocate memory to an area that Perl's memory
manager doesn't know about.

I suspect that it's an maybe issue with threads. The ActiveState perls
are usually threaded ones. Each thread has its own heap of memory. You
get this error when one threads allocates some memory and another thread
tries to free() it.

Also, this thread on the p5porters-list could be some help:

<http://www.nntp.perl.org/group/perl.perl5.porters/87622>

Tassilo

 

[Sisyphus]

The above-cited errors apparently result from the compiler's not
recognizing the pre-defined "stdin" file handle. It's supposed to be
declared in <stdio.h>, so these errors are odd indeed. I would suspect
that the MinGW definition of "stdin" is the source of the problem.

Those compiler warnings are not simply a mingw thing. The very same
lines produce warnings with the MSVC compilers, too.

In src/sha.c (following recomendations in 'perlclib' docs) I changed:

1) Every occurrence of 'FILE*' to 'PerlIO*'
2) Every occurrence of 'stdout' to 'PerlIO_stdout()'
3) Every occurrence of 'stdin' to 'PerlIO_stdin()'
4) Every occurrence of 'fprintf' to 'PerlIO_printf'

I now get only one compiler warning when building the module (as opposed
to the 11 warnings I used to get - some of which became fatal errors on
mingw-built perl 5.8). The one remaining warning is in relation to the
'fgets()' function at line 669 of src/sha.c. 'perldoc perlclib' has some
advice regarding this, too - but I'm a little unsure as to how to put
that advice into practice.
Anyway, with those changes in place, I've now managed to build the
module with my mingw-built perl 5.8.0. Of course, having achieved that,
I now find that it, too, suffers precisely the same problems wrt freeing
to "wrong pool"

Cheers,
Rob