memory managers and malloc/free

[arne.muller]

Hello,

I've read in the FAQ that modern versions of malloc/free can cache
memory, i.e. free does not return memory to the OS so that it may be
re-used by the next malloc call. I was thinking about writing some kind
of high level memory manger for a scientific project, but then I may
just be better off with malloc/free ... .

Here's the background: I need to allocate lots of vectros, matrices and
structures iteratively, specificly - I loop over a list of many data
sets and compare each of the sets with a list of other dataset. The
dataset are spectra with 2 double vectors of datapoints that were
recorded from a mass spectrometer, the spectra are of different size
(30 to 7000 data points in each vector).

For each comparsion I need to allocate a matrix, some vectors and
structures. So I was thinking to implement some kind of thread safe
(I'm planing to run the comparisons in parallel) memory manager that
will keep the matrices, vectors and structures so that they can be
re-used for the next round of comparsion (the vectors and matrices may
need to be re-sized, or I just increase but never decrese them). The
content of the memory can get lost after each round, it's just
important that memory allocation is fast (I've several mio.
comparsions).

Anyway, I'm not sure whether such a memory manager makes much sense if
malloc/free do caching, or maybe there are already efficient caching
systems available. What's your recommendation or expereince with large
numbers of malloc/free calls?

I don't have much experience in C, so even trivial answers may help ;-)
..

(I'm using gcc 2.96 under linux and the standard c-compiler on sun.9

thanks a lot for your help
+kind regards,

Arne

 

[Eric Sosman]

Hello,

I've read in the FAQ that modern versions of malloc/free can cache
memory, i.e. free does not return memory to the OS so that it may be
re-used by the next malloc call. I was thinking about writing some kind
of high level memory manger for a scientific project, but then I may
just be better off with malloc/free ... .
[...]
I don't have much experience in C, so even trivial answers may help ;-)

"You'd just be better off with malloc/free."

Don't optimize prematurely: you are likely to wind up chasing
imaginary problems and ignoring the real ones. This isn't a slam
at your inexperience; even expert programmers have been found to
be startlingly unable to predict which parts of their programs
will be the performance bottlenecks.

Begin by writing a straightforward program using straightforward
techniques. Work with the program until you have confidence that it
operates correctly. Then *if* the performance is inadequate and *if*
profiling and other measurement techniques show that a lot of time is
wasted in malloc/free -- then and only then should you start thinking
about customized wrappers and replacements.

(I'm using gcc 2.96 under linux and the standard c-compiler on sun.9

Aside: If performance is important to you, why are you using such
antique compilers? gcc is up to 4-dot-something, and the Sun Studio
compiler and tool set is free for the downloading. Compiler version
number does not always correlate positively with program speed, but
it's eccentric to insist on versions that were already old when
Clinton was elected.

 

[Clark S. Cox III]

Hello,

I've read in the FAQ that modern versions of malloc/free can cache
memory, i.e. free does not return memory to the OS so that it may be
re-used by the next malloc call. I was thinking about writing some kind
of high level memory manger for a scientific project, but then I may
just be better off with malloc/free ... .

Odds are malloc/free will do just fine for you; it is almost never a
good idea to second-guess your compiler/OS/libraries unless you
actually observe something amiss.

 

[Ark]

Hello,

I've read in the FAQ that modern versions of malloc/free can cache
memory, i.e. free does not return memory to the OS so that it may be
re-used by the next malloc call. I was thinking about writing some kind
of high level memory manger for a scientific project, but then I may
just be better off with malloc/free ... .
[...]
I don't have much experience in C, so even trivial answers may help ;-)

"You'd just be better off with malloc/free."

Don't optimize prematurely: you are likely to wind up chasing
imaginary problems and ignoring the real ones. This isn't a slam
at your inexperience; even expert programmers have been found to
be startlingly unable to predict which parts of their programs
will be the performance bottlenecks.

<snip>

Even if you find a necessity of custom memory manager after heeding to
Eric's very sound advice, do not rush to writing your own, unless you
KNOW your memory usage patterns that only you can take advantage of.
There are a few known approaches worth googling; customalloc is one of them.
- Ark