Memory release

[Sheldon]

Hi,

I have a script that uses up a lot of memory to process some data and
write it to a very large file.
This script is written in C. I've made sure that all the memory
allocated in the script is released before making a system() call to
compress the file and then exiting the script.
The attempt to compress failed because fork returned an -1 error
stating that there is insufficient memory available. It seems C keeps
the memory locked until it ends. Is there away to get C to release the
memory allocated before it exits?

Appreciate your help,
/M

 

[Joakim Hove]

Well,

show us some code - that will greatly increase your chance of getting
usable help.

This script is written in C.

I am no language laywer; but calling a compiled program written in C a
"script" is at least unconventional.

Is there away to get C to release the memory allocated before it exits?

Well certainly YES.

Joakim

 

[Guest]

Well,

show us some code - that will greatly increase your chance of getting
usable help.


I am no language laywer; but calling a compiled program written in C a
"script" is at least unconventional.


Well certainly YES.

not necessarily in C there isn't.

 

[Chris Dollin]

Well certainly YES.

Using what? `free` is under no obligation to hand back memory to the
OS, as far as I can tell.

 

[Marston]

Using what? `free` is under no obligation to hand back memory to the
OS, as far as I can tell.

--
"It is seldom good news."      ~Crystal Ball~, /The Tough Guide to Fantasyland/

Hewlett-Packard Limited     Cain Road, Bracknell,                registered no:
registered office:          Berks RG12 1HN                       690597 England

The code/program is several about 2000 lines with several long
functions so I didn't want to post it all here.
Since the "problem" is more theoretical I thought I'd ask a general
question that Chris I think answered.
If free doesn't hand back memory to the OS, how can I force it to do
so?

I wrote a little program that does a system call from C and it works
(see below). When I insert this in my main C program just before
exiting main() and after freeing all allocated memory the system()
call returns -1 and the error "not enough memory". So I think that
when the memory is freed it's not returned to the OS before the
program ends. Is there a solution to this problem?
This file that I'm trying to zip is 504 MB so it needs a lot of
memory.


#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

int main(void) {

int status = -9999;

char cmd[] = "/bin/bzip2 /home/c_programming/trunk/complete/
ECMWF_NWP_20080910-0000-00.NC";

printf("Zipping file using bzip2 via system call...\n");
status = system(cmd);
printf("%d\n",status);

if(status != 0) {
perror("Error found!");
printf("errno = %d.\n", errno);
fprintf(stderr,"Failed to zip output file");
exit(EXIT_FAILURE);
}

return EXIT_SUCCESS;

}


Thanks,
/Marston

 

[Chris Dollin]

The code/program is several about 2000 lines with several long
functions so I didn't want to post it all here.
Since the "problem" is more theoretical I thought I'd ask a general
question that Chris I think answered.
If free doesn't hand back memory to the OS, how can I force it to do
so?

You can't. (It's /possible/ that some particular implemementation
would have a `freeEveryThingAndGiveItBackToTheOSRightNowBang()`
function, but you certainly can't count on it.)

I wrote a little program that does a system call from C and it works
(see below). When I insert this in my main C program just before
exiting main() and after freeing all allocated memory the system()
call returns -1 and the error "not enough memory". So I think that
when the memory is freed it's not returned to the OS before the
program ends. Is there a solution to this problem?

Yes, the solution is quite easy for this particular problem.

#whatever

int main(void)
{
system( "yourFileCreatingProgram" );
system( "yourBzippingCommandLine" );
return 0;
}

Now the UseMuchMemory program has almost certainly finished and gone
before you run bzip.

[Of course you could just write this as a two-line bash/ruby/perl/python
script, but that would be off-topic.]

 

[Bill Reid]

Joakim Hove wrote:


If free doesn't hand back memory to the OS, how can I force it to do
so?

Since you like to call your program a script, and are using a
system() call to run the second part of the script, why not write
a REAL script, with the first part running your memory hog
program, and the second part running the compress command.

If indeed your problem is that the OS still sees the gigantic
chunk of memory as allocated to your program, when the
memory hog program in your script exits, THEN the memory
should be available to other programs, such as your compress
command...

 

[BartC]

If free doesn't hand back memory to the OS, how can I force it to do
so?

If your program uses lots of small, fragmented allocations then C may have
trouble in recovering memory in big enough blocks to be usefully returned to
the OS. (Maybe it just doesn't include the logic to identify the blocks, but
I'm guessing.)

If your C's free() works better with fewer, bigger allocations, then you can
possibly create your own allocator which works with large malloc() blocks.
Then you deallocate these large blocks at the end.

This file that I'm trying to zip is 504 MB so it needs a lot of
memory.

That's surprising, I thought Zip only really needed memory for it's tables,
and would not be constrained by available memory. Maybe you need a better
zip program.

 

[Guest]

from OP
****
I've made sure that all the memory
allocated in the [program] is released before making a system() call
to
compress the file and then exiting the script.
The attempt to compress failed because fork returned an -1 error
stating that there is insufficient memory available. It seems C keeps
the memory locked until it ends. Is there away to get C to release
the
memory allocated before it exits?
****

That would be a surprise to many people since he didn't say
"automatically".

I fail to see your point. In C there is no guaranteed method to ensure
allocated memory is released for the use of other processes. Hence I
disagreed with the poster who said "certainly YES". So which bit did
I get wrong and what does "automatically" have to do with it?

 

[Nate Eldredge]

That's surprising, I thought Zip only really needed memory for it's tables,
and would not be constrained by available memory. Maybe you need a better
zip program.

It looks like it's bzip2. Like any good compression program, bzip2
compresses the file in blocks, so its memory requirement doesn't grow
with the size of the file. However, by default its baseline memory
requirement is fairly large (about 9MB on my machine). It has an
option -s which reduces it to about 3MB at the expense of compression
ratio.

 

[Beej Jorgensen]

char cmd[] = "/bin/bzip2 /home/c_programming/trunk/complete/
ECMWF_NWP_20080910-0000-00.NC";

The quickest solution is what the others have posted. But here are some
more ideas for the sake of fun.

It might (or might not) be useful to pass the "-s" switch to bzip2 to
reduce memory usage. It also looks like lower compression consumes less
memory.

A general approach that can sometimes be effective is to try to generate
your data incrementally and write it to a file as you go (as opposed to
storing all data in memory at the same time.) Sometimes the nature of
the data generation makes this non-trivial, though.

There are potentially quite a few non-portable ways around this problem;
here are a few Unixy ones, assuming this is a Unix-like you're using:

You could generate the data incrementally and pipe it into bzip2 (see
popen()); this is just a variant on the incremental approach, above.

But if you have to generate all the data at once:

One idea is to create your file empty, and then mmap() it in and
generate your data. After you unmap it, memory usage should return to
normal (it does under Linux anyhow.) Bonus: the file is all ready to
bzip after the unmap.

Some malloc libraries might allow you to control if free() returns
memory to the system.

Finally you could go to the OS level and allocate and deallocate memory
with sbrk()/brk(). This doesn't mix cleanly with malloc, so be careful.

A newsgroup specific to your system will have more and better
information if you wanted to go that route.

-Beej

 

[Chris Dollin]

That would be a surprise to many people since he didn't say
"automatically".

How would you (portably) do it manually?