fwrite consuming lot of memory help !!!!

[phil]

Hi ,

I am running a C/C++ program on a solaris 5.8 machine. This parituclar
application has a module which saves data to a file. The module uses
fwrite() function to save data.

The fwrite function write about 500 MB of data to a file. The problem
which I am facing is, the memory consumtion of the process increases
during the fwrite but does not decrease once the fwrite is over.

I tried fflush(filepointer) after the fwrite but did not help.

The fwrite function writes all the data in once single fwrite()
statement. I initially thought the increase in memory was because of
this. So I tried writing data in smaller chunks of sizes 100 MB and 50
MB but still the memory utilization of the process does not decreases
once the fwrite is over.

As a result a lot of the main memory is being eaten by this process and
thus making the system very slow.

Please help me .


Thanks in advance

 

[Walter Roberson]

I am running a C/C++ program on a solaris 5.8 machine.

The difference might be important to the question -- as might be
whether you have a C main routine calling C++ or a C++ main routine
calling C.

This parituclar
application has a module which saves data to a file. The module uses
fwrite() function to save data.

The fwrite function write about 500 MB of data to a file. The problem
which I am facing is, the memory consumtion of the process increases
during the fwrite but does not decrease once the fwrite is over.

I cannot -think- of any C library routine that might act like that.
I could imagine C++ potentially acting like that, but you'd want to
check with C++ people as my imagination might be overactive in that
regard.

Generally speaking, in Unix, if a process needs to allocate
memory, then it is unlikely to be able to give that memory back
to the operating system. Instead, the process would keep the
memory against future growth. There are some memory usage patterns
that result in memory fragmentation that can lead to indefinite
growth, at least with the typical internal memory deallocation
strategies.

But C's fwrite() is not likely to do dynamic memory allocation.
The C fwrite() interface does not require it to; and POSIX
defines a maximum object size that it promises will be written
"atomically", so writing by chunks that big would be sufficient.

 

[Chris Hulbert]

Hi ,

I am running a C/C++ program on a solaris 5.8 machine. This parituclar
application has a module which saves data to a file. The module uses
fwrite() function to save data.

The fwrite function write about 500 MB of data to a file. The problem
which I am facing is, the memory consumtion of the process increases
during the fwrite but does not decrease once the fwrite is over.

I tried fflush(filepointer) after the fwrite but did not help.

The fwrite function writes all the data in once single fwrite()
statement. I initially thought the increase in memory was because of
this. So I tried writing data in smaller chunks of sizes 100 MB and 50
MB but still the memory utilization of the process does not decreases
once the fwrite is over.

As a result a lot of the main memory is being eaten by this process and
thus making the system very slow.

Please help me .


Thanks in advance

Very likely it's an OS issue.

<OT>
Linux has the same problem. It buffers disk I/O. So when you call
fwrite it's being stored through a buffer. I had some similar problems
with some HDF5 files I was writing. The way to test this, is after you
write the file (and your memory is low), delete the file. The way this
works on Linux is that the buffer is determined to be dirty (since the
file doesn't exist anymore, so it clears the buffer restoring your
memory. I don't know if solaris has the dd command, but you should
seem similar behavior. These are the linux commands for that.

% dd if=/dev/zero of=junk bs=10MB count=50 #generate a 500MB file
% # by now top will show not much free memory left.
% rm junk
</OT>

 

[Lawrence Kirby]

Hi ,

I am running a C/C++ program on a solaris 5.8 machine.

You need to decide whether it is a C program or a C++ program. Since
you're posting to comp.lang.c I'll assume C.

This parituclar
application has a module which saves data to a file. The module uses
fwrite() function to save data.

The fwrite function write about 500 MB of data to a file. The problem
which I am facing is, the memory consumtion of the process increases
during the fwrite but does not decrease once the fwrite is over.

This may be just an artefact of your virtual memory system. The fwrite()
call clearly accesses all of the 500MB of memory which may cause it to be
allocated, swapped in etc. depending on the conditions.

Try replacing the fwrite() call with a loop that writes data out byte by
byte and see if the same thing happens. If so try a loop that simply reads
the data from memory, but make sure the compiler doesn't just optimise it
away.

I tried fflush(filepointer) after the fwrite but did not help.

Memory allocated for a stream is unlikely to be freed until the stream is
closed. Even then freed memory may not be returned to the OS. It depends
on the individual implementation of malloc() as to whether it is capable
of this, although streams may or may not use malloc() internally.

C streams usually just allocate a single fairly small buffer (perhaps a
few K). You can control that with setbuf() or setvbuf(). If you supply,
say, a static buffer calling one of those after opening the stream it is
unlikely that the library will allocate more.

The fwrite function writes all the data in once single fwrite()
statement. I initially thought the increase in memory was because of
this. So I tried writing data in smaller chunks of sizes 100 MB and 50
MB but still the memory utilization of the process does not decreases
once the fwrite is over.

As a result a lot of the main memory is being eaten by this process and
thus making the system very slow.

How much increase are we talking about? Does it increase further if you
write the data out a second time?

Lawrence

 

[phil]

You need to decide whether it is a C program or a C++ program. Since
you're posting to comp.lang.c I'll assume C.


This may be just an artefact of your virtual memory system. The fwrite()
call clearly accesses all of the 500MB of memory which may cause it to be
allocated, swapped in etc. depending on the conditions.

Try replacing the fwrite() call with a loop that writes data out byte by
byte and see if the same thing happens. If so try a loop that simply reads
the data from memory, but make sure the compiler doesn't just optimise it
away.


Memory allocated for a stream is unlikely to be freed until the stream is
closed. Even then freed memory may not be returned to the OS. It depends
on the individual implementation of malloc() as to whether it is capable
of this, although streams may or may not use malloc() internally.

C streams usually just allocate a single fairly small buffer (perhaps a
few K). You can control that with setbuf() or setvbuf(). If you supply,
say, a static buffer calling one of those after opening the stream it is
unlikely that the library will allocate more.



How much increase are we talking about? Does it increase further if you
write the data out a second time?

Lawrence

Hi

The program is written in C.The memory increases only the first time I
call the fwrite function. When I call the fwrite the second time,
memory consumtion does not increase. It stays at the same level where
the first fwrite had left it. I tried setvbuf. I allocated a 5 MB
buffer and used the buffer for fwrite. I wrote 5 Mb in once fwrite
call, however the memory kept increasing.

Phil

 

[phil]

You need to decide whether it is a C program or a C++ program. Since
you're posting to comp.lang.c I'll assume C.


This may be just an artefact of your virtual memory system. The fwrite()
call clearly accesses all of the 500MB of memory which may cause it to be
allocated, swapped in etc. depending on the conditions.

Try replacing the fwrite() call with a loop that writes data out byte by
byte and see if the same thing happens. If so try a loop that simply reads
the data from memory, but make sure the compiler doesn't just optimise it
away.


Memory allocated for a stream is unlikely to be freed until the stream is
closed. Even then freed memory may not be returned to the OS. It depends
on the individual implementation of malloc() as to whether it is capable
of this, although streams may or may not use malloc() internally.

C streams usually just allocate a single fairly small buffer (perhaps a
few K). You can control that with setbuf() or setvbuf(). If you supply,
say, a static buffer calling one of those after opening the stream it is
unlikely that the library will allocate more.



How much increase are we talking about? Does it increase further if you
write the data out a second time?

Lawrence

Hi

The program is written in C.The memory increases only the first time I
call the fwrite function. When I call the fwrite the second time,
memory consumtion does not increase. It stays at the same level where
the first fwrite had left it. I tried setvbuf. I allocated a 5 MB
buffer and used the buffer for fwrite. I wrote 5 Mb in once fwrite
call, however the memory kept increasing.

Phil

 

[Robert Harris]

[snip]

Hi

The program is written in C.The memory increases only the first time I
call the fwrite function. When I call the fwrite the second time,
memory consumtion does not increase. It stays at the same level where
the first fwrite had left it. I tried setvbuf. I allocated a 5 MB
buffer and used the buffer for fwrite. I wrote 5 Mb in once fwrite
call, however the memory kept increasing.

Phil

<ot>
Use truss on the program to see what is really happening.
</ot>

Robert

 

[Lawrence Kirby]

On Mon, 08 Aug 2005 02:51:54 -0700, phil wrote:

....

The program is written in C.The memory increases only the first time I
call the fwrite function. When I call the fwrite the second time,
memory consumtion does not increase. It stays at the same level where
the first fwrite had left it. I tried setvbuf. I allocated a 5 MB
buffer and used the buffer for fwrite. I wrote 5 Mb in once fwrite
call, however the memory kept increasing.

Then it sounds like a system/kernel issue in memory management and I/O
handling rather than a C issue. It sounds like you need to discuss this in
a Solaris newsgroup.

Lawrence