Weird. malloc() fails in spite of having enough virtual memory...

[itsolution]

Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

I attached the output of tracking memory usage (using vmstat) until
the malloc failure is encountered.

Any idea on why malloc(for 1K chunk) eventually fails despite it has
enough VM at that moment??

Only 20MB difference in free VM between (when a child myxmlreader
starts) and (when it dies due to malloc failure). So, still enough
VM remains.


(A new request comes in and a child process starts reading a big xml
file)
root 5471 18.6 0.4 75556 7888 d0 D 2:30AM 0:07.66 ./
myxmlreader
root 5468 0.0 0.2 71036 3280 d0 I 2:30AM 0:00.06 ./
myxmlreader
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
1 1 0 718380 1415840 2 0 0 0 2 0 0 1277 10288 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 1 0 719108 1414164 2 0 0 0 2 0 0 1277 10293 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 720632 1412600 2 0 0 0 2 0 0 1277 10299 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 722328 1410924 2 0 0 0 2 0 0 1277 10307 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 725316 1406360 2 0 0 0 2 0 0 1277 10317 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 0 0 727044 1404392 2 0 0 0 2 0 0 1277 10324 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 0 0 730300 1401128 2 0 0 0 2 0 0 1277 10336 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 732288 1399156 2 0 0 0 2 0 0 1277 10346 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 1 0 734096 1397312 2 0 0 0 2 0 0 1277 10357 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 1 0 712780 1352132 2 0 0 0 2 0 0 1277 10365 668 14
86 0
(At this moment, the child process has processed reading/saving one
thousand and several hundreds of xml object and finally malloc has
failed , the child process has died)

root@% ps -aux | grep myxmlreader
root 5468 0.0 0.2 71036 3192 d0 S 2:30AM 0:00.06 ./
myxmlreader

 

[santosh]

Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

I attached the output of tracking memory usage (using vmstat) until
the malloc failure is encountered.

Any idea on why malloc(for 1K chunk) eventually fails despite it has
enough VM at that moment??

Only 20MB difference in free VM between (when a child myxmlreader
starts) and (when it dies due to malloc failure). So, still enough
VM remains.

<snip>

It's hard to say what is the cause of your problem without a detailed look
at your code. To many allocations and deallocations can cause memory
fragmentation to become a real issue. One strategy is to allocate a fixed
number of containers and reuse them, (i.e. read and save, read again...),
until the entire file has been traversed.

You don't need to deallocate the containers after using them once. Why not
reuse them for the next object?

 

[Malcolm McLean]

Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

Sounds very much like you are not freeing the memory. If you have 1 GB
installed, then you will run out at about the 1000 mark if each is a few
hundred K.
However it could be that the request is for a corrupt amount. Try replacing
malloc() with a wrapper that prints out the amout requested on failure.

 

[Richard]

Sounds very much like you are not freeing the memory. If you have 1 GB
installed, then you will run out at about the 1000 mark if each is a
few hundred K.

Did you never hear of virtual memory? How do you think that might affect things?

However it could be that the request is for a corrupt amount. Try
replacing malloc() with a wrapper that prints out the amout requested
on failure.

--

 

[William Hughes]

Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

As pointed out elsethread, fragmentation may be an issue (there is
lots of free memory, but no one chunk that is big enough).
However, since it appears that you only allocate a few megabytes in
total
this does not seem likely. Have you tried not freeing?

However, malloc may be failing for a reason that has nothing to
do with the amount of free memory. It is very common for a malloc
failure to be due to memory corruption. Are you sure that
nowhere in your system is something that
corrupts memory? Does malloc return NULL or does it fail
with a segfault?

- William Hughes

 

[Richard Tobin]

Could you shed some light on this issue?

Without looking at your code, it's hard to say, but some possibilities
are:

- you have a memory limit set, perhaps system-wide or for the user
- you are leaking memory elsewhere (e.g. in the database API)
- you have a bug which is corrupting malloc's data structures

If possible, use a malloc debugger to verify that you are doing the
mallocs and frees you expect. Try running it on a different OS, and
see if you get the same behaviour. Cut out some of the code, and see
whether it still fails.

-- Richard

 

[Clever Monkey]

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

Probably pretty much off-topic:

Check the system docs for how malloc() is implemented. It is possible
for some systems to return an allocation failure for things like not
being able to satisfy the allocation out of swap at the specific moment.
That is, there is often no way to distinguish the specific reason the
allocation failed.

Someone else already mentioned fragmentation. Take a look at "ulimit",
as well, to see what the data segment of a process can be per process.
This value can often grow to the hard limit and stay there, chewing up
VM even though it has been freed.

If your allocations are implemented with brk()/sbrk() or similar system
calls, you can also swap in a library that uses mmap() or other more
modern ways of getting (and releasing) resources.