ASPNET 1.1 Memory Issues / Leak or heap problems

[Guest]

We have a large Asp.net application that is currently crashing our production
servers. What we are seeing is the aspnet_wp eat up a bunch of memory and
then stop unexpectedly. Does not recycle.

Example:
After about 5 hours with a concurrent user base of about 150 users the
application raises the aspnet_wp memory usage by almost 500MB. If our server
guys modify the web.config this data is released and the workerprocess goes
back to a manageable amount of memory.

What we have done:
We have been using the DevPartner studio for memory analysis and code review
as well as CLR Profiler and some Perfmon counters. Everything is showing
that for a single user the application is only using about 1 MB of ram with
temporary object spikes upto 25 mb which are getting collected. We have
scanned our code adding dispose and close methods were possible. Everything
looks fine at least to our eyes and the tools eyes.

Don't know what else there is. We think it may be the large object heap but
not sure. Also, we are using SQL Session state if that makes any difference.

In order to help replicate this problem we create a couple of simple pages
that did things a couple of ways.

1. Have a blank page that has in it's page load a nested for loop that uses
a string builder to create a table 3 columns wide and 20 rows deep. Then use
a response.write to push to the screen.

2. Have another page that does the same thing but uses a Server side html
table (<ASP:Table>) objects to create the same structure.

What we see is that when we do a refresh a bunch of times the worker process
sits stable for a few seconds and then starts to blow up in about 200KB
chunks for about 20MB of extra memory consumption. After leaving it idle for
a good 15 minutes or so it still consumes that memory. If we run the same
page with a manual GC.Collect it still tended to grow but not as much.

If we added the GC.GetTotalMemUsage(true) method call we saw that the memory
usage usually stayed pretty consistant.

What we are seeing is probably something similar to what we are getting in
our apps.

Is there something we are missing? We really need help with this one.
Also, I think our servers have the latest server packs for .net and
Win2KServer.

We also see this same behavior on our client machines when running through
the ide.

We are trying to figure this out because we are getting ready to release an
even larger app that will have roughly 500-600 concurrent users and we don't
want it to crash all the time either.

One thing that is weird is that in perfmon when setting up all the .net
memory and asp.net counters nothing even comes close to showing the memory
usage (even in totals) that taskmanager is reporting for memory consumption.
With that being said were is it going???????

We have been running some memory leak analysis as well and found that we had
a minor contained leak of about 150KB. I say contained because on a per user
basis we were never able to get it to grow beyond that...

Any help that people can provide would be great.

 

[Guest]

Please can you provide more information? Hardware specs will be pretty good
as well as if you are running Windows 2000 or 2003, what version of that
operating system. Running IIS 5 or 6? How much memory on the computer?

Are you using C# Cache class?

Let's see if we can figure it out.

Thanks
Al

 

[Guest]

Please can you provide more information? Hardware specs will be pretty good
as well as if you are running Windows 2000 or 2003, what version of that
operating system. Running IIS 5 or 6? How much memory on the computer?

Are you using C# Cache class?

Let's see if we can figure it out.

Thanks
Al

 

[Guest]

Please can you provide more information? Hardware specs will be pretty good
as well as if you are running Windows 2000 or 2003, what version of that
operating system. Running IIS 5 or 6? How much memory on the computer?

Are you using C# Cache class?

Let's see if we can figure it out.

Thanks
Al

 

[Guest]

Please can you provide more information? Hardware specs will be pretty good
as well as if you are running Windows 2000 or 2003, what version of that
operating system. Running IIS 5 or 6? How much memory on the computer?

Are you using C# Cache class?

Let's see if we can figure it out.

Thanks
Al

 

[Guest]

Please can you provide more information? Hardware specs will be pretty good
as well as if you are running Windows 2000 or 2003, what version of that
operating system. Running IIS 5 or 6? How much memory on the computer?

Are you using Cache class?

Let's see if we can figure it out.

Thanks
Al

 

[Guest]

The server is Windows 2000 Server it has 3GB of ram IIS 5, I know it's a dual
processor but not sure what the processors are.

We hope to get Window 2003 Server and IIS6 in place later this year but
that's dosen't help the problem now.

On the client I have a dual Xeon with 1GB of ram on w2K pro. Seeing a
similar issue for memory escelation without release. Even left it running
overnight with no activity and it never cleared up.

We are not using the Cache class. We even tried to tell it to not cache
some of the pages to see what would happen and it did not change anything.

Thanks.

 

[Scott Allen]

Hi JCauble:

You might want to use the ADPlus utility to get snapshots of the
process before it terminates. This and other tricks are described in
the following document:

http://msdn.microsoft.com/library/en-us/dnbda/html/DBGrm.asp?frame=true


HTH,

 

[Juan T. Llibre]

Your nntp poster is stuttering, Albert.

 

[Guest]

I had some problems using Windows 2000 with large applications, I moved the
same ASP.NET app to 2003 Enterprise/IIS 6 and memory got released.

Always thought that there was a problem with IIS 5 releasing memory

 

[Guest]

I am downloading this stuff now. I will look to see what it can find out.
However we have been using the Dev Partner Suite of tools to watch the Heap
and other memory usage and it still does not correlate to what Taskmanager is
saying is being used. Also, neither does the MemoryTest in the sample app
for the debugging stuff.

As an example if you load up the memory page and just keep clicking on the
Refresh stats page you will see the worker process start to consume memory.
even after a free memory command it does not change. It will only grow for a
short time since there is not much on the page but it does grow.

Our applications do not use the Cacheing objects so it does not directly
relate to what we are seeing.

Is it possible that ASP.Net is using Cacheing objects behind the scenes and
it is blowing up the process usage of ram?

Don't know.

 

[Guest]

So is the question really about IIS not releaseing or ASP.Net? We cannot
move to Win2003 for several more months and we need to fix the problem now.

Any other thoughts.

Thanks.

 

[Ken Cox [Microsoft MVP]]

If I were you, I'd open a support incident with Microsoft for something that
complicated. One-on-one interaction would have a better chance - plus they
have tools to help trace problems like yours.

 

[Scott Allen]

I am downloading this stuff now. I will look to see what it can find out.
However we have been using the Dev Partner Suite of tools to watch the Heap
and other memory usage and it still does not correlate to what Taskmanager is
saying is being used. Also, neither does the MemoryTest in the sample app
for the debugging stuff.

It can be tough, because different tools have a different
interpretation of "memory usage". There is reserved memory, committed
memory, the working set, and virtual memory. The task manager reports
the size of the working set, which is the sum of all memory pages the
application currently has in physical RAM only.

As an example if you load up the memory page and just keep clicking on the
Refresh stats page you will see the worker process start to consume memory.
even after a free memory command it does not change. It will only grow for a
short time since there is not much on the page but it does grow.

By 'free memory command' do you mean you force a garbage collection?

Our applications do not use the Cacheing objects so it does not directly
relate to what we are seeing.

Is it possible that ASP.Net is using Cacheing objects behind the scenes and
it is blowing up the process usage of ram?

It's always a possibility the cache has gone berserk, but generally
it's a bit smarter than this and at the least would have a graceful
"recycle" and not an unexpected termination.

 

[Guest]

Well, I am trying to use this SOS / ADPlus thing but it's just not getting me
anywhere is this able to work with .net 1.1. The samples all point out stuff
in the 1.0 framework.

Also, I added the virtual memory item in task manager and noticed that most
of the memory is in the virtual memory object. However when watching perfmon
only about 13 MB was being used mostly in the Gen(0) heap. When garbage
collection fired it cleared out most of it but the worker process did not go
down in size.

The only way I have been able to get it to go down is a recycle.

Thanks.

 

[Guest]

We may just do that. With it being a Friday though I'll probably wait until
next week. I figure we will need more time to do some of this analysis.
What I was hoping for was some pointers / info to maybe help eliminate or
shine some light on what we are seeing.

Thanks.

 

[Scott Allen]

Well, I am trying to use this SOS / ADPlus thing but it's just not getting me
anywhere is this able to work with .net 1.1. The samples all point out stuff
in the 1.0 framework.

Ken has a good suggestion - the MS support people will be able to
help. It's tough to learn these tools on the fly.

Also, I added the virtual memory item in task manager and noticed that most
of the memory is in the virtual memory object. However when watching perfmon
only about 13 MB was being used mostly in the Gen(0) heap. When garbage
collection fired it cleared out most of it but the worker process did not go
down in size.

There is a distinction to make between "total memory usage" and
"amount of free memory in the gen 0 heap". Overall memory usage can
stay consistent while the free heap space size bounces up and down
between collections. In other words, I can clean out my desk drawers
but they still take up the same amount of space in the room - it's
just now they are empty and I can fill them up with vodka bottles
again.