AppPool Recycle Failure

G

Guest

I'm running a Server Farm of 5 presentation layer machines
There are 55 different apppools, each is dedicated to a single .net applicatio
Only approximately 10 are running at any time and 8 of those get enough traffic to never shut down
1 in an unknown number of app pool restarts, from both timed recycle as well as timed shut down, the application is never able to answer a request
It appears to attempt the JITC and never completes it to give a response on the application
Have looked at this through the IIS discussion group, and they are pretty sure this is a code issue on our side, but since we have been unable to reproduce the problem in a development environment it is very difficult to troublshoot

The applications use the same basic code base with modifications in presentation
All uncached data comes from a Round Robined set of Middle tier servers, which switch machine after each request. Round Robin is provided by a list and custom code calls each machine in order

The only information we really have is that this occurs on recycles/restarts it does not matter if the server is being pounded, or is under light load
In looking at the output of IISState we found one of these errors for each thread that was being orphaned by IIS and showing up as Current Anonymous Users under Web Service Performance counters, as well as the Requests Current ASP.Net Counters

Thread ID: 1
System Thread ID: f0
Kernel Time: 0:0:0.
User Time: 0:0:0.
Other information: Thread is waiting for a lock to be released. Looking for lock owner
Owning thread System ID: 1f7
Thread Type: AS
Executing Page: ASP.dll symbols not found. Unable to locate ASP page
Continuing with other analysis

# ChildEBP RetAddr
00 009ffc3c 77f43741 SharedUserData!SystemCallStub+0x
01 009ffc40 77f5d64e ntdll!ZwWaitForSingleObject+0x
02 009ffc7c 77f42044 ntdll!RtlpWaitForCriticalSection+0x12
03 009ffc84 79e8d946 ntdll!RtlEnterCriticalSection+0x4
04 009ffccc 79e75739 aspnet_isapi!GetAppDomain+0x9
05 009fff1c 79e755e9 aspnet_isapi!HttpCompletion::processRequestInManagedCode+0x13
06 009fff24 79e8cb80 aspnet_isapi!HttpCompletion::processCompletion+0x2
07 009fff68 792e9155 aspnet_isapi!CorThreadPoolWorkitemCallback+0x2
08 009fff98 792e92cf mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x2
09 009fffb8 77e4a990 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x12
0a 009fffec 00000000 kernel32!BaseThreadStart+0x3

When the site fails to restore, each new connection creates a Concurrent request that is never answered. We have allowed some of these to go past 3 hours by removing the machine they were running on from the Load Balance

The fun part is when this happens on a site that has been set to use a Web Garden, this will happen to just one of the threads, so the Automated checking software won't pick it up as a failure, because only every 3rd request is failing

Where can we go from here?
 
N

Natty Gur

Hi,

mmm it wont be easy but certainly will be solved.

I suggest you to use post mortem debug tolls such as adplus and windows
debug tools. I add a link that will guide you through this process :
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/h
tml/DBGch01.asp

Those tools will help you to track the problem.

Natty Gur[MVP]

blog : http://weblogs.asp.net/ngur
Mobile: +972-(0)52-8888377


*** Sent via Devdex http://www.devdex.com ***
Don't just participate in USENET...get rewarded for it!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top