Help with debugging a hang issue in ASP.NET 1.1 webservice

[chen]

Periodically we see the worker process (w3wp.exe) get into a state that
suggests that it is hanging and results in it not handling requests.
Often we end up recycling the appropriate app pools to get around the
problem. But the frequency at which this is happening is beginning to
concern...

So when this happened again, we got a hang dump using the adplus.vbs &
started looking at the dump in WinDbg. While i'm not an expert with
WinDbg by any means, the following stack trace in one of the thread
caught my attention:

10 Id: 14f4.1580 Suspend: 1 Teb: 7ffaa000 Unfrozen
ChildEBP RetAddr Args to Child
01b0e2e8 7c821b74 77e99af9 d0000144 00000004 ntdll!KiFastSystemCallRet
01b0e2ec 77e99af9 d0000144 00000004 00000000 ntdll!NtRaiseHardError+0xc
01b0e984 01001623 01b0f078 01b0f078 010015e6
kernel32!UnhandledExceptionFilter+0x54b
01b0e9a0 7815e3b2 01b0f078 01b0f078 01b0f078
w3wp!WpUnhandledExceptionFilter+0x3d
01b0e9b0 7c35f0c3 01b0f078 7c35f07d 79030a86
msvcr80!__CxxUnhandledExceptionFilter+0x63
[f:\rtm\vctools\crt_bld\self_x86\crt\prebuild\eh\unhandld.cpp @ 86]
01b0e9bc 79030a86 01b0f078 79030a23 01b0ed20
msvcr71!__CxxUnhandledExceptionFilter+0x46
WARNING: Stack unwind information not available. Following frames may
be wrong.
01b0f050 77e84259 01b0f078 77e6b831 01b0f080
mscoree!EEDllGetClassObjectFromClass+0x11d38
01b0f058 77e6b831 01b0f080 00000000 01b0f080
kernel32!BaseThreadStart+0x4a
01b0f080 7c82eeb2 01b0f164 01b0ffdc 01b0f180
kernel32!_except_handler3+0x61
01b0f0a4 7c82ee84 01b0f164 01b0ffdc 01b0f180 ntdll!ExecuteHandler2+0x26
01b0f14c 7c82ecc6 01b09000 01b0f180 01b0f164 ntdll!ExecuteHandler+0x24
01b0f14c 79209a09 01b09000 01b0f180 01b0f164
ntdll!KiUserExceptionDispatcher+0xe
01b0f460 7922b33d 00000800 00000000 00000000 mscorsvr!SetupThread+0x21
01b0f490 7929b5e6 00000000 793f4438 793f4438
mscorsvr!CoInitializeEE+0x81
01b0f4a4 7929f6ac 00000800 77e6bbc2 00000000
mscorsvr!SetBufferOverrunHandler+0x24
01b0f4cc 7929f7d2 01b0f4d8 00000000 01b0f518
mscorsvr!CCompRC::GetLibrary+0x26
01b0f4dc 7929c0e2 0000171a 000d72b8 00000400
mscorsvr!CCompRC::LoadString+0x11
01b0f518 7925864b 00000000 0000171b 0000171a
mscorsvr!CorMessageBox+0x51
01b0f95c 7922d668 79e69af0 00000000 00000000 mscorsvr!EEDllMain+0x1f1
01b0f984 7922daa5 00000000 79e69af0 00000000 mscorsvr!EEStartup+0x1ac
01b0f9b4 7922db1b 00000000 79e69af0 0000fde9 mscorsvr!TryEEStartup+0x21
01b0f9e8 792292df 74737953 572e6d65 482e6265 mscorsvr!COMStartup+0x55
01b0fb60 79002aa7 79e69af0 79e699fc 79e9acc8
mscorsvr!ClrCreateManagedInstance+0xdf
01b0fb78 79e8ee37 79e69af0 79e699fc 79e9acc8
mscoree!ClrCreateManagedInstance+0x44
01b0fb94 79e8f081 01942d10 00000104 00000000
aspnet_isapi!CreateAppDomainFactory+0x31
01b0fc04 79e762e4 01b0fd48 01b0fc44 01b0fc3c
aspnet_isapi!GetAppDomain+0x118
01b0fe50 79e7619c 01942d10 79e8e231 79e8e244
aspnet_isapi!HttpCompletion:rocessRequestInManagedCode+0x139
01b0fe5c 79e8e244 01942d10 00000000 00000000
aspnet_isapi!HttpCompletion:rocessCompletion+0x24
01b0fe70 791d4096 01942d10 000baa20 793ecec8
aspnet_isapi!CorThreadPoolWorkitemCallback+0x13
01b0fe84 791d4055 000baa20 00000000 791d3fe5
mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19
01b0fea4 791fdacc 00000000 00000000 80a78bcc
mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129
01b0ffb8 77e6608b 000c7ef0 00000000 00000000
mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
01b0ffec 00000000 791fda8b 000c7ef0 00000000
kernel32!BaseThreadStart+0x34

It looks like while trying to create the AppDomain the there was an
unhandled exception. Can anyone shed more light on this? What else
should i be looking at?

Btw, the server is running Windows Server 2003 (SP1) behind a hardware
based load balancer.

chen