Net runtime crash

[ekz]

Hi guys
I have a very weird bug and will welcome gracefully any help
We developed ASP.Net application that run on production server, Net
Framework 1.1 , IIS 6.0, the application is huge, having hundreds of
user controls and worked under massive requests load : 200-1000
connected sessions and millions of page views per day
The application is stable; the only and urgent problem is that:

About 10 times per day, more frequently at pick hours, we see this
error in windows event log:

Event Type: Error
Event Source: .NET Runtime
Event Category: None
Event ID: 0
Date: 21/12/2005
Time: 18:42:28
User: N/A
Computer: XXX
Description:
The description for Event ID ( 0 ) in Source ( .NET Runtime ) cannot be
found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote
computer. You may be able to use the /AUXSOURCE= flag to retrieve this
description; see Help and Support for details. The following
information is part of the event: .NET Runtime version 1.1.4322.2300-
Setup Error: Failed to load resources from resource file
Please check your Setup.

After this the worker process that connected to application pool is
recycled and application is extremely slowly till it loads all
resources and updated all cache entries (we use server cache
(HttpContext.Current.Cache) to save expensive queries results), it is
unacceptable for our client.

The description of event is very general, and all my Google research
can't help me to find helpful information about source of event.

I do not believe that somebody can tell me exactly the source of the
problem,
But any instructions about what I can to do research the problem:
debugging tools, some tools to get more descriptive event source
description, general tips how to research this kind of problem would be
very, very welcome

Thanks a million for any help

 

[Guest]

Hi,

As you already it is a very generic error and the 2 (quite often) reasons
why you could see this issue is happening is because of...
1) Security issues -> Try using Filemon and Regmon utility from
www.sysinternals.com

2) Some corrupt files in your .NET framework. Uninstall the Framework and
re-install it to see if that helps...

HTH

 

[SharpSmith]

Thanks Rahul
we already used filemon and the outputs looks ok
but reinstalling framework is an interesting idea

many thanks
will keep you updated regarding process

EV

 

[SharpSmith]

hey Rahul,
if i will deploy the application in debug mode
what do you think, will it write more detailed info in event log

Thanks a million

EV

 

[Guest]

Well, deploying the application in debug mode sounded like an interesting
stuff to me initially. But, it is not a recommended practice for one thing
and more, since it is not related to ASP.NET apps (instead looks more related
to the .NET Runtime in general), I wonder if that would help!!

Are you able to reproduce this error by any means?
How often does it happen?
Did you try running the app in a different app pool and check if the issue
happens in the other pool as well?
Also, did you try isolating the application which you think could be causing
it?

 

[SharpSmith]

hi Rahul
i'm very appriciate your help
agree that deploying in debug isn't good, but at that moment i'm ready
for everything that might help to find the origin of the problem

regarding reproducing the error : i cant catch it and it can take weeks
of work cause the app is realy huge

the error occures 2 or 3 times at night time and
can appear 4 times in a hour during pick hours when we have a thousand
of visitors, so may be it is related to site load

the app runs at dedicated app.pool and i can't separate that app for
smaller units

maybe you are familiar with some debugging tool that i can attach to
process to see the crash source description ?

1000000 of thanks
EV

 

[Guest]

I would be very glad if I could help you point towards the cause of the error.

As you caught it quite right, it seems to be load related stuff. Let me ask
this...

1) Did you try using Perfmon logs? If not, do the following...

Create a Performance Monitor log file.
1. Expand "Performance Logs and Alerts"
2. Right Click on "Counter Logs"
3. Choose "New Log Settings..."
4. Enter a descriptive name
5. Note the log file location for later (or go to the "Log Files" tab and
change the location)
6. Click the "Add" button
7. Click the "All Counters" and "All Instances" radio buttons
8. Select the following from the "Performance Object" dropdown, being sure
to "Add" each one as you select it:
• Process
• Thread
• Processor
• Memory
• .NET CLR Data
• .NET CLR Exceptions
• .NET CLR Interop
• .NET CLR Jit
• .NET CLR Loading
• .NET CLR LocksAndThreads
• .NET CLR Memory
• .NET CLR Networking
• .NET CLR Remoting
• .NET CLR Security
• ASP.NET
• ASP.NET Applications
9. Click "Close"
10. Click "OK"

See how these logs behave... For Perfmon analysis, you can use the following
link...
http://msdn.microsoft.com/library/d...de/html/cpconPerformanceCountersForASPNET.asp

Do keep a tab on the PID of your worker process and see if that changes
whenever the error happens. If it does, then it is a crash and we can proceed
further accordingly. Let me know of the output and about PID stuff or if you
have any questions.

Hope that helps.

 

[SharpSmith]

Thanks Rahul
i'm out of office now, but i will test your suggestion first thing when
i'll back to work
regarding PID, i sure that it is changed after every new .Net Runtime
entry in event log
and we now seriously checking the option to reinstall Framework, even
that isn't simple for windows server 2003 and will cause down time for
the site
have to solve it

again, thanks a million for your help, i'll take care to update your if
i'll finally fix it

 

[Guest]

Sure!!

Also if you are sure that PID is getting changed each time, then it must be
a crash and you should be able to catch it using the following method.

1. Go to the link http://beta.microsoft.com.
2. Login in with your passport id.
3. Login using the guest ID "DebugDiag" (it’s case sensitive)
4. Click on the "IIS Debug Diagnostic Tool" link.

Configure DebugDiag
===================
Configure DebugDiag to capture the memory dump when the ExitProcess command
is called. This will ensure that we capture the memory dump when the
faulting thread is terminating the process.

1. Open DebugDiag
2. On the Rules tab, click Add Rule
3. Select Crash and click Next
4. Select "All IIS Processes" and click Next
5. Under Advanced Settings, click Breakpoints
6. Click Add Breakpoint...
7. Select KERNEL32!ExitProcess and change Action Type to Full UserDump
8. Click OK
9. Click Save and Close
10. Click Next through the rest of the wizard

Now, let it run for a while and wait for the issue to happen. Once that
happens, most probably you should be having the dump files created in the
respective folders. Let me know if you get any files with .dmp extension
created in the C:\Program Files\IIS Resources\DebugDiag\Logs\<Name of Rule>
folder

 

[SharpSmith]

Hey Rahul,

My name is Aviad, im replacing EV for taking care of the issue as he has
to get back into writing code

First let me thank you for all the references so far, i appriciate it
alot.

And now down and dirty with business, we have a dump file generated by
DebugDiag, it is huge (around 500mb), but the content in that file
doesnt seem clear to me as it all looks like core function calls...

What do we do from here?

Thanks again,

Aviad

 

[Guest]

Well, it is good news that the files have been generated. If you could send
it to (e-mail address removed) using http://www.yousendit.com/
that would be great.

I will see if there is anything obvious causing that crash and try giving
you some pointers!! Plz ensure that you zip the file before sending it.

 

[SharpSmith]

Hey Raul, iv sent the reports per your request, i hope to hear from you
soon with hopefully something that will help us resolve this!

Thanks alot..
Aviad

 

[Guest]

Hey Aviad,

THanks for the update, but I haven't received anything yet :-( Do you have
any place where you can upload it so that I could download the dumps and
analyze it for you??

 

[SharpSmith]

Hey Raul,

I'v uploaded the files to this locataion :
http://www.layla.co.il/downloads/reports.zip

The file size is 255kb, in that zip there is the memory analysis report
and the crash report.

Thanks again for your help!

Aviad

 

[Guest]

Hi,

First things first... I needed the actual dump file (.dmp) for further
analysis

As per the mht files you have sent me I can suggest TWO things (MUST HAVE
THESE SETTING INTO PLACE, VERRRRRRRY IMPORTANT)...

1) In your web.config for all applications, set debug=false
2) Follow the article http://support.microsoft.com/?id=821268 and do all the
suggested changes.

Hope that helps!!

 

[SharpSmith]

Hey Raul,

Thanks for the references! im going to test it out right now.

Here's the actual dump files :
http://www.layla.co.il/downloads/actualdumps.rar (122MB)

There are 3 dump files in there of 3 different crashes as we belive that
there might be more than 1 thing causing those hard crashes.

Thanks again, waiting to hear back.

Aviad

 

[Guest]

Hi,

Thanks for the dump files. Lets stick with the previous 2 suggestions and
create another set of dumps if those don't help. Pretty much a contenction
problem and the article settings along with the debug=false in your
web.config file should definitely help to some extent.

Let me know the progress!!

 

[Jim Cheshire]

Hey Raul,

Thanks for the references! im going to test it out right now.

Here's the actual dump files :
http://www.layla.co.il/downloads/actualdumps.rar (122MB)

There are 3 dump files in there of 3 different crashes as we belive
that there might be more than 1 thing causing those hard crashes.

Aviad,

Your problem is a stack overflow on thread 13. That is what is crashing the
process. The following stack continues over and over until the stack
overflows.


0x0198f5f4 0x046ef348 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.CMSBaseUserControl.get_CategoryId()
0x0198f5fc 0x064c1de2 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.ManagedModules.CMSManagedModulelBase.get_CIMId()
0x0198f630 0x046ef348 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.CMSBaseUserControl.get_CategoryId()
0x0198f638 0x064c1de2 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.ManagedModules.CMSManagedModulelBase.get_CIMId()
0x0198f66c 0x046ef348 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.CMSBaseUserControl.get_CategoryId()
0x0198f674 0x064c1de2 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.ManagedModules.CMSManagedModulelBase.get_CIMId()
0x0198f6a8 0x046ef348 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.CMSBaseUserControl.get_CategoryId()
0x0198f6b0 0x064c1de2 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.ManagedModules.CMSManagedModulelBase.get_CIMId()
0x0198f6e4 0x046ef348 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.CMSBaseUserControl.get_CategoryId()
0x0198f6ec 0x064c1de2 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.ManagedModules.CMSManagedModulelBase.get_CIMId()
0x0198f720 0x046ef348 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.CMSBaseUserControl.get_CategoryId()
0x0198f728 0x064c1de2 [DEFAULT] [hasThis] I4
Netcraft.Layla3.LaylaSite.Modules.ManagedModules.CMSManagedModulelBase.get_CIMId()
0x0198f75c 0x064c1cf1 [DEFAULT] [hasThis] Void
Polls.PollSingleQuestion.Page_Init(Object,Class System.EventArgs)
0x0198f768 0x03b97724 [DEFAULT] [hasThis] Void
System.Web.UI.Control.OnInit(Class System.EventArgs)
0x0198f778 0x0444783b [DEFAULT] [hasThis] Void
System.Web.UI.UserControl.OnInit(Class System.EventArgs)
0x0198f788 0x064c1c27 [DEFAULT] [hasThis] Void
Polls.PollSingleQuestion.OnInit(Class System.EventArgs)
0x0198f794 0x03aeac02 [DEFAULT] [hasThis] Void
System.Web.UI.Control.InitRecursive(Class System.Web.UI.Control)
0x0198f7b0 0x03aeabc4 [DEFAULT] [hasThis] Void
System.Web.UI.Control.InitRecursive(Class System.Web.UI.Control)
0x0198f7cc 0x03aeabc4 [DEFAULT] [hasThis] Void
System.Web.UI.Control.InitRecursive(Class System.Web.UI.Control)
0x0198f7e8 0x03ae843a [DEFAULT] [hasThis] Void
System.Web.UI.Control.AddedControl(Class System.Web.UI.Control,I4)
0x0198f804 0x03ae8312 [DEFAULT] [hasThis] Void
System.Web.UI.ControlCollection.Add(Class System.Web.UI.Control)
0x0198f814 0x058c7fc6 [DEFAULT] [hasThis] Void
Netcraft.Layla3.LaylaSite.CategoryViewModule.BindTable()
0x0198f824 0x058c7f43 [DEFAULT] [hasThis] Void
Netcraft.Layla3.LaylaSite.CategoryViewModule.Page_Load(Object,Class
System.EventArgs)


--
Jim Cheshire
================================
Blog: http://blogs.msdn.com/jamesche

Latest entry: Digging into Objects
Describes the details of digging into
memory usage with the debugger.

 

[SharpSmith]

First of all, i want to thank Raul for helping me and guiding me through
this, and Jim, thank you for pointing out the recursion. We will try and
address this issue immidietly and ill be sure to report back with the
results!

I want to thank you for your time put into helping me out and getting
this thing pinned down to a certain module in the application.

This has made my week....and the next one

Thank you guys again, ill be letting you know how the results come out
in a day or 2..

Aviad, and the whole development team.

 

[Jim Cheshire]

First of all, i want to thank Raul for helping me and guiding me
through this, and Jim, thank you for pointing out the recursion. We
will try and address this issue immidietly and ill be sure to report
back with the results!

I want to thank you for your time put into helping me out and getting
this thing pinned down to a certain module in the application.

This has made my week....and the next one

Thank you guys again, ill be letting you know how the results come out
in a day or 2..

Aviad, and the whole development team.

No problem! Here's the kicker. You have access to the tools that are
required to troubleshoot this kind of issue. However, most developers don't
have the skill set to debug this kind of thing. Keep an eye out on my blog.
That's why I created it.

--
Jim Cheshire
================================
Blog: http://blogs.msdn.com/jamesche

Latest entry: Digging into Objects
Describes the details of digging into
memory usage with the debugger.