Debbugging help! (.NET 1.1 Framework Garbage Collection Problems)

Discussion in 'ASP .Net' started by Cheung, Jeffrey Jing-Yen, Jul 8, 2003.

  1. Hi all,

    I need some help with debugging a "deadlock" in a production
    environment. I've managed to get a crash dump and have tried to
    analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
    in debugging to pinpoint the exact problem; however, I have narrowed it
    down to the garbage collection process. All threads seem to be waiting
    for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
    up 100% cpu.

    Can someone suggest what methods I could use to fully show and pinpoint
    where the GC is locking up? I have briefly skimmed the SOS docs and a
    lot of the exposed functions seem useful; however, I can't really make
    heads or tails out of them. I've examined the heap via !dumpheap -stat,
    and the largest object that comes back is a whopping 22 MB System.String
    object. In the MSDN docs, what is supposed to follow is a listing of
    Large Objects; however, I figure that maybe because I am using the 1.1
    framework, large objects aren't really differentiated any differently.
    I managed to get the address by issuing a dumpheap -mt <mt of the
    System.String> and I got a huge listing of addresses that really don't
    make much sense to me. I certainly don't want to issue an !objsize
    <addr> on each of these (there are over 30k addresses).

    I think my biggest concern is that I want to be able to load the symbols
    of the .NET assemblies that are being used so that I can analyze the
    managed stack thoroughly, yet I'm not too certain how to do this. If
    that's not necessary to find out the core problem, then I guess I need
    to be directed towards the right place.

    Thanks in advance everyone,

    Jeff Cheung
     
    Cheung, Jeffrey Jing-Yen, Jul 8, 2003
    #1
    1. Advertising

  2. Are you sure it's a single 22 MB string, and not 22 MB allocated
    for all strings?

    You can use !dumpheap (without parameters) to dump all objects.
    If you see any huge strings in the list you can then do !dumpobj on
    them to find out the contents.

    You could also try examining the way your app allocates objects
    using something like the CLR Allocation Profiler:

    http://msdn.microsoft.com/library/d...y/en-us/dndotnet/html/highperfmanagedapps.asp

    Basically, it looks like you need to either reduce the load on GC
    by optimizing your code or use a bigger machine. I noticed from your
    post to the windbg newsgroup that your process is using the
    workstation version of GC (mscorwks.dll) instead of mscorsvr.dll.
    Does the app run on a single proc machine? As far as I know
    ASP.NET should load mscorsrv.dll on MP systems.


    "Cheung, Jeffrey Jing-Yen" wrote:

    > Hi all,
    >
    > I need some help with debugging a "deadlock" in a production
    > environment. I've managed to get a crash dump and have tried to
    > analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
    > in debugging to pinpoint the exact problem; however, I have narrowed it
    > down to the garbage collection process. All threads seem to be waiting
    > for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
    > up 100% cpu.
    >
    > Can someone suggest what methods I could use to fully show and pinpoint
    > where the GC is locking up? I have briefly skimmed the SOS docs and a
    > lot of the exposed functions seem useful; however, I can't really make
    > heads or tails out of them. I've examined the heap via !dumpheap -stat,
    > and the largest object that comes back is a whopping 22 MB System.String
    > object. In the MSDN docs, what is supposed to follow is a listing of
    > Large Objects; however, I figure that maybe because I am using the 1.1
    > framework, large objects aren't really differentiated any differently.
    > I managed to get the address by issuing a dumpheap -mt <mt of the
    > System.String> and I got a huge listing of addresses that really don't
    > make much sense to me. I certainly don't want to issue an !objsize
    > <addr> on each of these (there are over 30k addresses).
    >
    > I think my biggest concern is that I want to be able to load the symbols
    > of the .NET assemblies that are being used so that I can analyze the
    > managed stack thoroughly, yet I'm not too certain how to do this. If
    > that's not necessary to find out the core problem, then I guess I need
    > to be directed towards the right place.
    >
    > Thanks in advance everyone,
    >
    > Jeff Cheung
     
    Pavel Lebedinsky, Jul 9, 2003
    #2
    1. Advertising

  3. Pavel,

    Thanks again for your replies; you've been a great help. While I am
    waiting for !dumpheap to finish (it is taking awhile), I have a few
    questions.

    Will I have to build the CLR Allocation Profiler with the 1.1 framework
    in order for it to profile my application? I guess this is kind of a
    silly question. I assume that the profiler would just attach on to the
    ASPNET worker process, so the framework would be independent, but I
    thought I'd check first since I'm known to be wrong on many occasions.
    Through a Visual Studio.NET command prompt, I ran nmake and built the
    CLRProfiler objects, ran it, and tried to Profile the ASP.NET
    application. It kept sitting there waiting for a response even though I
    kept hitting the page. Quite interesting...

    I'll post my findings once I get dumpheap's results.

    Thanks again,

    Jeff Cheung

    Pavel Lebedinsky wrote:

    > Are you sure it's a single 22 MB string, and not 22 MB allocated
    > for all strings?
    >
    > You can use !dumpheap (without parameters) to dump all objects.
    > If you see any huge strings in the list you can then do !dumpobj on
    > them to find out the contents.
    >
    > You could also try examining the way your app allocates objects
    > using something like the CLR Allocation Profiler:
    >
    > http://msdn.microsoft.com/library/d...y/en-us/dndotnet/html/highperfmanagedapps.asp
    >
    > Basically, it looks like you need to either reduce the load on GC
    > by optimizing your code or use a bigger machine. I noticed from your
    > post to the windbg newsgroup that your process is using the
    > workstation version of GC (mscorwks.dll) instead of mscorsvr.dll.
    > Does the app run on a single proc machine? As far as I know
    > ASP.NET should load mscorsrv.dll on MP systems.
    >
    >
    > "Cheung, Jeffrey Jing-Yen" wrote:
    >
    >
    >>Hi all,
    >>
    >>I need some help with debugging a "deadlock" in a production
    >>environment. I've managed to get a crash dump and have tried to
    >>analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
    >>in debugging to pinpoint the exact problem; however, I have narrowed it
    >>down to the garbage collection process. All threads seem to be waiting
    >>for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
    >>up 100% cpu.
    >>
    >>Can someone suggest what methods I could use to fully show and pinpoint
    >>where the GC is locking up? I have briefly skimmed the SOS docs and a
    >>lot of the exposed functions seem useful; however, I can't really make
    >>heads or tails out of them. I've examined the heap via !dumpheap -stat,
    >>and the largest object that comes back is a whopping 22 MB System.String
    >>object. In the MSDN docs, what is supposed to follow is a listing of
    >>Large Objects; however, I figure that maybe because I am using the 1.1
    >>framework, large objects aren't really differentiated any differently.
    >>I managed to get the address by issuing a dumpheap -mt <mt of the
    >>System.String> and I got a huge listing of addresses that really don't
    >>make much sense to me. I certainly don't want to issue an !objsize
    >><addr> on each of these (there are over 30k addresses).
    >>
    >>I think my biggest concern is that I want to be able to load the symbols
    >>of the .NET assemblies that are being used so that I can analyze the
    >>managed stack thoroughly, yet I'm not too certain how to do this. If
    >>that's not necessary to find out the core problem, then I guess I need
    >>to be directed towards the right place.
    >>
    >>Thanks in advance everyone,
    >>
    >>Jeff Cheung

    >
    >
    >
     
    Cheung, Jeffrey Jing-Yen, Jul 9, 2003
    #3
  4. As of right now, !dumpheap is still going. I had to restart windbg
    because it locked up on me yesterday, so I restarted the debug session
    and it's still going. I understand that it is dumping all the objects
    in the heap, and this is a 222 mb crash dump, but it has been running
    for quite some time... I wonder if maybe this is the problem. Does too
    many objects in the dump heap equate to too many objects for the GC to
    collect?

    Regards,

    Jeff Cheung

    Pavel Lebedinsky wrote:

    > Are you sure it's a single 22 MB string, and not 22 MB allocated
    > for all strings?
    >
    > You can use !dumpheap (without parameters) to dump all objects.
    > If you see any huge strings in the list you can then do !dumpobj on
    > them to find out the contents.
    >
    > You could also try examining the way your app allocates objects
    > using something like the CLR Allocation Profiler:
    >
    > http://msdn.microsoft.com/library/d...y/en-us/dndotnet/html/highperfmanagedapps.asp
    >
    > Basically, it looks like you need to either reduce the load on GC
    > by optimizing your code or use a bigger machine. I noticed from your
    > post to the windbg newsgroup that your process is using the
    > workstation version of GC (mscorwks.dll) instead of mscorsvr.dll.
    > Does the app run on a single proc machine? As far as I know
    > ASP.NET should load mscorsrv.dll on MP systems.
    >
    >
    > "Cheung, Jeffrey Jing-Yen" wrote:
    >
    >
    >>Hi all,
    >>
    >>I need some help with debugging a "deadlock" in a production
    >>environment. I've managed to get a crash dump and have tried to
    >>analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
    >>in debugging to pinpoint the exact problem; however, I have narrowed it
    >>down to the garbage collection process. All threads seem to be waiting
    >>for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
    >>up 100% cpu.
    >>
    >>Can someone suggest what methods I could use to fully show and pinpoint
    >>where the GC is locking up? I have briefly skimmed the SOS docs and a
    >>lot of the exposed functions seem useful; however, I can't really make
    >>heads or tails out of them. I've examined the heap via !dumpheap -stat,
    >>and the largest object that comes back is a whopping 22 MB System.String
    >>object. In the MSDN docs, what is supposed to follow is a listing of
    >>Large Objects; however, I figure that maybe because I am using the 1.1
    >>framework, large objects aren't really differentiated any differently.
    >>I managed to get the address by issuing a dumpheap -mt <mt of the
    >>System.String> and I got a huge listing of addresses that really don't
    >>make much sense to me. I certainly don't want to issue an !objsize
    >><addr> on each of these (there are over 30k addresses).
    >>
    >>I think my biggest concern is that I want to be able to load the symbols
    >>of the .NET assemblies that are being used so that I can analyze the
    >>managed stack thoroughly, yet I'm not too certain how to do this. If
    >>that's not necessary to find out the core problem, then I guess I need
    >>to be directed towards the right place.
    >>
    >>Thanks in advance everyone,
    >>
    >>Jeff Cheung

    >
    >
    >
     
    Cheung, Jeffrey Jing-Yen, Jul 10, 2003
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. James S
    Replies:
    2
    Views:
    417
    James S
    Jul 20, 2004
  2. Øyvind Isaksen
    Replies:
    1
    Views:
    977
    Øyvind Isaksen
    May 18, 2007
  3. Stephan Steiner

    debbugging in iis - serving outdated pages

    Stephan Steiner, Jun 12, 2007, in forum: ASP .Net
    Replies:
    1
    Views:
    283
    Stephan Steiner
    Jun 12, 2007
  4. jacob navia

    Garbage collection problems

    jacob navia, Nov 18, 2007, in forum: C Programming
    Replies:
    89
    Views:
    1,677
    David Thompson
    Dec 2, 2007
  5. Carbon Man

    Memory problems (garbage collection)

    Carbon Man, Apr 23, 2009, in forum: Python
    Replies:
    6
    Views:
    371
    Paul Hemans
    Apr 29, 2009
Loading...

Share This Page