High CPU Usage

Discussion in 'ASP .Net' started by Damien, May 15, 2006.

  1. Damien

    Damien Guest

    Hi guys,

    I'm looking for ideas for troubleshooting the following. We've tried
    some random things to try to treat the symptoms, but none seem robust
    enough to use when we go live, and we'd rather discover the root cause:

    We've got an ASP.NET application, running on framework 1.1 on Windows
    2003 (IIS 6). Under default settings, during testing by two users,
    we're seeing the CPU usage on w3wp.exe rocket up and stay up (and thus
    cause web pages to be served extremely slowly). Pages do continue to
    work, so fairly sure we've not got an infinite loop in there somewhere.

    The only two thing that seems to be new to this application compared to
    previous applications is that it uses remoting in a few places and
    makes use of datagrids on most pages. The remoting - in two places it
    places a call which can be reasonably slow (blocks internally until the
    operation is complete). In the other, a first remoting call is made to
    kick off a long-running process and then polling (via meta-refresh or
    AJAX, depending on whether JS is available) is used to determine when
    the operation is complete. The remoting channel is configured in
    Application_Start. This is just for information, and may be unrelated
    to our problem. The high CPU usage does not seem to be directly
    triggered by any of these calls, and the server end of the remoting is
    in a separate service (and it's CPU usage doesn't change)

    The datagrids use custom databinding, since the number and type of
    controls on each page is determined at runtime from the database. So,
    for instance, a grid can consist of three columns. Each column has a
    textbox control, a radio-button list, a drop-down list, etc. During the
    databinding, the unneeded controls are hidden, and the correct control
    has any necessary validation attached to it.

    I've not used IISState before, but tried using it when the CPU maxed
    out. Which resulted, after a while (and well before the dump was
    complete) in IIS deciding to recycle the process.

    Restarting IIS every thirty minutes is allowing testing to proceed, but
    obviously this isn't a long-term solution.

    Running under IIS 5 isolation mode resulted in no difference in
    behaviour.

    Letting IIS run more worker processes (we've tried values of 1, 3, 5
    and 1000) seems to work for some period of time but eventually one of
    the processes will again go up to high usage.

    So, ideas on how to diagnose this problem further? Like I say, it
    doesn't appear to be triggered by any particular page, and all requests
    still succeed. Thanks in advance for any help,

    Damien
     
    Damien, May 15, 2006
    #1
    1. Advertising

  2. Damien

    Damien Guest

    Damien wrote:
    > Hi guys,
    >
    > I'm looking for ideas for troubleshooting the following. We've tried
    > some random things to try to treat the symptoms, but none seem robust
    > enough to use when we go live, and we'd rather discover the root cause:
    >
    > We've got an ASP.NET application, running on framework 1.1 on Windows
    > 2003 (IIS 6). Under default settings, during testing by two users,
    > we're seeing the CPU usage on w3wp.exe rocket up and stay up (and thus
    > cause web pages to be served extremely slowly). Pages do continue to
    > work, so fairly sure we've not got an infinite loop in there somewhere.
    >
    > The only two thing that seems to be new to this application compared to
    > previous applications is that it uses remoting in a few places and
    > makes use of datagrids on most pages. The remoting - in two places it
    > places a call which can be reasonably slow (blocks internally until the
    > operation is complete). In the other, a first remoting call is made to
    > kick off a long-running process and then polling (via meta-refresh or
    > AJAX, depending on whether JS is available) is used to determine when
    > the operation is complete. The remoting channel is configured in
    > Application_Start. This is just for information, and may be unrelated
    > to our problem. The high CPU usage does not seem to be directly
    > triggered by any of these calls, and the server end of the remoting is
    > in a separate service (and it's CPU usage doesn't change)
    >
    > The datagrids use custom databinding, since the number and type of
    > controls on each page is determined at runtime from the database. So,
    > for instance, a grid can consist of three columns. Each column has a
    > textbox control, a radio-button list, a drop-down list, etc. During the
    > databinding, the unneeded controls are hidden, and the correct control
    > has any necessary validation attached to it.
    >
    > I've not used IISState before, but tried using it when the CPU maxed
    > out. Which resulted, after a while (and well before the dump was
    > complete) in IIS deciding to recycle the process.
    >
    > Restarting IIS every thirty minutes is allowing testing to proceed, but
    > obviously this isn't a long-term solution.
    >
    > Running under IIS 5 isolation mode resulted in no difference in
    > behaviour.
    >
    > Letting IIS run more worker processes (we've tried values of 1, 3, 5
    > and 1000) seems to work for some period of time but eventually one of
    > the processes will again go up to high usage.
    >
    > So, ideas on how to diagnose this problem further? Like I say, it
    > doesn't appear to be triggered by any particular page, and all requests
    > still succeed. Thanks in advance for any help,
    >
    > Damien


    Well, we had someone come in and provide some assistance to us today.
    We've now resolved our issues, mostly through the assistance of some
    tools we had already. Using a memory profiler led us to finding three
    memory leaks (essentially, places where objects hooked themselves up to
    static event handlers, and never unhooked themselves). But that wasn't
    the cause...

    We also found some older database access code which was using the
    "single connection, synclock it and pass it around" method of
    connecting to the database, which some of the code was spending a fair
    bit of time in (we thought we'd migrated all of that code, but the
    single shared connection was still available, and some code was still
    using it). We think this was the major cause...

    Once we'd switched to "grab a connection, do your work, close the
    connection", we started getting Connection Pool Timeouts. Turns out
    that two pieces of code we're doing "grab a connection, do your work,
    return", which was quickly swamping the connection pool. This may have
    been a contributor to the problems we were experiencing.

    We also found a few minor bugs, but I don't think any of them were
    contributing.

    Major thing I'm trying to fathom out is why we weren't seeing the
    Connection Pool Timeouts in the past.

    But we're now a lot happier that our code is "correct", and the speed
    issues have gone away. Definitely a lot happier using a profiler now
    (both for performance and for memory usage)

    Sorry if anyone spent any time trying to fathom anything out.

    Damien
     
    Damien, May 17, 2006
    #2
    1. Advertising

  3. Damien

    bmassicotte

    Joined:
    Jul 13, 2008
    Messages:
    1
    We used a tool to gather info

    We used a tool called IISPeek. You can find it on the web. It told us what pages were executing and for how long. It also told us what app pool and process id. We then reviewed the offending pages and corrected them. After running the new pages we saw that they did not cause 100% cpu spikes. It was a great help.

    Hope some of this helps.....:stupido:
     
    bmassicotte, Jul 13, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mk
    Replies:
    1
    Views:
    10,411
    Robert Olofsson
    Jul 25, 2003
  2. Arandil
    Replies:
    1
    Views:
    1,404
    Arandil
    Apr 12, 2004
  3. rajatag
    Replies:
    6
    Views:
    611
    rajatag
    Feb 1, 2006
  4. Jerry L. Gubka

    Web Page = High CPU Usage?

    Jerry L. Gubka, Jan 3, 2004, in forum: HTML
    Replies:
    2
    Views:
    814
  5. marchew
    Replies:
    1
    Views:
    589
    Lonnie Princehouse
    Jul 20, 2004
Loading...

Share This Page