asp.net 1.1 application pool locking

[Allan Ebdrup]

I have a webservice written in dotNet1.1 that does some stuff and calls a
webservice written in dotNet 2.0.

I'm using a Visual Studio 2005 webtest and loadtest to test webservice.

When I load test the 1.1 webservice with for example 25 simultaneous users
with no wait time between requests, something wierd happens.
After between 3 and 10 minutes of the test the 1.1 webservice stops
responding, when I stop the loadtest and look at the Process that is the
application pool of the 1.1 webservice, it is constantly using 20-30 % CPU
and continues to do so, even when I stop the loadtest.
The only way to get the 1.1 webservice to respond again is to kill it's
process.

What's even stranger is that if I comment out the calls to the dotNet 2.0
webservice the loadtest runs fine for 30 minutes.

And the really strange part is that if I loadtest the dotNet 2.0 webservice
it responds fine for the 30 minute loadtest.
(Well actually it responds fine after 1-2 minutes, If I run 25 simultaneous
users it gives 25 responses and then gives no responses for 1-2 minutes and
then gives responses evenly for the rest of the 30 minute loadtest. If i run
the webtest, that the loadtest runs, in the 1-2 minute gap where no reponses
are recieved in the loadtest. the webtest runs fine and gives responses in
1-2 seconds (is this a bug in Visual Studion 2005?)

What's going on, and how can I see what's causing the 1.1 webservice to run
at 20-30 % CPU and stop responding?

Any help will be greatly appreciated.

Kind Regards,
Allan Ebdrup

 

[Mahesh Prasad]

I have a webservice written in dotNet1.1 that does some stuff and calls a
webservice written in dotNet 2.0.

I'm using a Visual Studio 2005 webtest and loadtest to test webservice.

When I load test the 1.1 webservice with for example 25 simultaneous users
with no wait time between requests, something wierd happens.
After between 3 and 10 minutes of the test the 1.1 webservice stops
responding, when I stop the loadtest and look at the Process that is the
application pool of the 1.1 webservice, it is constantly using 20-30 % CPU
and continues to do so, even when I stop the loadtest.
The only way to get the 1.1 webservice to respond again is to kill it's
process.

What's even stranger is that if I comment out the calls to the dotNet 2.0
webservice the loadtest runs fine for 30 minutes.

And the really strange part is that if I loadtest the dotNet 2.0 webservice
it responds fine for the 30 minute loadtest.
(Well actually it responds fine after 1-2 minutes, If I run 25 simultaneous
users it gives 25 responses and then gives no responses for 1-2 minutes and
then gives responses evenly for the rest of the 30 minute loadtest. If i run
the webtest, that the loadtest runs, in the 1-2 minute gap where no reponses
are recieved in the loadtest. the webtest runs fine and gives responses in
1-2 seconds (is this a bug in Visual Studion 2005?)

What's going on, and how can I see what's causing the 1.1 webservice to run
at 20-30 % CPU and stop responding?

Any help will be greatly appreciated.

Kind Regards,
Allan Ebdrup

Hi,

As your WS 1.1 is making outgoing calls to WS 2.0 make sure the worker
thread settings in machine.config is set using the following formula:
--------------------------------------------------------------------------------------------------------------------------
* maxconnection = 24 (threads used to make remote web service
calls)
* minFreeThreads = 176 (threads kept reserved for serving non-
request calls)
* minLocalRequestFreeThreads = 152
* maxWorkerThreads = 100 (Maximum number of worker threads per CPU
in the thread pool)
* maxIoThreads = 100 (Maximum number of IO threads per CPU in the
thread pool)


No of threads for processing incoming requests : (#CPU *
maxWorkerThreads) - (#CPU * minFreeThreads)
--------------------------------------------------------------------------------------------------------------------------

There's a good article on MSDN which is a must read
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scalenetchapt17.asp

The most important setting in your case is the "maxconnection"; by
default this value is set to "2". This means only two threads will be
used to make outgoing calls from the ASP.NET app even if you are
loading 25 users. Change the machine.config on both WS 1.1 and WS 2.0
using the above formula and retest your app.

Also to see exactly what is happening on both these servers you need
to use "PerfMon". Add the following counters and watch them while you
do the perf testing. These counters will tell you exactly why the two
systems are behaving this way. The article listed above describes more
counters and you can add them as needed.
--------------------------------------------------------------
ASP .NET v1.1.4322\Request Current
ASP .NET v1.1.4322\Request Queued
ASP .NET v1.1.4322\Request Rejected

..NET CLR Memory\% Time in GC
--------------------------------------------------------------

As I don't have specifics on what your web services are doing I can
only assume based on the symptoms you described...

Regarding Symptom 1 where the WS 1.1 CPU utilization continues to be
at 20%-30% even after the test is stopped; this could be your code
specific. Check if the CPU utilization is being contributed by GC
(using the above perf counters). I had encountered a similar situation
where the web server after receiving the XML from a web service was
applying an XSLT and this XSLT had a bug which caused infinite loop,
causing the CPU usage to go high. Because of the infinite loop many
objects were getting created and GC was collecting them. So check your
code to see what could be contributing to the high CPU.

Regarding Symptom 2 where WS 2.0 doesn't respond for 1-2 minutes and
then starts responding fine; this could be because of the requests
getting Qed up. The perf counters will again help you in identifying
the issue.

Hope this points you in the right direction.

-Mahesh

 

[Steven Cheng[MSFT]]

Thanks for Mahesh's informative input.

Hi Allan,

I agree with Mahesh on troubleshooting the threading configuration as it is
the most likely problem here after read the problem description. For
ASP.NET web application, each worker process has a fixed number of worker
threads & IO threads (in a given managed thread pool). And there is certain
configuration setting which will affect the threading behavior. For
example, the maxFreeXXThreads determine how many concurent threads may
occur in your applicaiton worker process(ASP.NET requests are all processed
by thread pool threads). And since your ASP.NET web application will send
request(call webservice) to another local webservice, the "free Local
threads" setting is also very important. When any of these resource get
exhausted, the application will be blocked.

here are some ASP.NET threading related resource, you may have a look:

#Support WebCast: Microsoft ASP.NET Threading
http://support.microsoft.com/kb/820913


#How To: Monitor the ASP.NET Thread Pool Using Custom Counters
http://msdn2.microsoft.com/en-us/library/ms979194.aspx

#Threading Explained
http://msdn2.microsoft.com/en-us/library/ms998549.aspx#scalenetchapt06_topic
8

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead



==================================================

Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.



Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.

==================================================



This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Allan Ebdrup]

Thanks for the answers.
I found that it wasn't calling the other webservice that was causing dotNet
1.1 to stop responding.
It was this code:
-------
System.Collections.ArrayList sortedFilterIds = new
System.Collections.ArrayList(filterIds);
sortedFilterIds.Sort();
using(System.Data.IDataReader dr = DSI.PositionPosting.GetPostingIds())
{
int intIdOrdinal = dr.GetOrdinal("Id");
int intCount=0;
while(dr.Read())
{
if(!dr.IsDBNull(intIdOrdinal))
{
if(sortedFilterIds.BinarySearch(dr.GetInt32(intIdOrdinal))>=0)
{
intCount++;
}
}
}
return intCount;
}
------------
where filterIds is an int[] with up to somewhere between 0 and 1000 entries,
and DSI.PositionPosting.GetPostingIds returns a DataReader with ID's (up to
5000).
The code simply counts the size of the intersection of the filterId's and
the Id's returned by DSI.PositionPosting.GetPostingIds.

I can't see how this code could cause the dotNet 1.1 application pool to
lock up and use 20-30 % cpu.

The reason for not simply using a WHERE ID IN (...) with the filterIds in
the SQL was because SQL 2000 has VERY poor performance when the WHERE IN
(...) clause has very long lists of Id's, and there is even a very low limit
to how long the SQL Query can be.

In the meantime we have moved to SQL 2005 that doesn't have the problems
with long IN clauses in the SQL, that SQL 2000 had. So I changed the code to
use COUNT(*) WHERE ID IN (...), and everything runs fine.

Seems to be some bug in dotNet 1.1, perhaps when using BinarySearch?

Kind Regards,
Allan Ebdrup

 

[Steven Cheng[MSFT]]

Hi Allan,

Thanks for your reply.

So you've narrow the blocking issue to the ArrayList's BinarySearch
section? How did you find that it is that code fragment cause the page
request be blocked?

for BinarySearch, I haven't find any existing issue on it, not suring
whether it also related to the number collection used here. I think you can
move the code out in a normal console or a separate page (test the
intersection size counting code) to see whether it will show the same
problem.

BTW, since you're sorting and searching against integer list, why not use
the generic List<T> class?

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Allan Ebdrup]

Hi Allan,

Thanks for your reply.

So you've narrow the blocking issue to the ArrayList's BinarySearch
section? How did you find that it is that code fragment cause the page
request be blocked?

No I've narrowed the problem down to the code I pasted.
When I commented out the code the problem dissapeared. When I put it back in
the problem reappeared.
I'm just guessing that it might be BinarySearch that is the problem..

for BinarySearch, I haven't find any existing issue on it, not suring
whether it also related to the number collection used here. I think you
can
move the code out in a normal console or a separate page (test the
intersection size counting code) to see whether it will show the same
problem.

I've solved the problem by calculating the intersection in th SQL query, I
will not be investigating further.

BTW, since you're sorting and searching against integer list, why not use
the generic List<T> class?

As I write it is a dotNet 1.1 application. But you are right if it was a
dotNet 2.0 application a generic list would be the rigth choice.

Kind Regards,
Allan Ebdrup

 

[Steven Cheng[MSFT]]

Thanks for your followup Allan,

Glad to hear that you've found a means to workaround the issue. If you meet
any further problem on this and need help, please feel free to post here.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


This posting is provided "AS IS" with no warranties, and confers no rights.