Webservice seems to timeout during requests to other web reference

[ajnt]

My web service seems to be timing out as certain stages.
The web service has 5 different web references to other services (on the
same server) which we need to communicate with.
As the number of concurrent calls to the web service increases, the
operation time of the service lengthens.
However, when there are multiple requests and the web service is taking a
bit longer to run, the service will often stop running. This is accompanied
by a 500 error in the IIS logs and a timeout error if I am calling it from my
VB.Net test harness (usually its called from another system).

I have changed the machine.config file so that the executiontimeout (in
httpruntime) is set to 40000 (should be plenty of time!) and checked that the
debug=false in the web.configs of all the services. I have also rebooted the
server.

It seems that the problem is when the web service calls one of the
references (also ASP.Net web services) and the called function takes longer
than it should do to run.

I was wondering if there was another timeout property that is assigned to
the web references by the main service that is calling them? If so, how do I
override this?
Either way, shouldn't the setting in machine.config override any other
timeouts?

I would appreciate any help / input.

Thanks,

Ajnt

 

[Pandurang Nayak]

Hi,

As per your own explanation, the web service calls 5 other web services. Let
us just take a look at what that means to your web server. Each web service
call is a HTTP request that has to be processed by IIS. This request is
spawning 5 other HTTP requests and they are all running synchronously which
means one waits for the other to finish and blocks the thread till that is
done.

Now, you are saying that your test harness in VB is simulating multiple
users. Even if you did about 100 users - that means that you are calling 5 x
100 = 500 HTTP calls.

The ASP.NET worker process has a finite number of threads that it can spawn
(specified in machine.config) and further requests will just have to wait for
earlier threads to finish. And since your CPU utilization is probably running
at 100% by then, your IIS starts recycling the process. All this causes a lot
of HTTP failed requests and also the timeouts.

There are a few things you can do:
1. I don't know about your application, but surely making a Web Service call
five other services suggests to me that you have a design with too many
services. Since you have the other components that the Web Service wraps on
the same local system - why don't you reference them directly? You can also
write some proxy code to find out if the components exist locally and
instantiate them directly if available - and if not, go make a web service
call.

2. If you are on ASP.NET 2.0, consider using Asynchronous Pages - a new
feature that spawns threads, but does not block the thread - you'll
definetely get an improvement - but depends on how much that will really help.

3. Use memory profilers (like the CLR profiler, Windows perf monitor) and
find out what piece of code is causing the delays / bottlenecks. In all
probability you will find that it is just because all your ASP.NET worker
threads are getting used up.

Hope that helps!

Regards
Pandurang

 

[Pandurang Nayak]

Check http://www.thejoyofcode.com/Tuning_the_ThreadPool.aspx for an excellent
article!

Regards,
Pandurang