Nullreference exception in Asynchronous webservice call (.Net 2.0)

[vecozo]

[this has previously been posted buy raymondr at Nullreference exception in
Asynchronous webservice call (.Net 2.0)]


Hi,

First a brief description of out application:
We have a webapplication which calls a couple of webservices during one
request (postback). These calls to the webservices are made concurrent using
asynchronous webservices calls. The number of webservices called concurrent
is between 1 and 18.
The webservice calls are made using SSL with a X509 clientcertificate.
The application is underhigh load

The problem:
After migration to the .Net Framework 2.0 we once in a while get a
nullreferenceexception during the asynchronous call. The exception seems to
occur somewhere in the framework. It only occurs on our production
environment.

Maybe someone got a clue? See the stacktrace below

The stacktrace is :
NullreferenceException
at
System.Web.Services.Protocols.WebClientProtocol.ReadAsyncResponse(WebClientAsyncResult client)
at
System.Web.Services.Protocols.WebClientProtocol.GetResponseAsyncCallback(IAsyncResult asyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.runTryCode(Object userData)
at
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken)
at System.Net.HttpWebRequest.SetResponse(Exception E)
at System.Net.HttpWebRequest.CheckWriteSideResponseProcessing()
at System.Net.ConnectStream.ProcessWriteCallDone(ConnectionReturnResult
returnResult)
at System.Net.HttpWebRequest.WriteCallDone(ConnectStream stream,
ConnectionReturnResult returnResult)
at System.Net.ConnectStream.CallDone(ConnectionReturnResult returnResult)
at System.Net.ConnectStream.ResubmitWrite(ConnectStream oldStream,
Boolean suppressWrite)
at System.Net.HttpWebRequest.EndWriteHeaders_Part2()
at System.Net.HttpWebRequest.EndWriteHeaders(Boolean async)
at System.Net.HttpWebRequest.WriteHeadersCallback(WebExceptionStatus
errorStatus, ConnectStream stream, Boolean async)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken)
at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32
offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.WriteCallback(IAsyncResult
transportResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.runTryCode(Object userData)
at
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken)
at
System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Best Regards,
Raymond

www.VECOZO.nl

 

[Luke Zhang [MSFT]]

Hello Raymond,

I never saw similiar issue reported before. Did the exception always occur
with one web service\web method, or totally random? Can it be repeated
after you find one in the system?

Regards,

Luke Zhang
Microsoft Online Community Lead

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

(This posting is provided "AS IS", with no warranties, and confers no
rights.)

 

[Simon Hart]

You could try and register the AppDomain.UnhandledException and capture the
NullReference exception to see if it gives you any more clues.

Also ensure when you call the End method of the web service proxy class, the
IAsyncResult object is not null.

Regards
Simon.

[this has previously been posted buy raymondr at Nullreference exception
in
Asynchronous webservice call (.Net 2.0)]


Hi,

First a brief description of out application:
We have a webapplication which calls a couple of webservices during one
request (postback). These calls to the webservices are made concurrent
using
asynchronous webservices calls. The number of webservices called
concurrent
is between 1 and 18.
The webservice calls are made using SSL with a X509 clientcertificate.
The application is underhigh load

The problem:
After migration to the .Net Framework 2.0 we once in a while get a
nullreferenceexception during the asynchronous call. The exception seems
to
occur somewhere in the framework. It only occurs on our production
environment.

Maybe someone got a clue? See the stacktrace below

The stacktrace is :
NullreferenceException
at
System.Web.Services.Protocols.WebClientProtocol.ReadAsyncResponse(WebClientAsyncResult
client)
at
System.Web.Services.Protocols.WebClientProtocol.GetResponseAsyncCallback(IAsyncResult
asyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.runTryCode(Object userData)
at
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken)
at System.Net.HttpWebRequest.SetResponse(Exception E)
at System.Net.HttpWebRequest.CheckWriteSideResponseProcessing()
at System.Net.ConnectStream.ProcessWriteCallDone(ConnectionReturnResult
returnResult)
at System.Net.HttpWebRequest.WriteCallDone(ConnectStream stream,
ConnectionReturnResult returnResult)
at System.Net.ConnectStream.CallDone(ConnectionReturnResult
returnResult)
at System.Net.ConnectStream.ResubmitWrite(ConnectStream oldStream,
Boolean suppressWrite)
at System.Net.HttpWebRequest.EndWriteHeaders_Part2()
at System.Net.HttpWebRequest.EndWriteHeaders(Boolean async)
at System.Net.HttpWebRequest.WriteHeadersCallback(WebExceptionStatus
errorStatus, ConnectStream stream, Boolean async)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken)
at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32
offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.WriteCallback(IAsyncResult
transportResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.runTryCode(Object userData)
at
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken)
at
System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Best Regards,
Raymond

www.VECOZO.nl

 

[Raymondr]

Hello Luke,

We concurrently call different webservices endpoints which have implemented
the same wsdl. So it's always the same method but not the same webservice.
If a webservice doesn't respond in the time given, the asynchronous call is
aborted and the instance of the proxy class is disposed.

We have multiple servers in our cluster. It happens on all multiple servers.
The total number of requests made by our servers to the other webservers is
between 10.000 and 150.000 per hour.

Best Regards,

Raymond

 

[Luke Zhang [MSFT]]

Hell Raymond,

You may implement some exception handling code (like
AppDomain.UnhandledException as Simon suggested), record the server and web
method it happened with. And check if the server is connectable at that
time, or recall the web method, will it always failed if one exception
occurred?

Regards,

Luke Zhang
Microsoft Online Community Lead

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

(This posting is provided "AS IS", with no warranties, and confers no
rights.)

 

[Raymond Roelands]

Hello Luke,

The stacktrace in the first message is the Exception caught by the
AppDomain.UnhandledException event. The IAsyncResult object is not null, the
exception is not thrown in the End method of the webservice call.
It occurs somewhere in the framework when handling the asynchronous call. In
the exception is also not possible to detect which webservice is causing the
error.

Maybe this also did occure in the 1.1 framework but we didn't notice it
because we didn't handle the uncaught exceptions. We changed the
aspnet.config with legacyUnhandledExceptionPolicy enabled="true" so our
worker process isn't recycled when this occurs (it happens approx. each 5
minutes causing time out to our clients)

I think there's a bug in handling the asynchronous result. If something went
wrong with the webservice called I would expect this exception would be
caught and thrown when the End method of the webservice is called. The call
to the End Method is surrounded by a try/catch.
The way it's working now the exception is thrown in a separate thread
managed by the framework causing my application pool to crash.

Best regards,

Raymond Roelands