ASP Thread Scheduling Anomalies

[Tony Proctor]

Hi everyone

We're experiencing some serious anomalies with the scheduling of ASP
threads. I'd be interested to hear if anyone knows what algorithm is used
(e.g. simple round-robin, or something more sophisticated), and what
situations might perturb it. Even a hint as to what would be considered
normal scheduling might help.

The root of our problem is that we observed a normally well-behaved web
application suddenly limit itself to a single ASP thread. The application
consists mainly of a VB6 DLL instantiated via an ASP page. Note that the
application does not store STA objects in Session/Application, and normally
exhibits no thread affinities at all. The previous post asking for advice
about this is at:
http://www.google.ie/groups?safe=off&[email protected]&lr=lang_en&hl=en.

As that previous post says, we eventually found that the switch of all
incoming requests (irrespective of their Session ID) to a single ASP thread
occurred when one thread just happened to destroy a Collection of MSXML DOM
objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with its
new free-threaded capability (which we don't use). We then created a much
cut-down version of our server application with a view to reporting it to
Microsoft. Unfortunately, the cut-down version didn't show the same results.
Interestingly, though, its ASP thread scheduling was markedly different from
that in our real application.

The problem has only been observed under IIS 6, and so it could well be an
IIS/ASP difference rather than the fault of MSXML. Running the same simple,
single-client (with an auto-refreshed HTML page at 1 sec intervals) test
against our 2 servers (the real one and the cut-down one), we observe the
following strange thread behaviour. Any explanations here would be very
welcome:

IIS 6 (configured with 20 ASP threads):
Real server:
All 20 threads used in perfect round-robin fashion, until one
destroys
a Collection of MSXML DOM objects. After that, all incoming
requests
are handed to this one thread (sometimes to a pair of threads if
they
happen to have cleared their Collections down at the same time)
Cut-down server:
One thread handles the initial request. All subsequent requests
oscillate
evenly between just 2 ASP threads

IIS 5 (configured with 20 ASP threads):
Real server:
All 20 threads are used. However, the first 8 are scheduled
sequentially.
After that it gets more and more random which ASP thread is
used. No
affinity every arises though.
Cut-down server:
Only 8 ASP threads are ever used, and these are scheduled in
perfect
round-robin fashion

I can't explain these observations as the 2 servers are handling the same
requests. There seems to be more involved than simple round-robin
scheduling. Hence, I though if I understood the algorithm better, I might
have a clue as to the initial problem we encountered (which is now being
considered serious)

Tony Proctor

 

[Brian Staff]

Tony,

This is from the IIS help files...maybe you have server-side debugging
turned ON?

Debugging Flags

Enable ASP Server-Side Script Debugging

Select this check box to enable the Web server to enter the Microsoft Script
Debugger while processing ASP pages. You can then use the debugger to
examine your scripts.

Enabling ASP server-side debugging causes ASP to run in single-threaded
mode. For performance reasons, ASP debugging is not recommended in a
production environment.

Brian

 

[Tony Proctor]

Thanks Brian. No, that's not the reason. The application is multi-threaded
and well-behaved until one thread decides to clear something down

Tony Proctor

 

[Pat [MSFT]]

ASP.dll does not directly schedule threads (i.e. they are not fibres). ASP
established a thread pool and passes requests to the thread pool. The OS
schedules threads.

The thread scheduler will try to reuse an idle thread that was recently run
over a thread that has not run in a while (i.e. FIFO), but this is not
guaranteed. This is an attempt to preserve the L1/L2 cache and improve
performance. You cannot guarantee which thread will service a given ASP
page. Also, it is important to differentiate if a single thread is
executing the entire ASP page or if a different ASP page is starting the
executing and then marshalling over to the other thread b/c of a COM
apartment issue. One other point is that the thread pool size will grow on
demand up to the configured maximum threshold. I think by default it only
creates 4 threads. So you have to create some degree of parallel stress to
even get it to create the 20 threads you are looking for.

I would recommend that you get an iisstate log of the problem occurring and
post it. We can take a look and see if we can identify the cause of the
serialization.


Pat

 

[Pat [MSFT]]

This is the only thread executing:

Thread ID: 21
System Thread ID: 928
Kernel Time: 0:0:2.78
User Time: 0:0:5.921
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\MSVBVM60.DLL -
Thread Type: ASP
Executing Page:

# ChildEBP RetAddr
00 0264dec0 02ca3367 msado15!CRsetField::SetupDataAccessor+0x29d
01 0264dee4 02ca2604 msado15!CStdCollection::LookupItem+0x14e
02 0264def0 02ca7eb2 msado15!CStdComObjectRoot::InternalAddRef+0x10
03 0264df20 02ca279c msado15!CRsetFields::get_Item+0x3e
04 0264df3c 110b7b6d msado15!ATL::CComObject<CCommand>::AddRef+0xc
05 0264df48 660d9a27 ACExec!ACServices::LoadObject+0x2a3d
WARNING: Stack unwind information not available. Following frames may be
wrong.
06 0264e34c 040ccd98 MSVBVM60!_vbaFreeObj
07 00000000 00000000 0x40ccd98

I can't say why or even if this is a real problem. The thread has only used
5 seconds of time. On the good news side...No XML to be found. You can try
isolating the ACExec.dll to a separate COM+ server package and see if the
problem follows or not. Also, if you build the dll with symbols and place
the symbol file in the same directory with the dll, IISState will pick it up
and give you line number and .cls file names.

Pat

 

[Tony Proctor]

My test program can now reproduce this quite easily Pat. I will polish it a
little, to ensure that only the minimal amount of code is present, and then
submit it to MS as a problem with MSXML 3.0.

It feels as though specifying an MSXML2.Document in a WithEvents variable in
a VB class actually does something to the DOM itself. In our code, the DOM
is manipulated via a wrapper class that happens to contain the WithEvents
clause. Later, they become separated so that the DOM is cached, and the
wrapper destroyed. However, it seems that whatever was done to the DOM
previously persists after that, and causes repercussions inside a
multi-threaded environment like IIS.

Thanks for the help. Much appreciated

Tony Proctor

This is the only thread executing:

Thread ID: 21
System Thread ID: 928
Kernel Time: 0:0:2.78
User Time: 0:0:5.921
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\MSVBVM60.DLL -
Thread Type: ASP
Executing Page:

# ChildEBP RetAddr
00 0264dec0 02ca3367 msado15!CRsetField::SetupDataAccessor+0x29d
01 0264dee4 02ca2604 msado15!CStdCollection::LookupItem+0x14e
02 0264def0 02ca7eb2 msado15!CStdComObjectRoot::InternalAddRef+0x10
03 0264df20 02ca279c msado15!CRsetFields::get_Item+0x3e
04 0264df3c 110b7b6d msado15!ATL::CComObject<CCommand>::AddRef+0xc
05 0264df48 660d9a27 ACExec!ACServices::LoadObject+0x2a3d
WARNING: Stack unwind information not available. Following frames may be
wrong.
06 0264e34c 040ccd98 MSVBVM60!_vbaFreeObj
07 00000000 00000000 0x40ccd98

I can't say why or even if this is a real problem. The thread has only used
5 seconds of time. On the good news side...No XML to be found. You can try
isolating the ACExec.dll to a separate COM+ server package and see if the
problem follows or not. Also, if you build the dll with symbols and place

 

[Tony Proctor]

Just for the record: this issue has been reported to MS as case
SRQ050304601390, and has been related to the manipulation of an MSXML DOM in
a VB "WithEvents" variable. MS have reproduced the problem and are currently
investigating

Tony Proctor

My test program can now reproduce this quite easily Pat. I will polish it a
little, to ensure that only the minimal amount of code is present, and then
submit it to MS as a problem with MSXML 3.0.

It feels as though specifying an MSXML2.Document in a WithEvents variable in
a VB class actually does something to the DOM itself. In our code, the DOM
is manipulated via a wrapper class that happens to contain the WithEvents
clause. Later, they become separated so that the DOM is cached, and the
wrapper destroyed. However, it seems that whatever was done to the DOM
previously persists after that, and causes repercussions inside a
multi-threaded environment like IIS.

Thanks for the help. Much appreciated

Tony Proctor

This is the only thread executing:

Thread ID: 21
System Thread ID: 928
Kernel Time: 0:0:2.78
User Time: 0:0:5.921
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\MSVBVM60.DLL -
Thread Type: ASP
Executing Page:

# ChildEBP RetAddr
00 0264dec0 02ca3367 msado15!CRsetField::SetupDataAccessor+0x29d
01 0264dee4 02ca2604 msado15!CStdCollection::LookupItem+0x14e
02 0264def0 02ca7eb2 msado15!CStdComObjectRoot::InternalAddRef+0x10
03 0264df20 02ca279c msado15!CRsetFields::get_Item+0x3e
04 0264df3c 110b7b6d msado15!ATL::CComObject<CCommand>::AddRef+0xc
05 0264df48 660d9a27 ACExec!ACServices::LoadObject+0x2a3d
WARNING: Stack unwind information not available. Following frames may be
wrong.
06 0264e34c 040ccd98 MSVBVM60!_vbaFreeObj
07 00000000 00000000 0x40ccd98

I can't say why or even if this is a real problem. The thread has only used
5 seconds of time. On the good news side...No XML to be found. You can try
isolating the ACExec.dll to a separate COM+ server package and see if the
problem follows or not. Also, if you build the dll with symbols and

place

the symbol file in the same directory with the dll, IISState will pick

it

up
and give you line number and .cls file names.

Pat

it,

The

OS

algorithm

is

http://www.google.ie/groups?safe=off&[email protected]&lr=lang_en&hl=en. created

a

better,

I