I am using the Debug Diagnostic Tool (DebugDiag v1.2) on a few production Windows Server 2008 R2 Standard servers running a custom web application built in-house on IIS 7.5 to create crash dump files.
The issue is that Windows Process Activation Service (WAS) warning events keep occurring on all of my production servers, and are increasing in frequency, but are still random in terms of when they occur, all throughout each 24 hour period.
Here is one sample System log event I'm concerned with:
Level: Warning Source: WAS Event ID: 5011 Description: A process serving application pool 'MyAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '7776'. The data field contains the error number.
I analyze the dump and get something like this each time (below). The problem is, how can I proceed to drill down into the issue after hitting the '0x3d9b9c16' lines, which repeat hundreds of times? What tools or approach could I take to debug further?
Thread 252 - System ID 6380 Entry point 0x00000000 Create time 3/25/2015 10:52:49 PM Time spent in user mode 0 Days 00:00:01.856 Time spent in kernel mode 0 Days 00:00:00.124
This thread is blocked by an unhandled exception
.NET Call Stack
Function System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
Full Call Stack
Function Source
ntdll!ZwTerminateProcess
KERNELBASE!TerminateProcess+2c
clr!EEPolicy::HandleFatalStackOverflow+1ba
clr!EEPolicy::HandleStackOverflow+1ac
clr!COMPlusFrameHandler+9b
ntdll!ExecuteHandler2+26
ntdll!ExecuteHandler+24
ntdll!RtlDispatchException+127
ntdll!KiUserExceptionDispatcher+f
clr!COMNlsHashProvider::HashiStringKnownLower80+10
clr!COMNlsInfo::InternalGetCaseInsHash+fa
mscorlib_ni+2e7a9b
mscorlib_ni+2e7a0a
mscorlib_ni+38aba8
mscorlib_ni+2e1a10
mscorlib_ni+2e19cf
mscorlib_ni+2e18ce
System_ni+1a41b4
System_Web_ni+1c6785
System_Web_ni+1f3581
System_Web_ni+1c67f8
0x1b2bcc27
0x3d9b9c0e
0x3d9b9c16
0x3d9b9c16
0x3d9b9c16
0x3d9b9c16
...
(this is repeated for hundreds of lines !)
...
0x3d9b9c16
0x3d9b9c16
0x3d9b9c16
0x3d9b9c16
System_Web_ni+217774
System_Web_ni+1ea04d
System_Web_ni+1ea087
System_Web_ni+1ea0e2
System_Web_ni+1ea0e2
System_Web_ni+1ea0e2
System_Web_ni+1fd845
System_Web_ni+1fd3a5
System_Web_ni+1fd2d9
System_Web_ni+1fd216
System_Web_ni+1fc16b
System_Web_ni+2021f6
System_Web_ni+1df356
System_Web_ni+1f1277
System_Web_ni+1df5f0
System_Web_ni+1dc0cb
System_Web_ni+1e1561
System_Web_ni+1e126f
webengine4!W3_MGD_HANDLER::ProcessNotification+5a
webengine4!W3_MGD_HANDLER::DoWork+2cb
webengine4!RequestDoWork+2ea
webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+18
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+128
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+305
iiscore!NOTIFICATION_CONTEXT::CallModules+28
iiscore!W3_CONTEXT::DoStateRequestExecuteHandler+36
iiscore!W3_CONTEXT::DoWork+d7
iiscore!W3_MAIN_CONTEXT::ContinueNotificationLoop+1f
iiscore!W3_MAIN_CONTEXT::ProcessIndicateCompletion+1f
iiscore!W3_CONTEXT::IndicateCompletion+75
webengine4!W3_MGD_HANDLER::IndicateCompletion+38
webengine4!MgdIndicateCompletion+22
System_Web_ni+240211
System_Web_ni+1e16e0
System_Web_ni+1e126f
clr!UM2MThunk_WrapperHelper+10
clr!UM2MThunk_Wrapper+76
clr!Thread::DoADCallBack+b4
clr!UM2MDoADCallBack+92
0x0098b361
webengine4!W3_MGD_HANDLER::ProcessNotification+5a
webengine4!ProcessNotificationCallback+32
clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+1ce
clr!ThreadpoolMgr::ExecuteWorkRequest+42
clr!ThreadpoolMgr::WorkerThreadStart+36c
clr!Thread::intermediateThreadProc+4d
kernel32!BaseThreadInitThunk+e
ntdll!__RtlUserThreadStart+70
ntdll!_RtlUserThreadStart+1b
Thanks!
