15

I have five apps in an Azure App Service Plan, all different copies of the same app for different clients. It's an ASP.NET MVC app with an SQL database.

This morning I woke up to slow and non-responsive sites, sometimes resulting in a 503 error. Upon checking the CPU/Memory metrics for the App Service Plan I discovered the CPU pegged at 100%:

CPU at 100%

And the graph of the individual sites' CPU time shows them all going off at the same time, though some worse than others:

Individual Sites CPU usage

I tried to resolve the issue by doing an Advanced Application Restart, but the blade wouldn't load - I assume because it was trying to get information from the server that couldn't respond.

In the end I tried stopping each of the sites one by one, which brought the CPU back to zero, then starting them back up again, which seemed to have resolved the issue, but now a few of them are starting to climb back up.

I experienced a similar issue a couple of weeks ago and assumed I was growing too big for my App Plan, so upgraded from S1 to S2 and it'd been fine until today.

I'm at a total loss as to how to figure out what the cause of this CPU hungry behaviour is. Can anyone point me in the right direction of how I can start to diagnose this?

littlecharva
  • 301
  • 2
  • 3
  • 14
  • this seems to happen on Azure app service plans a bit, have you contacted MS? - unless you have made any changes, I would let them know there is a problem with the platform. – Sum1sAdmin May 04 '16 at 09:15
  • I don't have a support plan with them, so don't think there's any way to let them know. That's a bit of a worry if I'm just going to randomly experience 100% CPU issues - might have to look for another supplier. Would having two instances help do you think? – littlecharva May 04 '16 at 09:33
  • I would use the open forums like technet - If it's critical you could scale it up but I wouldn't pay more for something that they need to fix, if that is the case. – Sum1sAdmin May 04 '16 at 09:35

3 Answers3

6

We've had this issue a few times and each time it's turned out to be run away GC (garbage collection). It's hard to prove and diagnose, but what I end up doing is using the kudo (scm) site, clicking tools => support (which takes you to the support site for apps.

From here you pick your directory (incase you have several) and site, click on Analyze => Metrics then the Diagnose button (NOTE this has already changed so these steps can change at any time), then back on Analyze => Dianotics, you'll eventually get a Memory Dump => "Analysis status" report. This should be a mht file (which you can open in the hateful browser IE or Edge), then search for the key "gc".

you'll find some interesting call stack frames with references to things like "GCFrame" or more interestingly calls to "System.Threading.WaitHandle.WaitMultiple" if you get too many of these, you're system might be having issues with garbage collection.

How to solve that... that's a topic covered in many other threads because it's like asking "how do I live in a world with IE 6 still in use?"...

cdmdotnet
  • 161
  • 1
  • 3
  • 2
    This has moved. It's now located in the App Service (not scm site, regular Azure site) at "Diagnose and Solve Problems->Diagnostic Tools->Collect Memory Dump" – Josh Noe Dec 06 '18 at 17:15
4

Your best bet would be installing New Relic or Application Insights for this particular app. It can be easily installed through the App Service -> Tools -> Performance Monitoring. This will give you a detailed view of what is happening in both server and client side.

Application Monitoring

Article: Monitor Azure web app performance

magicandre1981
  • 1,110
  • 2
  • 10
  • 20
Bruno Faria
  • 3,804
  • 1
  • 11
  • 18
1

Watch these videos to learn more about troubleshooting High CPU Issues in Azure App Service:

Puneet Gupta
  • 111
  • 1