8

On a Small Business Server 2011 installation a whole number of w3wp.exe processes appear to be using a disproportional lot of memory. The SBS out-of-the-box installations comes with a total of 7 sites and 20 ASP.NET application pools (Sharepoint, Exchange, WSUS and SBS-specific stuff like Remote Web Workplace).

The resulting dozen of w3wp.exe processes tends to consume more than 4 GB of the server's memory over time with the peak application pool being the one belonging to WSUS with around 800 MB in the working set. Manually recycling the application pools through the IIS MMC helps temporarily reduce the memory usage (the w3wp.exe processes shrink back to 10 MB, some of them regrowing quickly), but obviously is not something an admin wants to do all day. I was unable to find any recommendations on automatic recycling of the SBS-preinstalled application pools, so I am somewhat reluctant to "just do it" on production systems.

My research on the net on how to limit this only threw up a number of posts stating that w3wp memory consumption would not hurt but benefit performance as memory would be "freed when needed by other applications". The trouble is that it does not work out:

  • for one, an SBS is a multi-role server, one of the roles (the major one) being CIFS network storage which immensely benefits from filesystem caching which again relies on memory being "free" as in "not used by other processes in any way" - ASP.NET application pools which are hardly ever seeing users and eating memory are counterproductive
  • another thing is that I still have to see substantial decrease of the w3wp instances memory consumption upon memory shortage - what I see is a minor decrease by significantly less than 100 mb and excessive swapping instead - again hurting performance

I hardly ever administer IIS or ASP.NET apps, so any ideas on how to effectively trim the memory requirements for the application pools are welcome.

the-wabbit
  • 40,319
  • 13
  • 105
  • 169
  • What does this ASP.NET app do? I've seen w3wp.exe use lots of memory when an app fails to close connections with Linq-To-SQL or Entity Framework. – Nate Jan 05 '12 at 16:13
  • There are several. Many are related to Exchange like OWA/OMA and Sync services, one exceptionally large one is WSUS, some are used for Sharepoint or customized SBS-specific (CompanyWeb, Remote Web Workplace) sites – the-wabbit Jan 05 '12 at 16:29

4 Answers4

7

Welcome to the wonderful world of SBS. Recommended requirements for RAM = 10GB... and it REQUIRES a minimum of 8gb. (according to Microsoft.) for a good reason. It's not a fine-tuned well-oiled machine... it's very sloppy, bloated, and has everything under the sun bundled together. The more RAM you can throw at that box... the better. Unfortunately, you're limited to 32gb max. Which imho... is silly.

TheCompWiz
  • 7,349
  • 16
  • 23
  • I guess I should refine my answer a bit. If you're concerned by the amount of RAM it's consuming, You'd save yourself a lot of time/headaches by doing one of the following: A) Don't use SBS... use standard server and setup the roles you NEED individually. or B) Throw more RAM into the system... as RAM is pretty cheap. SBS is designed for *very* small offices... (10-20 workstations...) and does not scale very well. – TheCompWiz Jan 05 '12 at 14:32
  • Thanks for your answer. The system where I observed the behavior had 16 GB of RAM which is the physical limit. The available memory quickly filled up with Exchange's store.exe and numerous instances of SQL Server and w3wp.exe which is not just "not fine-tuned" but outright braindead. I know how to deal with excessive memory consumption issues of the other two as I happen to manage Exchange and SQL Server systems frequently, but with w3wp I am somewhat at a loss. The network itself has just 11 users. – the-wabbit Jan 05 '12 at 14:47
  • Exchange, SQL Server, and IIS all have mechanisms that will try & consume 100% of RAM for "cached" data. If something else requests more RAM... all 3 are *supposed* to scale-back to allow other services to run without resorting to swapping. (in theory) In practice, however I find that you just have to roll with the punches on some things... You can try & tweak everything manually & set hard-limits... but you'll be ever chasing your tail on that one. I'll see if I can find an article I read a few years ago on limiting application pools in IIS for you... – TheCompWiz Jan 05 '12 at 14:51
  • http://technet.microsoft.com/en-us/library/cc732519(WS.10).aspx – TheCompWiz Jan 05 '12 at 14:53
  • http://technet.microsoft.com/en-us/library/cc725749(WS.10).aspx – TheCompWiz Jan 05 '12 at 14:53
  • Thanks for the references. What I need to know if it is safe to recycle the app pools on a **very** frequent basis (like once every 15 minutes) in order to keep memory requirements low or if doing so is asking for trouble (this is what I suspect). – the-wabbit Jan 05 '12 at 16:17
  • BTW: "memory is cheap" is a tough argument. On behalf of our customers we are running several SBS installations virtualized (due to lower licensing costs) along with Terminal Servers. Surely there is a *big* difference if a machine with 184 GB of RAM (this is where RAM is not so cheap anymore) can host 5 or 20 of these installations. Fortunately, the app pools are not so much of an issue there as we can disable most of the .NET services and do not care (much) about the support status. – the-wabbit Jan 05 '12 at 16:23
  • @syneticon-dj: As each SBS has a maximum 32gb limit... We're not talking about high-end machines. The cost for RAM in the majority of new-ish (within 3 years) servers is ~$100-$200 bucks (worst-case). Cost of re-configuring each service to consume a fixed amount... plus time debugging all the potential problems that may arise from trying to limit everything... is far more. – TheCompWiz Jan 05 '12 at 16:29
  • @syneticon-dj: and if you're putting 20 SBS instances on the same hardware... your customers must hate you... or are too stupid to know they're getting screwed. – TheCompWiz Jan 05 '12 at 16:35
  • While there is no need to be rude, TheCompWiz is right, virtualising SBS seems crazy. You could provide each of the apps it includes as a managed service much cheaper and would be a far more optimal setup. – JamesRyan Jan 05 '12 at 17:03
  • @JamesRyan while you are technically right, there are licensing and regulatory constraints preventing us from doing that. Anyway, the virtualization scenario (where we do not have problems with w3wp processes due to the named reasons) was just an example for why memory conservation is a good thing. The systems where this issue hurts the most are physical ones in the customer's premises where I often cannot give SBS more than 16 GB without replacing the systems. TheCompWiz: as the SBS instances see minimal load, there are no issues consolidating them on powerful hardware. – the-wabbit Jan 05 '12 at 17:28
  • "SBS" and "minimal load" are antonyms... imho. But I see the point you're trying to make. The very point of SBS is to load up everything into one box... even if the majority isn't going to be used by a customer. – TheCompWiz Jan 05 '12 at 17:33
  • So there is no 'solution'? I'm in the same scenario as @syneticon-dj... Without as much knowledge and background! – HaydnWVN Nov 19 '12 at 12:10
7

This is what I ended up doing:

setting the server application cache for the .NET AppPools to a low value (5 MB) by setting the privateBytesLimit parameter in the web.config at %WINDIR%\Microsoft.NET\Framework\<version>\Config as suggested in this answer:

    <configuration>
      <system.web>
         <caching>
           <cache privateBytesLimit="5242880" privateBytesPollTime="00:01:00" />
         </caching>
      </system.web>
    </configuration>

This helped reduce the memory usage to somewhat more than 1 GB with the default pool recycling settings.

Apparently, using the "server" type of garbage collector (<gcServer = "true">) can lead to significant memory consumption as well, but as it seems, <gcServer> is set to false by default.

Community
  • 1
the-wabbit
  • 40,319
  • 13
  • 105
  • 169
  • FYI - a while back we started setting gcserver to false after reading these articles: http://forums.asp.net/t/1654596.aspx/1 and http://msdn.microsoft.com/en-us/library/ff647787.aspx – Greg Askew Sep 26 '12 at 11:41
  • @the-wabbit Is the advice to set gcServer = false still current now that server garbage collection happens concurrently? – Michael Steele Jun 14 '17 at 17:25
6

If you suspect that the resulting memory consumption is a problem due to a software defect, you can use the Microsoft DebugDiag 1.2 to create a full memory dump and analyze the dump for common issues. If you think there may be a memory issue, you need to enable leak tracking by selecting the "Monitor for Leaks" option and let it run for a while before creating/analyzing the dump.

DebugDiag 1.2 Download
https://www.microsoft.com/download/en/details.aspx?id=26798

enter image description here

enter image description here

enter image description here

Greg Askew
  • 34,339
  • 3
  • 52
  • 81
  • Thanks for the link, I am going to give it a try. The MSI installation package apparently has trouble when run on localized (non-US-Englisch) versions of Windows, I have to see if there is a workaround to this. – the-wabbit Jan 05 '12 at 17:50
2

You don't need a separate app pool for every app, just those that are unreliable or you wish to give priority to. Many can share (keeping different .net versions separate). You can then more realistically limit the memory an app pool will use. There should be no need to repeatedly recycle pools more than once a day.

Also, there is only so much memory that can be freed up in this manner. While some of it will be cache, each app needs a certain amount of working memory which is highly dependent on the specific web app. Trying to restrict this too much is going to bring things to a grinding halt.

The problem really is that SBS tries to do too much at once, you need to look at what you actually use and shut off what you don't.

But to be honest for just 11 users, where is the rest of the memory going? Exchange and SQL for light use certainly don't need more than 12Gb!

Jim G.
  • 338
  • 2
  • 6
  • 15
JamesRyan
  • 8,138
  • 2
  • 24
  • 36
  • Ah, you never experienced the beauty of an SBS, did you? Exchange's information store grows to 8 GB when not restricted, multiple SQL server instances will happily take up another 3 GB. Most of the app pools are running just 1-2 applications with the same .NET version and within the same security context. Yet, I cannot regroup them because of support issues. Also it is unlikely to solve the memory problems - if I just have 4 1-GB processes instead of 12 smaller ones, not much is won. – the-wabbit Jan 05 '12 at 16:15
  • The difference with having fewer app pools is that the various caches will be full of what is regularly used, pushing out what isn't. With lots of app pools there will be more wasted. How many security contexts do you actually need with 11 users? Yes, Exchange will merrily use as much of your memory that it can get it's hands on but it will also happily run on just 2GB for those few users if you restrict it. With SBS you need to accept that you can't use best practices everywhere or run any of the options optimally, you need to be a bit pragmatic and not over engineer things. – JamesRyan Jan 05 '12 at 16:52
  • Well if you are trying to run 20 instances of it on a server no wonder you are trying to cram it all into memory. You really should have mentioned that in the question since it puts things in an entirely different light. – JamesRyan Jan 05 '12 at 17:05
  • I don't actually - as I stated the problem arises with a physical system where rarely used app pools steal memory from the much more valuable filesystem cache. I suspected this would be due to some object caching within the application pools and was seeking some means to reduce / disable the cache. – the-wabbit Jan 05 '12 at 17:32