Windows 10, 'System' process taking massive amounts of RAM

Question

Since I upgraded to Windows 10, my system has been consuming RAM excessively

I've been reading a bit and determined it's likely a driver leaking memory. So I got myself the Windows Driver Kit and tracked memory usage with poolmon:

However, I don't really know how to proceed from here. Is the item tagged "smNp" the culprit in this issue? How do I go from there to actually identifying the driver?

I tried some stuff like "C:\Windows\System32\drivers>findstr /s smnp ." but it returned no results. I also took a look at the pooltag.txt file and this is the description I found for it:

So yeah, any help would be appreciated. Thanks in advance.

Answer 1

By going into services.msc (via Win+R) and disabling Superfetch completely solves this. I am not sure if Superfetch is just broken as of now or it's "by design".

In addition, apparently getting rid of the paging file will have the same effect but the above solution is a safer bet.

Answer 2

I've looked at xperf traces of several users and here the function ntoskrnl.exe!SmKmStoreHelperWorker of the Kernel starts to allocate memory.

^{(Click image to enlarge)}

I discovered this on sysinternals.

I've asked Microsoft about it and the answer is that this is by design. It is related to System Memory compression.

In the announcement of Windows 10 Build 10525, Microsoft explained it a bit:

In Windows 10, we have added a new concept in the Memory Manager called a compression store, which is an in-memory collection of compressed pages. This means that when Memory Manager feels memory pressure, it will compress unused pages instead of writing them to disk. This reduces the amount of memory used per process, allowing Windows 10 to maintain more applications in physical memory at a time. This also helps provide better responsiveness across Windows 10. The compression store lives in the System process’s working set. Since the system process holds the store in memory, its working set grows larger exactly when memory is being made available for other processes. This is visible in Task Manager and the reason the System process appears to be consuming more memory than previous releases.

So instead of writing memory data to the pagefile it compresses them. And this compressed memory is shown in the System process.

Microsoft also posted more details in the inside hub. Winbeta created a article which includes more details.

Apparently, the reason for this happened to do with Microsoft choosing to suspended UWP apps when they were not in the foreground, very similar to some smartphone OS management. Windows 8 users understood (perhaps not) that if apps weren’t on screen, they wouldn’t run until the user switched back to them. The ‘all or nothing’ approach is being updated with Windows 10 introducing a layer between the pagefile and normal paging activity. Now, when faced with memory pressure issues, MM will determine which pages should be moved to the modified list in a process called trimming. The modified list is a secondary list of pagefiles backing up a list of standby pagefiles. A backup list is captured in case memory is reclaimed from the standby list by another process, and the original process comes looking for its page. Instead of all or nothing, Windows 10 MM will compress unused pages rather than writing them to disk. With less writing, the result should be fewer disk operations – thanks to the compression – and now more data can be stored in memory.

According to the Windows team, “In practice, compressed memory takes up about40% of the uncompressed size, and as a result of a typical device running a typical workload, Windows 10 writes pages out to disk only 50% as often as previous versions of the OS.” If all goes according to plan, Windows users could be experiencing reduced waiting times for all devices as well as extended lifespans on systems that have flash-based hard drives.

Decompression is also something Windows 10 is designed to do well. Windows 10 is using the combination of parallelizability and sequential reads to produce pages into memory once called. The new decompression should result in a speedier experience as Windows 10 is simultaneously decompressing data and reading it in parallel using multiple CPUs. Older versions of Windows may have felt sluggish because of the transfer rates between the disk.

Microsoft also released a Video on channel9 which explains the feature.

Memory Compression in Windows 10 RTM
https://channel9.msdn.com/Blogs/Seth-Juarez/Memory-Compression-in-Windows-10-RTM

In this video Mehmet Iyigun spent some time discussing why the System process in Windows 10 is taking a bit more memory and why it's a good thing. A process taking more memory sounds like a bad thing - that is until I understood more about memory management, paging, and hard / soft page faults. Turns out that that the OS is doing some clever optimizations that allow your processes to trim some of the memory but not necessarily page it out to disk. Not only is the memory preserved in RAM, but it is also compressed - making hard page faults a more rare occurrence. The results should make for a snappier experience.

In the latest TH2 Builds, Microsoft updated the description in the task manager and now also shows that the SYSTEM process hosts the compressed memory:

to avoid confusions about the "high" usage.

In the Window 10 Anniversary Update which was released in August 2016, Microsoft extracted the Compression into now shown in a pseudo process called Memory Compression to no longer confuse users why SYSTEM has such a large memory usage:

But it looks like Taskmgr doesn't show this process, only ProcessExplorer/ProcessHacker are able to show it. The Taskmgr only shows the amount of compressed memory in the overview:

If you hover over the used memory graph in Taskmgr you see a tooltip that shows the amout of data that are compressed.

In this demo 388MB are compressed to 122MB so 267MB are saved with the compression.

Answer 3

I found an outlier case that causes high System memory usage, and wanted to include it in case this information benefits anyone.

If you heavily utilize Microsoft's Volume Snapshots (the software snapshot, not the hardware snapshot), the more snapshots you keep combined with large data changes, then System will consume more RAM.

Normally the amount of RAM used for Volume Snapshots is small and will not be noticed, unless you have a giant volume (i.e. 64 TB) with multi-terabyte deltas between snapshots. By default, snapshots will simply delete themselves if write IO's get too high, but there are ways to prevent that, allowing you to reach massive deltas.

Below is an extreme case showing a server's System process using 13GB of RAM. This server has only two Volume Snapshots, taken 15 days apart, with around 10 TB of data written in between each snapshot.

The System process above was previously at 24GB of usage, and the following three behaviors were observed:

1. After a reboot and logging back in, the system would hang for a period of time at a blank screen until the desktop would appear.
2. During this hang, pulling up Task Manager (CTRL-SHIFT-ESC) showed the System memory usage growing.
3. During this hang, the disk with the Volume Snapshots performed a lot of reads that did not show up in Performance Monitor. Though, because the disk utilized iSCSI, the network card showed a steady read stream around 200 Mbps.

I suspected Volume Snapshots, so I tried deleting the oldest snapshot which instantly dropped System's memory usage from 24 GB to 13 GB.

Under these circumstances this may be normal behavior, though I have not confirmed this with Microsoft. In the meantime I will be adding an additional 32 GB of RAM to this server to handle the Snapshot overhead.

(Note: this is a high volume backup server running Windows 2016 with a 64 TB SSD iSCSI drive attached. It maintains an average of three volume snapshots at any given time, with a new one created every 15 days. There is around 10 TB of data written in between each snapshot).

Answer 4

Disable prefetcher in regedit key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters you probably have Enable Prefetcher at a value of 2 or 3 so change it to 0

Next you need to disable Superfetch in services

1. Search for services.msc

2. Find superfetch click properties then set it to disabled and stop the service as well.

I do these steps and when I am gaming and normally using PC and the system process uses only 28k