How does Windows deal with process memory or resource exhaustion?

Question

The other day I was working on a Windows 7 32-bit PC, with a few applications running including Firefox.

Over time I kept opening new tabs in Firefox until, suddenly, the Firefox window started flickering. From that point, most of Firefox's GUI turned black and refused to repaint. After trying to blindly close some tabs, the Firefox process crashed entirely. All other applications seemed to continue to work just fine.

I'm assuming (correct me if I'm wrong) that the Firefox process exhausted its 2GB virtual memory entitlement or some other resource. Which brings me to the key questions:

1. Can memory/resource exhaustion in one process de-stabilize other processes or even the kernel itself?

2. How does modern Windows deal with this scenario? How robust is it under these high-pressure conditions?

My concern is that, if other processes or kernel services are compromised, that could conceivably result in a broken system (e.g. file/disk/registry corruption).

Answer 1

How does Windows deal with process memory or resource exhaustion?

How a program behaves under "stress conditions" depends on how the particular program has been written (what privileges it has, does it contain bugs, etc) and this behaviour will determine how the rest of the system is affected when something "goes wrong".

It is possible that data can be lost. It is possible for the system to crash with a BSOD or just be unable to write data because of a delayed write failed error.

---

Pushing the Limits of Windows

The blog post series by Mark Russinovich (starting at Pushing the Limits of Windows: Physical Memory) goes into this topic in some detail.

In particular, in Pushing the Limits of Windows: Paged and Nonpaged Pool, he illustrates the effect of running a test program:

If you want to witness first hand how a system behaves when pool runs low, use the Notmyfault tool. It has options that cause it to leak either nonpaged or paged pool in the increment that you specify. You can change the leak size while it’s leaking if you want to change the rate of the leak and Notmyfault frees all the leaked memory when you exit it

On one test system, he said about the results:

On one test system, I eventually saw this error message indicating that data had potentially been lost. I hope you never see this dialog on a real system!

---

Further Reading

Windows Internals, 6th edition by Mark Russinovich contains a lot more information on how Windows works under the hood.