2

I have a specific ram-intensive program I run which slows down my entire system for hours after each run - does the Windows NT 6.0/6.1 kernel have any tunables to encourage processes' working sets to be paged back into RAM quicker?

The program in question reads a 1.5GB file into a 1.5GB malloc'ed buffer, does a little processing for a minute, and then quits.

While it is running, all other programs are paged out to disk as expected. When the program ends, the programs are not paged back again. Even if I wait hours, they do not get brought back. That means when I show up and try to use my computer it's painfully slow for ages while it slowly pages each process back in as I use it.

It appears when the process has exited, the "Cached" memory number shown by task manager is very large, and "Free" is very low. Every running process only uses 1 or 2 MB of "Working set", whereas they have normal amounts of "Commit size", indicating most of their pages are on disk.

To me this appears that Windows is using RAM for disk cache to keep my large 1.5GB file just incase I want to re-read it (which I don't), rather than fetching back my normal processes. Is there any way to force this balance towards processes and away from disk cache?

Oliver Mattos
  • 121
  • 1
  • Pretty remarkable to be able to allocate 1.5 GB of memory, you must be booting the 64-bit version of Windows or use the /3GB boot option. With what sounds like only 2 GB of RAM. And a possibly badly fragmented paging file. Important details I can only guess at. If it is anywhere accurate then you should spend your time tuning your program. 1.5 GB is excessive and scales *very* poorly. –  Sep 05 '11 at 21:16

2 Answers2

3

When the program ends, the programs are not paged back again. Even if I wait hours, they do not get brought back.

Windows does not page memory back in for your other processes back in until those pages are actually used by them. There are no parameters (that I know of) that will cause pages to be paged back in until those pages are actually referenced.

If it did do that, it would frequently waste time and RAM paging in stuff you don't end up using. Supposed you have five processes mostly paged out. Which one should it bring back? Take a guess? A little bit of all of them? Or wait until you start using one and then make as much RAM available for that one as possible?

To me this appears that Windows is using RAM for disk cache to keep my large 1.5GB file just incase I want to re-read it (which I don't), rather than fetching back my normal processes.

Windows, especially newer versions, does aggressively use excess RAM to cache "hot" files, but this never comes at the expense of programs that need RAM. These pages are quickly sacrificed to other processes that need memory. It's fast, because disk cache pages don't need to be paged out to the page file. They are just remapped into the address space of the process that needs it.

(And I sincerely doubt that a 1.5GB file that's opened once makes it to "hot" file status.)

That means when I show up and try to use my computer it's painfully slow for ages while it slowly pages each process back in as I use it.

It sounds to me like everything is working normally, but that you could stand to add more RAM or perhaps upgrade to a faster drive.

If you still think something is fishy, please post more specific information, like which version of Windows you're using, how much RAM the machine has, etc.

Adrian McCarthy
  • 131
  • 4
0

You can pass hints to CreateFile to suggest that Windows doesn't cache the file. This will discourage it from paging out the other processes in the first place.

Roger Lipscombe
  • 2,057
  • 5
  • 24
  • 37
  • Unfortunately, it's not my binary and I don't have source code to it, but since CreateFile has a pretty simple API, patching the binary isn't out of the question if a single bit needs flipping to activate that flag. –  Sep 05 '11 at 21:01
  • If it's not your binary, you might be able to do something by playing with job objects. I can't, given a couple of seconds searching, find anything obvious though. – Roger Lipscombe Sep 05 '11 at 21:03
  • CreateFile needs the `FILE_FLAG_NO_BUFFERING` flag, but that imposes requirements on how the file is read. It may not work for you. – Roger Lipscombe Sep 05 '11 at 21:05