How do you keep a specific process out of virtual memory (aka paging)?

1

1

In windows, my guest VM keeps getting put into virtual memory... and then the VM freezes... because the VM is not able to respond at all. I even wait like 5 minutes and no response from the VM.

How do you keep a specific process, in this case the process associated with the VM, out of virtual memory?

My VM typically freezes every hour or so. A ballpark estimate is that I am losing 10 minutes out of every hour... due to restarting the VM. So any help would be very much appreciated.

UPDATE as per request:

  • host: winxp-sp3-32-bit
  • host-physical-memory: 3000 MB
  • guest: fedora14-64-bit
  • guest-allocated- 1400 MB
  • host memory as seen from task-manager when guest freezes: ~100 MB typically

Trevor Boyd Smith

Posted 2011-05-24T18:20:07.897

Reputation: 2 093

Can you prove that the VM is being put into virtual memory, and also that this is the problem? Please tell us what your guest OS is, how much RAM is allocated to the guest and how much is free in the guest when in freezes, and how much total RAM and free RAM in the host? – Spectre – 2011-05-24T18:26:33.420

Answers

1

I'm fairly certain in Windows you cannot keep a process out of virtual memory. IIRC all memory apart from certain areas of the Windows kernel can be paged. I don't think this is your problem though, especially not if the rest of Windows is responding OK.

Spectre

Posted 2011-05-24T18:20:07.897

Reputation: 1 045

So you if you have a paging file, you can't control where the process resides whether in virtual/physical memory? – Trevor Boyd Smith – 2011-05-24T20:04:29.823

Correct, Windows will do its best to decide for you - its not an all or nothing situation either, frequently accessed parts of an apps memory will stay in physical, less used will get paged. If the VM really is getting aggressively paged, its a sign that you have allocated it too much memory, or that your system as a whole doesn't have enough. Given 32bit WinXP with 3GB of RAM, I wouldn't allocate the VM more than 1GB. – Spectre – 2011-05-24T20:24:14.627

Oh, and a 64bit guest on a 32bit host is (IMO) asking for trouble, as the VM app probably has to emulate all the 64 bit instructions to 32 bit and back, which might be quite a bottleneck. Try making a 32bit Fedora VM and see if you have the same problem. Also, if you have a processor that has virtualisation features (Intel VT and AMD-V iirc), make sure they are turned on in the BIOS. – Spectre – 2011-05-24T20:30:58.573

FYI, once you enable virtualization on your mobo, doing a 64bit guest on a 32bit host is not a problem. Both VirtualBox and VMWare implement this feature robustly. (you can navigate to their respective websites and see information about this specific feature). – Trevor Boyd Smith – 2011-05-24T20:36:21.923

I am leaning towards... "I think I allocated too much to the VM". I recently racheted down the VM from 1400 to 1200 MB and I haven't had the guest freeze in more than an hour. Given that I used to freeze every 30 min and it's been more than an hour, I think the problem was allocating too much memory. – Trevor Boyd Smith – 2011-05-24T20:38:05.353

I'm guessing then that 64bit guest on 32bit host would still require a 64bit capable CPU? And if you have one of them, why keep running 32bit XP :P – Spectre – 2011-05-24T20:55:24.433

sigh yes... to run 64bit guest on 32bit host does require 64bit CPU... double sigh... very very sad panda =( ... (not to mention i have 4 GB ram... win-32bit only lets me see 3gb) – Trevor Boyd Smith – 2011-05-24T20:58:57.137

Have linux host an XP guest instead?? – Spectre – 2011-05-24T21:08:19.070

-1

Set your page file size to 0 :) You'd be amazed how much better your system works.

In XP,
My Computer->Properties->Advanced->Performance:Settings->Advanced->Virtual Memory:Change->"No paging file"->Set->OK

This obviously affects all processes, as far as I know it isn't possible to prevent the swapping out of memory for any one particular process. As mentioned in the comments though, I'm not really convinced (based on your description) that this is what is happening to your VM.

Codebling

Posted 2011-05-24T18:20:07.897

Reputation: 631

Even if you disable the pagefile, portions of virtual address space that are mapped to specific files - and this includes all your code files, exes and dlls and so on - will still be paged, i.e. only brought in from disk when necessary, and dropped from RAM if not accessed for a while and RAM is getting scarce. You can't get rid of paging. Keeping the pagefile allows the OS to make better choices as to what to keep in RAM. Without a pagefile, ALL private committed virtual address space has to be kept in RAM, no matter how seldom or long ago it's been accessed. That is bad. – Jamie Hanrahan – 2016-03-24T10:52:25.513

@JamieHanrahan virtual address space is never mapped to files. You load files into memory, or you open handles to them and these handles are kept in memory. When you execute an EXE, it is loaded into memory, and so are its statically linked DLLs. The virutal addres space maps into paged physical memory, and these pages can be swapped to disk. The whole idea behind disabling swapping is that Windows is bad at deciding what to keep in RAM and what not to. You can absolutely disable swap. This can be good. – Codebling – 2016-03-25T10:53:57.983

@CodeBling I'm sorry but you're very mistaken. In fact, a great deal of your VAS is mapped to files. Look at the MapViewOfFile API in Windows, or mmap in *nix. These are used by applications to access large data files, but file mapping is also how code files work in both OSs. Creating a process to run an EXE with its DLLs absolutely does NOT read the entire EXE and DLLs into RAM. They are simply mapped into the process VAS, which means they become the backing store - i.e. "where to page this in from" - for the address ranges assigned to them. See Windows Internals by Solomon and Russinovich. – Jamie Hanrahan – 2016-03-25T12:07:44.340

@JamieHanrahan nice. Taught me something today. I knew you could map hardware to addresses but not files. But..you're saying that when we launch an executable, the kernel just maps the file to memory and immediately just page faults in order to read the PE header, set up the IAT and jumps into main without loading anything else? Hmm..Could be. Ultimately it's not germane the answer: "Can you keep a specific process from being swapped out?" "No, but here is how you disable swap". – Codebling – 2016-03-25T23:07:00.407

@JamieHanrahan I also don't think that you can say that keeping all virtual address space in RAM is arbitrarily A Bad Thing. I think it depends on your system resources and how you use your system. Happy to discuss in chat. Personally I've happily been with no swap and SuperFetch disabled for at least 5 years, and distinctly found it more responsive than it had otherwise. – Codebling – 2016-03-25T23:10:23.817

@CodeBling It IS germane, b/c the code, etc., that is paged in from code files can be later dropped from RAM if ithasn't been used for a while. Later, if it's used again, it is paged back in from the same file. That's "paging" even though there's (usually) no writing of pages. And, you can do read/write mapping of data files and modified pages out of those mappings DO get written back to the file ... and then paged back in as necessary. (This is in fact the preferred way to access large files.) Getting rid of the pagefile does not turn any of this off, so it absolutely does not "disable swap". – Jamie Hanrahan – 2016-03-26T07:53:48.780

@CodeBling And even if you are doing traditional read/write calls to access files, Windows file system cache is actually doing file mapping behind your back; your "read" and "write" calls simply move the data to or from the cache, which lives in kernel address space and is mapped to (i.e. backed by) the files you're accessing. Paging again! and it is not disabled by getting rid of the pagefile. An app can bypass this by opening the file handle with FILE_FLAG_NOBUFFERING, but then must work under some annoying constraints. – Jamie Hanrahan – 2016-03-27T02:25:43.040

@JamieHanrahan I see what you're saying, but I still don't see what you're getting at. What is the best way to prevent a process from being swapped to disk (in so far as swapping to disk can be prevented -- I don't think that the OP was hoping to disable file-to-memory mapping)? To me the best way to do THAT (in fact, the only way for an end-user) is still to set your page file size to zero, regardless of whether this is the best solution for the average user in the long run and regardless of whether this "completely disables" swap or caching. ..no? – Codebling – 2016-03-28T19:17:17.050

@JamieHanrahan that's pretty good info for anyone reading the answer to have, though, and for anyone interested in Windows internals and Windows paging. Nice job on the research (or know-how) – Codebling – 2016-03-28T19:28:40.383

But but but... the code you're running is part of your process. So are mapped data files. There's nothing you can do to keep that stuff from being paged out. Removing the pagefile keeps only the private v.a.s. (what Task Manager calls "Commit size") in RAM. And it forces that on every process in the system (and pageable kernel-space stuff like the paged pool), not just the one you care about. So no, there is no way to disable paging for JUST your process. – Jamie Hanrahan – 2016-03-28T23:26:20.190

@JamieHanrahan agreed. In the case of a virtual machine, though, don't you think that the private VAS whose paging is causing the slowdown? It does affect every process, but that's already noted in the answer. – Codebling – 2016-03-29T00:18:03.413

1

there is a good discussion about whether or not to use windows paging... http://serverfault.com/questions/23621/any-benefit-or-detriment-from-removing-a-pagefile-on-an-8gb-ram-machine . Given that there is soo many up votes for "keep paging at the default settings", I will keep paging.

– Trevor Boyd Smith – 2011-05-24T20:01:52.730

Good find. Don't forget that there is no blanket solution for everyone. It all depends on how you use your machine. You should consider trying it, even if you wind up restoring the old settings afterwards. I have a 3gb machine and run with no page file, and have been able to eliminate significant performance issues by doing so. But I have to watch my total mem usage. – Codebling – 2011-05-24T20:09:51.703

Yeah I would avoid disabling the page file, all you need is one rogue process to bring the system to its knees and not be able to recover. Windows is fairly smart in that in some situations it will use RAM to maintain a system cache, even if it means caching unfrequently used memory, so that common OS functions and frequently launched apps are sped up. – Spectre – 2011-05-24T20:21:33.097

Disabling page file might not work out, but again, there's no harm in trying. At least you will know if that is what is really causing your VM slowdown. – Codebling – 2011-05-24T20:34:48.560

Yes he did. but he meant "even if it means caching [in]frequently used [applications/processes/etc]". – Trevor Boyd Smith – 2011-05-24T20:40:50.763

@Code Bling. Ya having to watch your memory usage is really sad in this day and age. Man I need to fix this ASAP. I mean wow... totally sux. – Trevor Boyd Smith – 2011-05-24T20:42:12.303

1FYI, the windows limitation of 3GB is caused by a windows "feature" called "ignore the hardware's PAE capability". All Linux distros that are 32 bit can have 64 GB of ram (but each 32 bit process is limited to 2 GB of memory). – Trevor Boyd Smith – 2011-05-24T20:43:40.150

@TrevorBoydSmith: I knew what he meant but I decided to poke fun at him anyways :) Then I decided that my comment might be interpretted as being mean-spirited, so I deleted it. – Codebling – 2011-05-24T20:45:16.707

@TrevorBoydSmith: It is pretty sad. But my system was becoming completely unusable whenever I started using more memory than I physically had. So I had to watch it anyways. – Codebling – 2011-05-24T20:47:07.783

Well i missed it so :) The windows 32bit "limitation" is a "feature" so to speak, iirc called backwards compatibility (mostly for drivers I assume). It varies from 3 to 4GB depending on other address space that is required (eg GPU mem). iirc 32bit server editions of Windows can support more than 4GB ram, just as linux can. – Spectre – 2011-05-24T20:52:36.663

@Spectre: I asked if you had just said that Windows uses RAM to cache memory :) – Codebling – 2011-05-24T20:56:28.380

So if I turn off that 'feature', my swap will work again ? wow. +1 for answering the comment in my answer! – Codebling – 2011-05-24T20:58:16.163

yes you can easily turn off the feature. you just need to buy the software enhancement called "buy 64-bit windows and suck it"^{TM Microsft}. – Trevor Boyd Smith – 2011-05-24T21:02:27.247

oh haha, i mistyped, should have been "even if it means paging unfrequently used memory" :P – Spectre – 2011-05-24T21:07:35.793

infrequently... there is no word "unfrequently". http://www.youtube.com/watch?v=8iSD9lPVY6Q

– Trevor Boyd Smith – 2011-05-24T21:09:13.957

@Trevor, @Spectre, a 32-bit system can use more than 4GB of RAM? How? Is it like the segment/offset system that real-mode DOS used to use? – Synetech – 2011-05-24T21:09:24.903

look up PAE. in a nutshell, hardware makers decided in the 90s to add 4 extra bits to the address... so you have 2^36 addressable bytes (hence 64 gb memory). msft decided to ignore the 4 extra bits because they don't like to be standards compliant. – Trevor Boyd Smith – 2011-05-24T21:12:32.953

@TrevorBoydSmith: lolz. I will look this up. In other news, I feel like this comment thread almost deserves a question of its own. – Codebling – 2011-05-24T21:45:18.923

Asked: 2011-05-24T18:20:07.897

Viewed: 917 times

Active: 2011-05-24T20:00:08.503