Is it possible to hint Windows to bring a process completely out of swap

Is it possible to hint the Windows virtual-memory-manager to bring a particular process out of swap?

The situation I often find myself in is normally using about 50% of my RAM for all the normal programs, one in particular being an IDE for an embedded target, along with its associated debugger.

Then I leave a memory hungry process running - eg wireshark or something that's accumulating data over a weekend, everything gets swapped out - as they should. After the big process is finished, most processes are left in swap.

Then the sporadic pauses to swap the process back in cause problems with the debugger, presumably due to timing issues in the comms between IDE and the hardware-debugger. So control of the target device can be sporadic until enough attempts have brought the debug-data in the IDE back out of swap.

Greg

Posted 2013-05-28T07:57:18.483

Reputation: 865

This is would be very useful! From the users' point of view, it is annoying to wait for the swapped off process after each click or user action. It would be much better to say: "unswap this process!", go for a coffee and return to a responsive system! – Tomas – 2013-12-11T11:28:49.940

I'm not familiar with the specifics of the swapping process of Windows but is this something you're looking for? There should be a working binary example in the comments.

– Rik – 2013-12-12T17:05:43.710

I once had a similar question, and wrote a program based on the answers I got on Stack Overflow.

– Kerrek SB – 2013-12-16T23:37:16.733

@Kerrek, I tried compiling your code under MinGW and got errors on each read "Failed to read one byte from 0x161000, error 299 (0 bytes read)." Since I only have Windows at work I wasn't brave enough to run the random binary you attached so the problem could be in my compilation. – Greg – 2013-12-17T05:44:16.937

But yes your approach sounds good, and exactly what I was looking for. – Greg – 2013-12-17T05:44:53.563

I told @KerrekSB to post his answer here. Greg, Kerrek, I also got lots of errors "Failed to read one byte from 0x....000, error 299 (0 bytes read)." But the tool works as a charm!!! So Greg I think it works, what do you think? – Tomas – 2013-12-18T11:00:15.657

@Greg: Yeah, I get those errors too and just ignore them. If you look at the code, you'll see that most pages are read fine, and there are only occasional errors. If anyone with Windows skills can tell me what the error means and how I should handle it, I'd welcome that, but note that it's totally benign. – Kerrek SB – 2013-12-18T11:04:18.663

1@KerrekSB, seems like minor problem. Maybe you are touching memory area which is not mapped? Anyway I think your answer deserves the bounty :) PS: the unswap seems much slower than it could be, compared to HDD speed. Seems that the bottleneck is somewhere else, but where? – Tomas – 2013-12-18T11:07:13.600

@Tomas I didn't get a chance to verify it - with the error happening and a very quick response I assumed nothing had happened, but then I wasn't massively overswapped at the time - I was just watching task-manager for signs of IO/unpaging. Its your rep to award and you've seen the proof. I'm convinced its the right solution assuming the app worked for you. – Greg – 2013-12-18T12:30:01.427

@KerrekSB it's your pity - you could have gained 50rep had you posted your answer; now the bounty will go waste. You have 10 minutes. – Tomas – 2013-12-19T10:58:59.937

@Tomas: That's perfectly fine. I got plenty of rep for the original question already :-) – Kerrek SB – 2013-12-19T11:15:24.440

@Tomas: The hint I got from Rik solved the problem: The erroneous pages are guard pages which aren't meant to be read. I added an explicit condition for that, so that the program should now run silently. – Kerrek SB – 2013-12-20T00:32:04.367

@KerrekSB great! I thought it will be something like this :) – Tomas – 2013-12-20T08:21:50.120

Answers

KerrekSB developed a special tool for this purpose:

https://stackoverflow.com/a/2940209/684229

It is available on GiTHUB with binaries: https://github.com/louisdx/unpage

During the run, you get lots of errors "Failed to read one byte from 0x....000, error 299 (0 bytes read)", but it is not a problem, the tool works great.

Tomas

Posted 2013-05-28T07:57:18.483

Reputation: 5 107

"but it is a problem"? I don't think it's a problem. – Kerrek SB – 2013-12-19T11:16:12.630

Quote: "tool that tries to get all applications back into physical memory". With the poster's "memory hungry process" this might not work. – harrymc – 2013-12-19T11:46:35.947

@harrymc Quote from the question: After the big process is finished, most processes are left in swap.. So presumably he already ended the "memory hungry process". – Rik – 2013-12-19T12:01:09.410

@KerrekSB of course, that was a typo. Thanks :-) – Tomas – 2013-12-19T13:42:35.620

@Tomas I found this. With my very limited knowledge of c++ i verified that for every failed access the page is indeed protected by the PAGE_GUARD-bit. I used if (meminfo.Protect > 50) { std::cerr << meminfo.Protect << " - " << PAGE_GUARD << std::endl; } just before the ReadProcessMemory-line. PAGE_GUARD is 256 and i got 260 with the failed pages. So it is possible to get rid of the errors. Somebody with some c++ and Paging knowledge needs to read up on the PAGE_GUARD bit.

– Rik – 2013-12-19T14:52:29.147

@Rik great! Please pass it to KerrekSB under his original post, he is the author (I posted it as an answer here only because he refused to do so). – Tomas – 2013-12-19T14:59:40.320

You can use Process Lasso to give a process a Memory page priority :

Beginning with Windows Vista, each memory page has a priority ranging from 0 to 7. The Standby List is divided into eight lists that each handle pages of a different priority. When the Memory Manager wants to take a page from the Standby List, it takes pages from the low-priority lists first. "

Process Lasso can help manage memory priorities by allowing persistent memory priorities to be set for processes, so that their virtual memory pages are set to a specific priority each time run.

Process Lasso has two versions : free and commercial ($18.95 with trial).

harrymc

Posted 2013-05-28T07:57:18.483

Reputation: 306 093

I am afraid that you are answering completely different question. Giving process process a memory priority is different task, probably useless in the context OP and me sketched out: when you already have a process that is in swap and want to unswap it. You don't want to prevent it from going to swap. – Tomas – 2013-12-16T22:39:03.580

@Tomas: That's not my understanding. The request is to prevent a given process from being swapped out. If the process itself is not programmed to lock its pages into memory, the next best solution is to give it the highest memory priority to make it less likely to be swapped out. – harrymc – 2013-12-17T06:14:21.817

3@harrymc, OP says when the memory hog runs, "everything gets swapped out - as they should" so that is what the OP wants. It's just that on Monday morning he wants the important processes' working sets to be brought in all at once, not in dribs and drabs by page faulting over time. – mgkrebbs – 2013-12-18T00:15:28.530

@mgkrebbs: That's a different solution to the problem, which I believe is impossible to do. My solution is that if the IDE is never swapped out, then there is no need to swap it back in. The "memory hungry process" will just have a few megabytes less of RAM, which won't impact too much on its performance. – harrymc – 2013-12-18T12:40:34.620