Linux system completely unresponsive with lots of free swap space

I have a .NET (Core 2.0) service that I moved from Windows to Linux instances on AWS. The instances are micro with 1Gb RAM.

I added 1Gb of swap space to the linux instance, and also set swappiness=100 but the server freezes up when physical memory is filled up. The process itself slows down to almost stopping and even pressing ENTER on bash takes 10 seconds sometimes for the new line to appear.

Running top I see the free memory usually 10, 20mb. The process using 800Mb+ of RAM, and swap is always almost empty, with usage of 20mb max. Even letting there for an hour didn't swap more of it.

I can see Disk and CPU credits on AWS are at almost 100%, so it's not limiting resource use. Also, there are about a hundred of these instances, and I've replaced them multiple times, the behavior is always the same, so it doesn't look like a bad instance problem.

What is bothering me is that this didn't happen on windows, and the linux instances uses about 200mb less memory for the base system.

Is there any setting I need to tweak other than swappiness to make Linux move more memory to swap?

Edit: Swap is setup correctly through cloud-init, and it is surviving reboots fine:

Setup:

fallocate -l 1024M /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
sysctl vm.swappiness=100

free -m after boot:

             total       used       free     shared    buffers     cached
Mem:           993        232        760          0          7        152
-/+ buffers/cache:         72        921
Swap:         1023          0       1023

Natan

Posted 2018-01-03T15:38:02.030

Reputation: 141

It seems that you haven't configured your swap correctly: run free -h to check the amount of swap configured; or it could be that both your memory and swap are full (1GB is quite small for a swap file, especially if you have limited memory). This link tells you how to set it up for Ubuntu, but I guess it's a basic enough function that it will be similar for other Linux flavours.

– AFH – 2018-01-03T15:48:33.807

It's a swap file. The system is recognizing the swap. It's just not using almost none of it. I'll add these details to the question. – Natan – 2018-01-03T16:44:06.973

Depending on how Linux would access this memory, it may be too late by the time it tries. You may want to take a look at Azure. If you’re going to run .NET applications, it may be more suitable for you. – Daniel B – 2018-01-03T16:48:46.490

Do you have a disc access LED? If this is constantly lit when the system freezes, then maybe your swappiness value is too high, meaning that frequently-accessed programs may be swapping out and in to the exclusion of other programs. Try the opposite extreme and set swappiness=10 (sudo sysctl vm.swappiness=10 for temporary change) to see the effect: this should stop frequently-called processes from swapping out. – AFH – 2018-01-03T17:00:27.720

I tried with multiple values from 0 to 100, 100 was the one that worked best. With less than 60, I could barely SSH into the VM. There is only 1 process running, nothing else. It's an AWS instance, but no, there is no disk activity in the monitoring graph. – Natan – 2018-01-03T17:02:14.953

Sorry, I've run out of things to suggest, other than increasing the memory allocated to the VM, which doesn't explain the problem, but may work round it. Just make sure that kswapd is running (it may have a suffix or a different name, but ps -efl | grep -i swap should find the daemon). – AFH – 2018-01-03T17:19:09.810

@AFH I can confirm kswapd is running. Yes, I can run with higher memory instances, but because there are hundreds of instances, the costs increase a lot. We're profiling the app to decrease memory use, but it still doesn't explain why it works fine on windows. – Natan – 2018-01-03T17:24:52.943

Answers

I found the real issue. The application is running inside docker, and AWS deliberately blocks swap usage within ECS containers for some reason. This block didn't affect windows because we were not using ECS to manage docker before.

After talking to their support, they don't support swap within containers and don't know when they will. So, I'll have to move out of ECS and manage docker myself.

Natan

Posted 2018-01-03T15:38:02.030

Reputation: 141

You can accept your own answer, it'd be worth while as this is something that maybe of interest to others!. – djsmiley2k TMW – 2018-01-05T11:38:44.630

Well, your error may be actually quite small. That high swappiness value will cause a problem with some OS configurations. Try a value like 15. (Just a note: forcing your system to prefer swap is a horrible idea. Your system needs to use actual RAM to function normally. [in case you didn't know or reversed it, swappiness is the % of ram free before swap is used, so 15 is that 85% of RAM must be used before the SWAP partition is.])

Also, how did you add swap space? If you just changed configs and did not create a new partition, or left errors in your /etc/fstab file, you will be unable to use swap, and all hell will break use as the system tries to write to something that isnt there or it can not write to (or something way more interesting will happen). I've broken way more than one install through those methods.

Eric

Posted 2018-01-03T15:38:02.030

Reputation: 31

There is no error. The swap is properly configured and the system uses some of it. I also tried with many swappiness values before, from 0 to 100. With less then 60 the OS won't even bother using the swap partition. With 100 it swaps 20mb, and I'm able to use ssh without too many problems. – Natan – 2018-01-03T16:46:23.577

Its possible I reversed it, but I dont think so, but I do have dyslexia. – Eric – 2018-01-03T17:12:27.047