AWS EC2 Amazon Linux is using swap even if it shouldn't

Question

I have a few ECS servers running various apps. After a few mysterious outages I noticed that the instance is swapping hard and it consumes its IO Burst Balance rendering the instance useless.

The facts:

the machines are t2 and t3 instances with 2GB or 4GB RAM
there is more than enough real RAM on the instances for the apps running
vm.swappiness is set to 1 or 5
according to my monitoring the memory usage on the instances is constantly between 60% and 68% (this is the usage without buffer/cache, the remaining "free" memory is always used by buffer/cache)
SWAP usage is always ~30-40% (with 1-1,5GB SWAP configured); and vmstat is reporting some si and so, like

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 434948  83616  73468 818316  580    0   580     0  883 1453  5  1 93  1  0
 2  0 429060  65308  73468 818320 17760    0 17760    76 5927 6318 15 21 52 12  1
 0  3 440660  81180  73464 817208 4352 12880  4352 12880 5791 2672  8 39  0 52  0
 1  1 443164  78080  73452 816304 10104 3272 10104  3272 3850 3807  7 34  0 58  1
 1  1 438456  63944  73460 816356 13168  156 13216   212 3673 3686 12 34 21 32  0
 1  1 449472  72748  73460 815872 1600 11016  1600 11016 3815 1445  1 23  0 74  1
 0  4 453592  59728  73460 815964 3324 8748  4080  8752 4891 2881 10 34  1 54  0
 1  2 462724  68380  73456 815152 1704 10412  1704 10412 5381 2195  7 35  1 55  0
 0  2 473444  81148  73456 815108 1024 10976  1024 10976 4637 1608  2 34  2 63  1
 0  1 483124  87432  73444 818604 2352 9684  6136  9684 4526 1800  8 28  8 54  1
 0  1 480308  66860  73452 824476 13656    0 19536    36 3982 4103 11 27 39 22  0
 0  3 483744  75016  73448 825132 5016 7532  5860  7532 5645 4748 14 33  6 47  0
 2  2 493140  80348  73448 825068 1888 10164  1888 10164 4128 1993  2 34  0 62  0
 1  0 497544  78488  73440 824976 6248 6452  6364  6452 4192 2916  3 39 16 41  1
 0  1 494216  65096  73440 825068 12488    0 12616     0 4070 4620  7 26 42 24  1
 0  2 503856  69436  73452 824760 1968 10152  1968 10192 4885 1821  4 28  2 65  0
 1  0 514264  88532  73440 824188 1332 10668  1332 10672 4749 1622  2 25  1 71  0
 0  1 512984  76628  73440 824120 12844    0 12844     0 3653 3517  6 26 48 20  0
 1  3 479192  65220  73440 824556 12484    0 12832     0 3729 4821  6 29 34 31  0
 0  5 462680  78612  73440 824824 6120 6784  6532  6784 4111 3582  5 45  0 49  1
 0  2 433376  99148  73448 824992 5884 7560  6024  7596 3954 3407  3 33  1 62  0

the disks are ~20GB EBS gp2 volumes shared with the swapfile, so the overall IO capacity is kind of limited
vm.overcommit_memory is 0

The output of cat /proc/meminfo on 2 identical (one is with, the other without SWAP) is

with SWAP disabled:

MemTotal:        2004488 kB
MemFree:          178976 kB
MemAvailable:     489360 kB
Buffers:          136892 kB
Cached:           290396 kB
SwapCached:            0 kB
Active:          1063520 kB
Inactive:         612900 kB
Active(anon):     817380 kB
Inactive(anon):   436892 kB
Active(file):     246140 kB
Inactive(file):   176008 kB
Unevictable:       17620 kB
Mlocked:           17620 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:       1262220 kB
Mapped:           140336 kB
Shmem:               252 kB
Slab:              73448 kB
SReclaimable:      39612 kB
SUnreclaim:        33836 kB
KernelStack:        6944 kB
PageTables:        15300 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1002244 kB
Committed_AS:    3076324 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      182248 kB
DirectMap2M:     1875968 kB
DirectMap1G:           0 kB

and with SWAP enabled:

MemTotal:        2004488 kB
MemFree:          116220 kB
MemAvailable:     925764 kB
Buffers:           73472 kB
Cached:           827312 kB
SwapCached:       136268 kB
Active:           653736 kB
Inactive:        1041616 kB
Active(anon):     375736 kB
Inactive(anon):   424652 kB
Active(file):     278000 kB
Inactive(file):   616964 kB
Unevictable:       17620 kB
Mlocked:           17620 kB
SwapTotal:       1499132 kB
SwapFree:        1061032 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:        749004 kB
Mapped:           180880 kB
Shmem:               252 kB
Slab:             118940 kB
SReclaimable:      92304 kB
SUnreclaim:        26636 kB
KernelStack:        6784 kB
PageTables:        14212 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2501376 kB
Committed_AS:    2881136 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      110568 kB
DirectMap2M:     1947648 kB
DirectMap1G:           0 kB

What I tried:

I experimented with various vm.swappiness values but they seem to do nothing
I even tried to reboot and do swapoff -a && swapon -a after vm.swappiness change

What I am expecting to happen:

in my imagination with my settings the SWAP should remain all free all the time except in case of some unfortunate event when the total memory usage would exceed the available memory

Why is this happening? What am I missing? What should I do differently? Unfortunately AFAIK it is not possible to use Instance Store (and putting SWAP on that volume) with t2/t3 ATM.

At the moment my machines are running with SWAP turned off, and they are happy with this; only I am a little bit worried. (Edit:) My main objective would be to convince the system to only touch SWAP as a last resort.

My other simple EC2 Ubuntu machines are behaving the way they should.

Please edit your question to add the contents of `/proc/meminfo`. Likely `Committed_AS` is a significant fraction of `MemTotal` so you get moderate but not severe paging out. — John Mahowald, Apr 06 '19 at 11:47

score 0 · Accepted Answer · answered Apr 08 '19 at 15:53

in my imagination with my settings the SWAP should remain all free all the time except in case of some unfortunate event when the total memory usage would exceed the available memory

An out of memory condition is too late. Reclaim by paging out is extremely slow, and Linux VMM tries to stay away from OOM.

Instead, a slow trickle of inactive pages out to swap space makes use of the space and I/O available. Why would you have a paging space if it was not to be used? A higher swappiness tunable would just get to the same paging space use faster.

For what documentation there is on the algorithms, have a look at the MM wiki, LWN articles, and some references over at UNIX Stack Exchange: What page replacement algorithms are used in Linux kernel for OS file cache?

On your host, the total allocated (Committed_AS) is about 3 GB while MemTotalis 2 GB. 150% is not necessarily bad, some of that is inactive or unused. But Linux will page out where it can because in the conservative worst case everything wouldn't fit in RAM. Safe memory sizing will be less than Committed_AS, but how much less is a bit tricky to quantify.

In your environment, IOPS are very expensive as they count against your quotas for permanent storage. Disable paging space and monitor for OOM.

AWS EC2 Amazon Linux is using swap even if it shouldn't

1 Answers1