I have a few ECS servers running various apps. After a few mysterious outages I noticed that the instance is swapping hard and it consumes its IO Burst Balance rendering the instance useless.
The facts:
- the machines are
t2
andt3
instances with 2GB or 4GB RAM - there is more than enough real RAM on the instances for the apps running
vm.swappiness
is set to1
or5
- according to my monitoring the memory usage on the instances is constantly between 60% and 68% (this is the usage without buffer/cache, the remaining "free" memory is always used by buffer/cache)
- SWAP usage is always ~30-40% (with 1-1,5GB SWAP configured); and
vmstat
is reporting somesi
andso
, like
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 434948 83616 73468 818316 580 0 580 0 883 1453 5 1 93 1 0
2 0 429060 65308 73468 818320 17760 0 17760 76 5927 6318 15 21 52 12 1
0 3 440660 81180 73464 817208 4352 12880 4352 12880 5791 2672 8 39 0 52 0
1 1 443164 78080 73452 816304 10104 3272 10104 3272 3850 3807 7 34 0 58 1
1 1 438456 63944 73460 816356 13168 156 13216 212 3673 3686 12 34 21 32 0
1 1 449472 72748 73460 815872 1600 11016 1600 11016 3815 1445 1 23 0 74 1
0 4 453592 59728 73460 815964 3324 8748 4080 8752 4891 2881 10 34 1 54 0
1 2 462724 68380 73456 815152 1704 10412 1704 10412 5381 2195 7 35 1 55 0
0 2 473444 81148 73456 815108 1024 10976 1024 10976 4637 1608 2 34 2 63 1
0 1 483124 87432 73444 818604 2352 9684 6136 9684 4526 1800 8 28 8 54 1
0 1 480308 66860 73452 824476 13656 0 19536 36 3982 4103 11 27 39 22 0
0 3 483744 75016 73448 825132 5016 7532 5860 7532 5645 4748 14 33 6 47 0
2 2 493140 80348 73448 825068 1888 10164 1888 10164 4128 1993 2 34 0 62 0
1 0 497544 78488 73440 824976 6248 6452 6364 6452 4192 2916 3 39 16 41 1
0 1 494216 65096 73440 825068 12488 0 12616 0 4070 4620 7 26 42 24 1
0 2 503856 69436 73452 824760 1968 10152 1968 10192 4885 1821 4 28 2 65 0
1 0 514264 88532 73440 824188 1332 10668 1332 10672 4749 1622 2 25 1 71 0
0 1 512984 76628 73440 824120 12844 0 12844 0 3653 3517 6 26 48 20 0
1 3 479192 65220 73440 824556 12484 0 12832 0 3729 4821 6 29 34 31 0
0 5 462680 78612 73440 824824 6120 6784 6532 6784 4111 3582 5 45 0 49 1
0 2 433376 99148 73448 824992 5884 7560 6024 7596 3954 3407 3 33 1 62 0
- the disks are ~20GB EBS gp2 volumes shared with the swapfile, so the overall IO capacity is kind of limited
vm.overcommit_memory
is0
The output of cat /proc/meminfo
on 2 identical (one is with, the other without SWAP) is
with SWAP disabled:
MemTotal: 2004488 kB
MemFree: 178976 kB
MemAvailable: 489360 kB
Buffers: 136892 kB
Cached: 290396 kB
SwapCached: 0 kB
Active: 1063520 kB
Inactive: 612900 kB
Active(anon): 817380 kB
Inactive(anon): 436892 kB
Active(file): 246140 kB
Inactive(file): 176008 kB
Unevictable: 17620 kB
Mlocked: 17620 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 20 kB
Writeback: 0 kB
AnonPages: 1262220 kB
Mapped: 140336 kB
Shmem: 252 kB
Slab: 73448 kB
SReclaimable: 39612 kB
SUnreclaim: 33836 kB
KernelStack: 6944 kB
PageTables: 15300 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1002244 kB
Committed_AS: 3076324 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 182248 kB
DirectMap2M: 1875968 kB
DirectMap1G: 0 kB
and with SWAP enabled:
MemTotal: 2004488 kB
MemFree: 116220 kB
MemAvailable: 925764 kB
Buffers: 73472 kB
Cached: 827312 kB
SwapCached: 136268 kB
Active: 653736 kB
Inactive: 1041616 kB
Active(anon): 375736 kB
Inactive(anon): 424652 kB
Active(file): 278000 kB
Inactive(file): 616964 kB
Unevictable: 17620 kB
Mlocked: 17620 kB
SwapTotal: 1499132 kB
SwapFree: 1061032 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 749004 kB
Mapped: 180880 kB
Shmem: 252 kB
Slab: 118940 kB
SReclaimable: 92304 kB
SUnreclaim: 26636 kB
KernelStack: 6784 kB
PageTables: 14212 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2501376 kB
Committed_AS: 2881136 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 110568 kB
DirectMap2M: 1947648 kB
DirectMap1G: 0 kB
What I tried:
- I experimented with various
vm.swappiness
values but they seem to do nothing - I even tried to reboot and do
swapoff -a && swapon -a
aftervm.swappiness
change
What I am expecting to happen:
- in my imagination with my settings the SWAP should remain all free all the time except in case of some unfortunate event when the total memory usage would exceed the available memory
Why is this happening? What am I missing? What should I do differently?
Unfortunately AFAIK it is not possible to use Instance Store (and putting SWAP on that volume) with t2
/t3
ATM.
At the moment my machines are running with SWAP turned off, and they are happy with this; only I am a little bit worried. (Edit:) My main objective would be to convince the system to only touch SWAP as a last resort.
My other simple EC2 Ubuntu machines are behaving the way they should.