1

In managing a server with 512GB of RAM I encountered a process that keeps consuming swap until it hits 100% of the swap space then stops consuming more (6GB of swap) but keeps working fine (albeit, when a request enters the process, it takes a long time (20+ min) to get to the required performance).

Even setting swappiness to 0 doesn't prevent this proces from swapping.

Swapping happens while free shows this:

# free -h --giga
              total        used        free      shared  buff/cache   available
Mem:           515G         16G        2.3G         30M        497G        496G
Swap:          5.8G        1.0M        5.8G

The process in question:

# smem -s swap -t -n -k
  PID User     Command                         Swap      USS      PSS      RSS
(...)
36776 1000     java -XX:+UseG1GC -Xms1G -X     1.6M    13.4G    13.4G    13.4G
-------------------------------------------------------------------------------
  148 11                                       2.0M    15.1G    15.2G    15.9G

It keeps growing and growing over time (at a rate of ~20 MB / hour) until 100% swap is consumed. It might be worth mentioning it is running in a docker container but I don't know if that influences anything.

Swappiness:

# cat /proc/sys/vm/swappiness
0

I really want to disable swap completely at this point but this answer strongly recommends against it. What are my options to keep this programs' memory completely in RAM?

Gizmo
  • 279
  • 2
  • 11

1 Answers1

3

Since your question is about process inside Docker container, it is worth checking if you are not missing vm.overcommit_memory=1 configuration as described here: Node using swap memory instead of host memory

By default, Docker recommends using a value of vm.swappiness=0 for Docker environments, which prevents swapping except in the case of an OOM (OutOfMemory) condition. All nodes must set vm.overcommit_memory=1, which tells the kernel to always allow memory allocations until there is no truly memory. This article explains a situation that can occur whenever a value other 0 is used for vm.swappiness. If vm.swappiness is set to a value higher than 0, you might notice that only swap memory is being used on the node even though host memory was available.

eider96
  • 56
  • 5
  • however your swap should be incraise regardless because ususally you takt at least 50 percent of RAM, in your case in my point you should have st least 50/100gb swap – djdomi Aug 11 '19 at 20:18
  • This solution did slow down the increase of the swap "eating" but did not completely stop it. For now my only solution seems to disable swap completely as I cannot find any good reasons to keep swap enabled on a 512GB RAM system. My reason: of those 512GB there is 5 GB truly 'free/unused/wasted' by the Linux Kernel. The swapping has a negative performance impact in the long run on a proces that must be responsive. Out of the 512GB only 30GB's are truly "used up", the rest is cache/buffers/shared. Swapping is also very slow, in the ranges of kilobytes to 2megabytes/s, not 500MB/s (SSD). – Gizmo Aug 11 '19 at 20:28