1

I have a one a one node Kubernetes cluster and the memory usage reported by the metrics server does not seem to be the same as the memory usage shown with the free command

# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
<node_ip>   1631m        10%    13477Mi         43%

# free -m
              total        used        free      shared  buff/cache   available
Mem:          32010       10794         488          81       20727       19133
Swap:         16127        1735       14392

And the difference is significant ~ 3 GB.

I have also tested this on a 3 node cluster, and the issue is present there too:

# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
<node_ip1>   1254m        8%     26211Mi         84%
<node_ip2>   221m         1%     5021Mi          16%
<node_ip3>   363m         2%     8731Mi          28%
<node_ip4>   1860m        11%    20399Mi         66%

# free -m (this is on node 1)
              total        used        free      shared  buff/cache   available
Mem:          32010        5787         369        1676       25853       24128
Swap:         16127           0       16127

Why is there a difference?

bitscuit
  • 111
  • 3

1 Answers1

3

Let's start from understanding how both commands works.

Free - Display amount of free and used memory in the system

Kubectl top - Allows you to see the resource consumption for nodes or pods.

There is an article explaining why you might see the differences:

Most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others. Most of the Linux tools providing system resource metrics were created before cgroups even existed. They usually read memory metrics from the proc filesystem: /proc/meminfo, /proc/vmstat, /proc/PID/smaps and others.

Also, the swap part (as mentioned by @uav) should trigger a red light as you must disable it in order for the kubelet to work properly. It is one of the prerequisites for installing kubeadm.

I hope it helps.

Wytrzymały Wiktor
  • 731
  • 3
  • 10
  • I actually have the opposite problem. `kubectl top node` reports really lower mem usage than `free` and I cannot find where all the different memory is spent (https://unix.stackexchange.com/questions/621512/cannot-find-memory-usage-in-kubernetes-cluster-node) – mitsos1os Nov 25 '20 at 20:17