We are using a centos 7 server. The free -m command has the following output.
[root@211 ~]# free -m
total used free shared buff/cache available
Mem: 31841 31173 389 0 278 178
Swap: 382975 45 382930
It shows that most of the memory is being used. It's not used for caching the filesystem but by some kernel/user processes. We don't have any processes running on this system though. The output of ps aux --sort=-%mem | head is as follows
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 603 0.0 0.0 109300 14524 ? Ss Apr25 0:12 /usr/lib/systemd/systemd-journald
root 1249 0.0 0.0 658752 6156 ? Ssl Apr25 36:56 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
root 1 0.0 0.0 194160 4640 ? Ss Apr25 0:13 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root 58818 0.0 0.0 155636 4008 pts/0 R+ 12:26 0:00 ps aux --sort=-%mem
root 56093 0.0 0.0 117168 3080 pts/0 Ss 10:01 0:00 -bash
root 1252 0.0 0.0 728324 2404 ? Ssl Apr25 0:25 /usr/sbin/rsyslogd -n
root 867 0.0 0.0 26388 1896 ? Ss Apr25 0:03 /usr/lib/systemd/systemd-logind
dbus 856 0.0 0.0 66440 1868 ? Ssl Apr25 0:04 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 58819 0.0 0.0 107968 1672 pts/0 S+ 12:26 0:00 head
I also checked tmpfs filesystems and /dev/shm. None of these have any huge files.
What could be consuming all the memory? Any pointers on how to debug this will be appreciated. Thanks!
Edit:
Adding cat proc/meminfo output
MemTotal: 32605760 kB
MemFree: 405320 kB
MemAvailable: 185992 kB
Buffers: 8824 kB
Cached: 35724 kB
SwapCached: 5040 kB
Active: 23176 kB
Inactive: 31500 kB
Active(anon): 4064 kB
Inactive(anon): 6136 kB
Active(file): 19112 kB
Inactive(file): 25364 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 392167420 kB
SwapFree: 392123388 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 8684 kB
Mapped: 24512 kB
Shmem: 32 kB
Slab: 234744 kB
SReclaimable: 59568 kB
SUnreclaim: 175176 kB
KernelStack: 4224 kB
PageTables: 4932 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 408470300 kB
Committed_AS: 238592 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8891936 kB
DirectMap2M: 24334336 kB
DirectMap1G: 0 kB
Adding output of df -t tmpfs
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 16302880 0 16302880 0% /dev/shm
tmpfs 16302880 1340 16301540 1% /run
tmpfs 16302880 0 16302880 0% /sys/fs/cgroup
tmpfs 398458880 46050 398412830 1% /tmp
tmpfs 3260576 0 3260576 0% /run/user/0
tmpfs 3260576 0 3260576 0% /run/user/1001
tmpfs 3260576 0 3260576 0% /run/user/1000
