Linux using 15GB of RAM with nothing

Question

I'm having a problem with my linux server, I searched many times but I couldn't find a case like mine.

I have a linux server with 32GB RAM, it's fine when it boots but after some days in execution the RAM gets used for basically nothing.

Below is what htop shows after I stop almost all services. 16GB is being used by something that I don't know, 2GB is being used by buffers and 0.6GB by caches. All 45 tasks are listed there, what happened to my RAM?

And below is what it looks like when I reboot the system and stop all process right after it boots.

I really don't know what's happening with the RAM...

---

Edit: I left the server running for 3 days after I posted this, stopped almost all services again and I ran the following suggested commands:

htop

free -m

            total       used       free     shared    buffers     cached
Mem:         32068      18243      13824          0       2542       1138
-/+ buffers/cache:      14563      17505
Swap:         4204          5       4199

cat /proc/meminfo

MemTotal:       32837972 kB
MemFree:        14153316 kB
Buffers:         2603212 kB
Cached:          1167872 kB
SwapCached:          964 kB
Active:          2591368 kB
Inactive:        1187300 kB
Active(anon):       6492 kB
Inactive(anon):     1304 kB
Active(file):    2584876 kB
Inactive(file):  1185996 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4305416 kB
SwapFree:        4300244 kB
Dirty:                28 kB
Writeback:             4 kB
AnonPages:          6968 kB
Mapped:             6220 kB
Shmem:               212 kB
Slab:           14685364 kB
SReclaimable:   14646664 kB
SUnreclaim:        38700 kB
KernelStack:        1544 kB
PageTables:         1936 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    20724400 kB
Committed_AS:      73192 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      133928 kB
VmallocChunk:   34359506832 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        4096 kB
DirectMap2M:     2084864 kB
DirectMap1G:    31457280 kB

ps aux|sort -k4

root       916  0.0  0.0      0     0 ?        SN   Nov07   0:00 [kipmi0]
root       101  0.0  0.0      0     0 ?        SN   Nov07   0:00 [ksmd]
root       102  0.0  0.0      0     0 ?        SN   Nov07   0:22 [khugepaged]
root       103  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/0]
root       104  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/1]
root       105  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/2]
root       106  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/3]
root       107  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/4]
root       108  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/5]
root       109  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/6]
root       110  0.0  0.0      0     0 ?        S    Nov07   0:00 [aio/7]
root        46  0.0  0.0      0     0 ?        S    Nov07   0:00 [async/mgr]
root        69  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_aux]
root        70  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/0]
root        71  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/1]
root        72  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/2]
root        73  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/3]
root        74  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/4]
root        75  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/5]
root        76  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/6]
root        77  0.0  0.0      0     0 ?        S    Nov07   0:00 [ata_sff/7]
root        49  0.0  0.0      0     0 ?        S    Nov07   0:00 [bdi-default]
root        43  0.0  0.0      0     0 ?        S    Nov07   0:00 [cgroup]
root       111  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/0]
root       112  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/1]
root       113  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/2]
root       114  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/3]
root       115  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/4]
root       116  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/5]
root       117  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/6]
root       118  0.0  0.0      0     0 ?        S    Nov07   0:00 [crypto/7]
root       137  0.0  0.0      0     0 ?        S    Nov07   0:00 [deferwq]
root      1026  0.0  0.0      0     0 ?        S    Nov07   0:00 [ext4-dio-unwrit]
root        66  0.0  0.0      0     0 ?        S    Nov07   0:00 [kacpid]
root        68  0.0  0.0      0     0 ?        S    Nov07   0:00 [kacpi_hotplug]
root        67  0.0  0.0      0     0 ?        S    Nov07   0:00 [kacpi_notify]
root      1068  0.0  0.0      0     0 ?        S    Nov07   0:00 [kauditd]
root       168  0.0  0.0      0     0 ?        S    Nov07   0:00 [kdmremove]
root        44  0.0  0.0      0     0 ?        S    Nov07   0:00 [khelper]
root        79  0.0  0.0      0     0 ?        S    Nov07   0:00 [khubd]
root        99  0.0  0.0      0     0 ?        S    Nov07   0:00 [khungtaskd]
root        50  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/0]
root        51  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/1]
root        52  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/2]
root        53  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/3]
root        54  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/4]
root        55  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/5]
root        56  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/6]
root        57  0.0  0.0      0     0 ?        S    Nov07   0:00 [kintegrityd/7]
root      1021  0.0  0.0      0     0 ?        S    Nov07   0:00 [kjournald]
root       135  0.0  0.0      0     0 ?        S    Nov07   0:00 [kpsmoused]
root        80  0.0  0.0      0     0 ?        S    Nov07   0:00 [kseriod]
root       169  0.0  0.0      0     0 ?        S    Nov07   0:00 [kstriped]
root        78  0.0  0.0      0     0 ?        S    Nov07   0:00 [ksuspend_usbd]
root         2  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthreadd]
root       126  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/0]
root       127  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/1]
root       128  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/2]
root       129  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/3]
root       130  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/4]
root       131  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/5]
root       132  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/6]
root       133  0.0  0.0      0     0 ?        S    Nov07   0:00 [kthrotld/7]
root        97  0.0  0.0      0     0 ?        S    Nov07   0:00 [linkwatch]
root        81  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/0]
root        82  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/1]
root        83  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/2]
root        84  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/3]
root        85  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/4]
root        86  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/5]
root        87  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/6]
root        88  0.0  0.0      0     0 ?        S    Nov07   0:00 [md/7]
root        89  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/0]
root        90  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/1]
root        91  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/2]
root        92  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/3]
root        93  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/4]
root        94  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/5]
root        95  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/6]
root        96  0.0  0.0      0     0 ?        S    Nov07   0:00 [md_misc/7]
root        45  0.0  0.0      0     0 ?        S    Nov07   0:00 [netns]
root        47  0.0  0.0      0     0 ?        S    Nov07   0:00 [pm]
root       360  0.0  0.0      0     0 ?        S    Nov07   0:00 [scsi_eh_0]
root       361  0.0  0.0      0     0 ?        S    Nov07   0:00 [scsi_eh_1]
root       362  0.0  0.0      0     0 ?        S    Nov07   0:00 [scsi_eh_2]
root       363  0.0  0.0      0     0 ?        S    Nov07   0:00 [scsi_eh_3]
root       364  0.0  0.0      0     0 ?        S    Nov07   0:00 [scsi_eh_4]
root       365  0.0  0.0      0     0 ?        S    Nov07   0:00 [scsi_eh_5]
root         5  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/0]
root         8  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/1]
root        12  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/2]
root        16  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/3]
root        20  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/4]
root        24  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/5]
root        28  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/6]
root        32  0.0  0.0      0     0 ?        S    Nov07   0:00 [stopper/7]
root        48  0.0  0.0      0     0 ?        S    Nov07   0:00 [sync_supers]
root       136  0.0  0.0      0     0 ?        S    Nov07   0:00 [usbhid_resumer]
root         6  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/0]
root        10  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/1]
root        14  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/2]
root        18  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/3]
root        22  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/4]
root        26  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/5]
root        30  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/6]
root        34  0.0  0.0      0     0 ?        S    Nov07   0:00 [watchdog/7]
root        11  0.0  0.0      0     0 ?        S    Nov07   0:04 [migration/2]
root      1022  0.0  0.0      0     0 ?        S    Nov07   0:05 [kjournald]
root         3  0.0  0.0      0     0 ?        S    Nov07   0:05 [migration/0]
root         7  0.0  0.0      0     0 ?        S    Nov07   0:05 [migration/1]
root        15  0.0  0.0      0     0 ?        S    Nov07   0:05 [migration/3]
root        60  0.0  0.0      0     0 ?        S    Nov07   0:06 [kblockd/2]
root        64  0.0  0.0      0     0 ?        S    Nov07   0:06 [kblockd/6]
root        62  0.0  0.0      0     0 ?        S    Nov07   0:07 [kblockd/4]
root        19  0.0  0.0      0     0 ?        S    Nov07   0:07 [migration/4]
root        59  0.0  0.0      0     0 ?        S    Nov07   0:08 [kblockd/1]
root        31  0.0  0.0      0     0 ?        S    Nov07   0:08 [migration/7]
root        61  0.0  0.0      0     0 ?        S    Nov07   0:10 [kblockd/3]
root        65  0.0  0.0      0     0 ?        S    Nov07   0:10 [kblockd/7]
root        63  0.0  0.0      0     0 ?        S    Nov07   0:12 [kblockd/5]
root        27  0.0  0.0      0     0 ?        S    Nov07   0:12 [migration/6]
root        58  0.0  0.0      0     0 ?        S    Nov07   0:13 [kblockd/0]
root        23  0.0  0.0      0     0 ?        S    Nov07   0:16 [migration/5]
root        38  0.0  0.0      0     0 ?        S    Nov07   0:18 [events/3]
root        29  0.0  0.0      0     0 ?        S    Nov07   0:18 [ksoftirqd/6]
root        13  0.0  0.0      0     0 ?        S    Nov07   0:19 [ksoftirqd/2]
root        37  0.0  0.0      0     0 ?        S    Nov07   0:20 [events/2]
root        40  0.0  0.0      0     0 ?        S    Nov07   0:20 [events/5]
root        41  0.0  0.0      0     0 ?        S    Nov07   0:20 [events/6]
root        21  0.0  0.0      0     0 ?        S    Nov07   0:22 [ksoftirqd/4]
root        36  0.0  0.0      0     0 ?        S    Nov07   0:23 [events/1]
root        42  0.0  0.0      0     0 ?        S    Nov07   0:24 [events/7]
root         9  0.0  0.0      0     0 ?        S    Nov07   0:24 [ksoftirqd/1]
root        35  0.0  0.0      0     0 ?        S    Nov07   0:27 [events/0]
root      1221  0.0  0.0      0     0 ?        S    Nov07   0:58 [flush-8:0]
root        17  0.0  0.0      0     0 ?        S    Nov07   1:01 [ksoftirqd/3]
root      1025  0.0  0.0      0     0 ?        S    Nov07   1:08 [jbd2/sdb1-8]
root         4  0.0  0.0      0     0 ?        S    Nov07   1:11 [ksoftirqd/0]
root        25  0.0  0.0      0     0 ?        S    Nov07   1:13 [ksoftirqd/5]
root        33  0.0  0.0      0     0 ?        S    Nov07   1:24 [ksoftirqd/7]
root        39  0.0  0.0      0     0 ?        S    Nov07   2:28 [events/4]
root       450  0.0  0.0      0     0 ?        S    Nov07   2:49 [kjournald]
root      1341  0.0  0.0      0     0 ?        S    Nov07   3:22 [kondemand/6]
root      1337  0.0  0.0      0     0 ?        S    Nov07   3:47 [kondemand/2]
root      1339  0.0  0.0      0     0 ?        S    Nov07   3:57 [kondemand/4]
root      1336  0.0  0.0      0     0 ?        S    Nov07   4:05 [kondemand/1]
root      1342  0.0  0.0      0     0 ?        S    Nov07   4:38 [kondemand/7]
root      1340  0.0  0.0      0     0 ?        S    Nov07   4:39 [kondemand/5]
root      1338  0.0  0.0      0     0 ?        S    Nov07   4:47 [kondemand/3]
root      1335  0.0  0.0      0     0 ?        S    Nov07   5:10 [kondemand/0]
root       100  0.1  0.0      0     0 ?        S    Nov07   7:53 [kswapd0]
root      1216  0.0  0.0  10812   236 ?        S<   Nov07   0:00 /sbin/udevd -d
root      1217  0.0  0.0  10812   236 ?        S<   Nov07   0:00 /sbin/udevd -d
root       544  0.0  0.0  10816   268 ?        S<s  Nov07   0:00 /sbin/udevd -d
root     27762  0.0  0.0 110232  1120 pts/1    R+   05:33   0:00 ps aux
root     27188  0.0  0.0 111024  4440 pts/1    Ss   05:21   0:00 -bash
root     27184  0.0  0.0 112008  5076 ?        Ss   05:21   0:00 sshd: root@pts/1
root      1949  0.0  0.0 112584   844 ?        Ss   Nov07   0:00 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
root      1941  0.0  0.0 114704   940 ?        Ss   Nov07   0:00 /usr/sbin/abrtd
root      2558  0.0  0.0 116856   728 ?        Ss   Nov07   0:02 crond
root      1873  0.0  0.0  11912   576 ?        S    Nov07   0:00 /usr/lib64/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 143 /usr/sbin/imaplogin /usr/bin/imapd Maildir
root     27763  0.0  0.0 155428   852 pts/1    S+   05:33   0:00 sort -k4
68        1785  0.0  0.0  18008   692 ?        S    Nov07   0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root      1364  0.0  0.0  18316   548 ?        Ss   Nov07   0:25 irqbalance --pid=/var/run/irqbalance.pid
root         1  0.0  0.0  19356  1156 ?        Ss   Nov07   0:00 /sbin/init
root      1745  0.0  0.0  20400   664 ?        S    Nov07   0:00 hald-runner
root      2584  0.0  0.0  21104   304 ?        Ss   Nov07   0:00 /usr/sbin/atd
dbus      1702  0.0  0.0  21432   772 ?        Ss   Nov07   0:00 dbus-daemon --system
root      1849  0.0  0.0  21716   768 ?        Ss   Nov07   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root      1777  0.0  0.0  22520   612 ?        S    Nov07   0:00 hald-addon-input: Listening on /dev/input/event4 /dev/input/event0 /dev/input/event1 /dev/input/event2
root      1312  0.0  0.0 249088  1872 ?        Sl   Nov07   0:01 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
68        1744  0.0  0.0  38052  1824 ?        Ssl  Nov07   0:01 hald
root      2611  0.0  0.0   4064   464 tty1     Ss+  Nov07   0:00 /sbin/mingetty /dev/tty1
root      2613  0.0  0.0   4064   464 tty2     Ss+  Nov07   0:00 /sbin/mingetty /dev/tty2
root      2615  0.0  0.0   4064   464 tty3     Ss+  Nov07   0:00 /sbin/mingetty /dev/tty3
root      2617  0.0  0.0   4064   464 tty4     Ss+  Nov07   0:00 /sbin/mingetty /dev/tty4
root      2619  0.0  0.0   4064   464 tty5     Ss+  Nov07   0:00 /sbin/mingetty /dev/tty5
root      2621  0.0  0.0   4064   464 tty6     Ss+  Nov07   0:00 /sbin/mingetty /dev/tty6
root      1872  0.0  0.0   4068   408 ?        S    Nov07   0:00 /usr/sbin/courierlogger -name=imapd -pid=/var/run/imapd.pid -lockfile=/var/lock/subsys/courier-imapd -start -name=courier-imapd /usr/lib64/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 143 /usr/sbin/imaplogin /usr/bin/imapd Maildir
root      1734  0.0  0.0   4080   488 ?        Ss   Nov07   0:00 /usr/sbin/acpid
root      1596  0.0  0.0   4576   224 ?        Ss   Nov07   0:00 mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
root      1840  0.0  0.0  66216   508 ?        Ss   Nov07   0:00 /usr/sbin/sshd
root      1282  0.0  0.0  93156   720 ?        S<sl Nov07   0:00 auditd
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

vmstat -s -S M | grep mem

    32068 M total memory
    18243 M used memory
     2530 M active memory
     1157 M inactive memory
    13824 M free memory
     2542 M buffer memory

After this commands I noticed that this big numbers on /proc/meminfo:

Slab:           14685364 kB
SReclaimable:   14646664 kB
CommitLimit:    20724400 kB
VmallocTotal:   34359738367 kB
VmallocChunk:   34359506832 kB
DirectMap1G:    31457280 kB

What can I do to fix this high memory usage?

---

Edit: After researching about slab memory leak I found that it could be a driver or something related to the kernel and that slabtop could help, I noticed it happening in 20 minutes of execution. I'm starting to think that it's related to my backup crontab that uses rsnapshot and creates many hardlinks. I'm going to test it.

slabtop

Answer 1

htop versions provided by various Linux distributions differ significantly in how they report memory usage.

Look into /proc/meminfo to see which type of memory is growing and is being accounted by your version of htop as being in use.

---

Updated after meminfo and slabtop outputs were provided:

If you look at the data in detail, you'll find that the missing memory is all used by reclaimable SLAB, that means for in-kernel cache-like allocations. Reclaimable SLAB will be freed in case memory pressure grows.

The slabinfo output shows that most of the SLAB is used for Dentry cache and that only 9% of that cache is hot, meaning that 91% of the missing 14G can be considered available memory (even though not free).

The Dentry cache is used for caching directory entry data to speed up path searches. You have mentioned that the memory usage skyrockets after you do a backup. That seems indeed the reason - the kernel is keeping the directory entry data of all your files in memory just in case you'd want to scan all the directories for another backup run soon.

The conclusion is that this is correct and proper behavior of the kernel and again, like always, the simple userspace tools are giving only a very limited insight into what is happening on the system.

Answer 2

Observe in /proc/meminfo, that Slab and SReclaimable are both very high, accounting for most of the usage. Slab is what the kernel uses for its own internal data structures, and SReclaimable is the portion of Slab that can be recovered.

What the kernel is doing with all that memory, I don't know. It doesn't seem to be caches or buffers; those are both relatively small. But it's mostly reclaimable, so unless you're seeing an actual performance problem, it's probably okay.

More details here: https://serverfault.com/questions/240277

There's a posting here where someone did have a problem with large Slab, eventually leading to a crash. Upgrading their kernel fixed it. However, this was back in 2011, so you probably don't have that bug: https://serverfault.com/questions/258717

Answer 3

That doesn't look right at all... all of the listed processes are using 0.0% CPU and MEM, and only 45 processes sounds very low, should a server be that low? My desktop Mint has 200-some processes (from ps aux|wc -l).

You can sort htop by MEM% with the F6 key, then select MEM% and enter, or see

sudo ps aux|sort -k4

that should sort by the 4th column, %MEM if your defaults are the same as mine, change accordingly if they're different.

Also, free (or free -m for megabytes) shows the line -/+ buffers/cache: that works out the math with the buffers & cache:

             total       used       free     shared    buffers     cached
Mem:         12011       9825       2186          0        243       5829
-/+ buffers/cache:       3752       8259
Swap:        16378        313      16065

This might be interesting too: vmstat -s -S M | grep mem