0
1
We have an EC2 instance that has a very large (50+GB) dentry cache and most of it is in use (99%). "drop_cache" commands (sysctl -w vm.drop_caches=2
or sysctl -w vm.drop_caches=3
) don't help much, because almost all of it is in use. Is there any way to find out what's in this large in-use dentry cache or pin down what's causing it?
$ cat /proc/meminfo
MemTotal: 64461192 kB
MemFree: 8585000 kB
MemAvailable: 62782360 kB
Buffers: 194872 kB
Cached: 1956268 kB
SwapCached: 0 kB
Active: 1345168 kB
Inactive: 1161952 kB
Active(anon): 432384 kB
Inactive(anon): 55304 kB
Active(file): 912784 kB
Inactive(file): 1106648 kB
Unevictable: 180872 kB
Mlocked: 180872 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 660 kB
Writeback: 0 kB
AnonPages: 536000 kB
Mapped: 241356 kB
Shmem: 58028 kB
KReclaimable: 52894088 kB
Slab: 52981812 kB
SReclaimable: 52894088 kB
SUnreclaim: 87724 kB
KernelStack: 7888 kB
PageTables: 9700 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 32230596 kB
Committed_AS: 3014752 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 3520 kB
HardwareCorrupted: 0 kB
AnonHugePages: 274432 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1101796 kB
DirectMap2M: 35117056 kB
DirectMap1G: 29360128 kB
$ slabtop -o
Active / Total Objects (% used) : 280524292 / 280718042 (99.9%)
Active / Total Slabs (% used) : 6648104 / 6648104 (100.0%)
Active / Total Caches (% used) : 106 / 145 (73.1%)
Active / Total Size (% used) : 52133002.99K / 52158079.36K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.19K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
276062010 276061880 99% 0.19K 6572905 42 52583240K dentry
3012800 2962891 98% 0.06K 47075 64 188300K kmalloc-rcl-64
537147 427803 79% 0.10K 13773 39 55092K buffer_head
214710 213860 99% 0.02K 1263 170 5052K avtab_node
140288 138299 98% 0.01K 274 512 1096K kmalloc-8
107776 107008 99% 0.02K 421 256 1684K kmalloc-16
97024 93353 96% 0.03K 758 128 3032K kmalloc-32
65688 61376 93% 0.04K 644 102 2576K Acpi-Namespace
40128 38686 96% 0.06K 627 64 2508K kmalloc-64
34304 33033 96% 0.03K 268 128 1072K avc_xperms_data
28140 28140 100% 0.13K 938 30 3752K kernfs_node_cache
27264 26871 98% 0.06K 426 64 1704K ebitmap_node
24300 22497 92% 0.88K 675 36 21600K xfs_inode
23040 23040 100% 0.02K 90 256 360K selinux_file_security
22865 22865 100% 0.05K 269 85 1076K ftrace_event_field
21616 17768 82% 0.57K 772 28 12352K radix_tree_node
$ uname -r
4.20.8-1.el7.elrepo.x86_64