Enormously large in-use dentry cache

0

1

We have an EC2 instance that has a very large (50+GB) dentry cache and most of it is in use (99%). "drop_cache" commands (sysctl -w vm.drop_caches=2 or sysctl -w vm.drop_caches=3) don't help much, because almost all of it is in use. Is there any way to find out what's in this large in-use dentry cache or pin down what's causing it?

$ cat /proc/meminfo
MemTotal:       64461192 kB
MemFree:         8585000 kB
MemAvailable:   62782360 kB
Buffers:          194872 kB
Cached:          1956268 kB
SwapCached:            0 kB
Active:          1345168 kB
Inactive:        1161952 kB
Active(anon):     432384 kB
Inactive(anon):    55304 kB
Active(file):     912784 kB
Inactive(file):  1106648 kB
Unevictable:      180872 kB
Mlocked:          180872 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               660 kB
Writeback:             0 kB
AnonPages:        536000 kB
Mapped:           241356 kB
Shmem:             58028 kB
KReclaimable:   52894088 kB
Slab:           52981812 kB
SReclaimable:   52894088 kB
SUnreclaim:        87724 kB
KernelStack:        7888 kB
PageTables:         9700 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    32230596 kB
Committed_AS:    3014752 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             3520 kB
HardwareCorrupted:     0 kB
AnonHugePages:    274432 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1101796 kB
DirectMap2M:    35117056 kB
DirectMap1G:    29360128 kB
$ slabtop -o
 Active / Total Objects (% used)    : 280524292 / 280718042 (99.9%)
 Active / Total Slabs (% used)      : 6648104 / 6648104 (100.0%)
 Active / Total Caches (% used)     : 106 / 145 (73.1%)
 Active / Total Size (% used)       : 52133002.99K / 52158079.36K (100.0%)
 Minimum / Average / Maximum Object : 0.01K / 0.19K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
276062010 276061880  99%    0.19K 6572905       42  52583240K dentry                 
3012800 2962891  98%    0.06K  47075       64    188300K kmalloc-rcl-64         
537147 427803  79%    0.10K  13773       39     55092K buffer_head            
214710 213860  99%    0.02K   1263      170      5052K avtab_node             
140288 138299  98%    0.01K    274      512      1096K kmalloc-8              
107776 107008  99%    0.02K    421      256      1684K kmalloc-16             
 97024  93353  96%    0.03K    758      128      3032K kmalloc-32             
 65688  61376  93%    0.04K    644      102      2576K Acpi-Namespace         
 40128  38686  96%    0.06K    627       64      2508K kmalloc-64             
 34304  33033  96%    0.03K    268      128      1072K avc_xperms_data        
 28140  28140 100%    0.13K    938       30      3752K kernfs_node_cache      
 27264  26871  98%    0.06K    426       64      1704K ebitmap_node           
 24300  22497  92%    0.88K    675       36     21600K xfs_inode              
 23040  23040 100%    0.02K     90      256       360K selinux_file_security  
 22865  22865 100%    0.05K    269       85      1076K ftrace_event_field     
 21616  17768  82%    0.57K    772       28     12352K radix_tree_node
$ uname -r
4.20.8-1.el7.elrepo.x86_64

ssppjj

Posted 2019-08-19T20:54:44.827

Reputation: 21

No answers