2

I have a server that has 2GB ram and is running on MediaTemple's DV managed server, which uses CentOS. The command free -m gives this following: 

             total       used       free     shared    buffers     cached
Mem:          2048       1948         99          0          0        140
-/+ buffers/cache:       1808        239
Swap:         1536        288       1247

But when I try to figure out which processes are using the memory using top, I get this:

1417 mysql     20   0  496m 496m 7512 S  1.3 24.3  42:10.92 mysqld                                                                                                                                         
 3259 okgo      20   0 49436  48m  11m S  0.0  2.4   0:21.96 php-cgi                                                                                                                                        
 3175 root      20   0 49144  47m 3196 S  0.0  2.3   0:01.80 spamd                                                                                                                                          
 3176 popuser   20   0 46940  45m  992 S  0.0  2.2   0:00.00 spamd                                                                                                                                          
 3269 okgo      20   0 44372  43m  10m S 19.2  2.1   0:02.11 php-cgi                                                                                                                                        
 3272 okgo_for  20   0 23808  23m  10m S  0.0  1.1   0:00.79 php-cgi                                                                                                                                        
27161 root      20   0 10816  10m 9460 S  0.0  0.5   0:01.97 httpd                                                                                                                                          
  786 root      20   0  4844 4844 2140 S  0.0  0.2   2:28.96 newrelic-daemon                                                                                                                                
 1017 apache    20   0  4552 4552 2228 S  0.0  0.2   0:01.38 httpd                                                                                                                                          
 3277 postfix   20   0  4432 4432 2548 S  0.0  0.2   0:00.00 trivial-rewrite

What is confusing me here is the fact that the free -m command seems to show that this memory is not cached, i.e. the -/+ buffers/cache line also shows 1827 used! What am I missing here?

Edit

Upon request, here is the output of /proc/meminfo (which seems to contradict free -m ?):

MemTotal:        2097152 kB
MemFree:           55808 kB
Cached:           158656 kB
Buffers:               0 kB
Active:           368924 kB
Inactive:         428580 kB
Active(anon):     273480 kB
Inactive(anon):   365368 kB
Active(file):      95444 kB
Inactive(file):    63212 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1572864 kB
SwapFree:        1277160 kB
Dirty:                68 kB
Writeback:             0 kB
AnonPages:        638848 kB
Shmem:              4424 kB
Slab:            1243780 kB
SReclaimable:    1222584 kB
SUnreclaim:        21196 kB

Also, this post may shed light on the situation, but the /proc/meminfo here suggest that there is a very large percentage of memory used, which is not being listed in top.

fraxture
  • 151
  • 1
  • 5
  • It looks like the output from `free` got mangled in some way. Could you try to paste it again? Maybe include the contents of `/proc/meminfo` as well. – kasperd Apr 07 '14 at 09:19

1 Answers1

2

Slab is memory used by kernel and it will not show up in top. It is about 1,2GB at this time and that correctly adds to your top output to the one that is given by free. You can inspect in more details where it is used using slabtop command.

P.S. One of most common scenarios when a lot slab memory is useed when system has a lot files and reads/writes them quite much. Then a lot memory goes to cache inode information. This caching can be tuned a bit, but at the price of performance.

Kazimieras Aliulis
  • 2,324
  • 2
  • 26
  • 45
  • Is there any way to try to trace what might be causing all the read/write, or at least verify that it is occurring? – fraxture Apr 07 '14 at 10:59
  • First check slabtop where this ram is used. Slab has a lot different usage and the example was only one, that I have personally encountered. And it is mainly connected not with reads and writes, but with number of files in disks. – Kazimieras Aliulis Apr 07 '14 at 11:04
  • 1
    So post at least slabtop output and df -i. – Kazimieras Aliulis Apr 07 '14 at 11:05
  • This will sort by cache size: slabtop -o -s c – Kazimieras Aliulis Apr 07 '14 at 11:07
  • Strange, when I try to run slabtop I get the following error: `fopen /proc/slabinfo: No such file or directory` (tried running it as root and normal user). – fraxture Apr 07 '14 at 11:21
  • Thats probably kernel or virtualization related. This makes debuging harder, but still how much files do you have? For example my server with about 500000 files uses about 300MB of slab for ext4_inode_cache and dentry cache. – Kazimieras Aliulis Apr 07 '14 at 11:50
  • You can increase sysctl parameter vm.vfs_cache_pressure over 100 to increase slab cache recovery, but that may introduce a big performance hit. – Kazimieras Aliulis Apr 07 '14 at 11:51
  • I do not know how many files I have on my server. Is there a way to determine this? – fraxture Apr 07 '14 at 12:56
  • *df -i* - will show number of inodes (file and folder metadata structures) used. – Kazimieras Aliulis Apr 07 '14 at 13:16
  • Hmm, okay well i get five different lines for five different file systems when I run that at `/`. The first three are mounted at `/`, `/tmp`, and `/var/tmp` respectively, and they all have 3188640 Inodes of which the IUse% is 6%. The other two files systems are mounted at `/dev/` and `/dev/shm` and have 262144 Inodes at 1% use. – fraxture Apr 07 '14 at 13:22
  • Its quite low. But without /proc/slabinfo have no idea, what for slab is using its cache. – Kazimieras Aliulis Apr 07 '14 at 13:25
  • I am sorry, I did not understand the last part of your question. Could you perhaps rephrase. Thanks so much for your help. – fraxture Apr 07 '14 at 13:37
  • That means: can not help without /proc/slabinfo :) – Kazimieras Aliulis Apr 08 '14 at 10:36