free shows more memory used than top process total

Question

UPDATE:

This was being caused due to a known issue with nss-softkn. See this article: https://www.splyt.com/blog/2014-05-16-optimizing-aws-nss-softoken

---

When I run free -m from my centos 6.5 box I am seeing that I only have around 1400 mb of available ram, including the cache. When I do top, sort by memory and add up the processes I am only seeing around 1600 being used. I should have a LOT more free memory. This is happening on a couple of our boxes.

[root@db1 ~]# free -m
              total      used       free     shared    buffers     cached
Mem:          7840       7793         47          0        287       1357
-/+ buffers/cache:       6148       1692
Swap:         7983          7       7976

top output

Here is a better way to visualize the memory consumed by the processes:

Answer 1

You have about 1692m in free memory according to Linux.

In the top "Mem" line it would seem like your memory is almost 99% (7793m / 7840m) but in reality you're only using about 78% of available memory. See also http://www.linuxatemyram.com/ for a more in-depth explanation.

---

Free vs Top vs /proc/meminfo

Process memory.
Adding up the memory used by processes in top and trying to match this with the used memory of free is going to give you a bad time. Due to top paging, you may not even get all the processes in this list a better way to go is to use ps aux --sort -rss to order processes by memory use.

Disk caching.
Linux is going to use the memory you aren't using to make things run much faster for you behind the scenes. This is noted in free's "cached" column and is why you should look at the values next to "-/+ buffers/cache" for actual used and free memory.

Kernel Slab cache.
Kernel modules and drivers have a way to use your unused memory too. The "SReclaimable" is memory that you could use under memory pressure. There's a cool tool called slabtop that can identify how the kernel is this cache.

Clearing caches.

echo 1 > /proc/sys/vm/drop_caches To free pagecache

echo 2 > /proc/sys/vm/drop_caches To free dentries and inodes

echo 3 > /proc/sys/vm/drop_caches To free pagecache, dentries and inodes

See also

• Meaning of the buffers/cache line in the output of free
• Explanation on Resident Set Size/Virtual Size
• Slab Allocation

Answer 2

I found the issue here. I needed to look at /proc/meminfo and see what the the kernel was caching as slab space. My problem was not understanding how much "free" memory top/ps/free said I had. It was the fact that these numbers were not adding up. The "free" command would tell me I had roughly 1692mb free but my processes only totaled up to about 20% of my total memory.

This article describes my exact issues:

http://blog.logicmonitor.com/2014/10/09/more-linux-memory-free-memory-that-is-not-free-nor-cache/

Answer 3

Based on http://blog.logicmonitor.com/2014/10/09/more-linux-memory-free-memory-that-is-not-free-nor-cache, I created a script that I called free (which I placed in a path ahead of /usr/bin in my .bash_profile) that produces useful results:

#!/usr/bin/perl

$result = `/usr/bin/free`;
print "$result";

if ($result =~ /cache:\s+(\d+)/)
{
 $mem = $1;
 $slab = `cat /proc/meminfo|grep SReclaimable`; $slab =~ s/[^\d]//g;
 print "Slab                  $slab\n";
 print "Actual Used        ** ",$mem-$slab," **\n";
}

Answer 4

I believe you are misreading the Buffers/Cache - there is > 6 gigs being used as memory cache as per the +/- Buffers/Cache line. If this is needed it can be eaten into - as its not, the OS is using it to cache disk reads and similar.