Why is Linux reporting "free" memory strangely?

Question

This is a canonical question about how Unix operating systems report memory usage.
Similar Questions:

• Server refuses to use swap partition
• Memory Usage in LINUX

I have production server that is running Debian 6.0.6 Squeeze

#uname -a

Linux debsrv 2.6.32-5-xen-amd64 #1 SMP Sun Sep 23 13:49:30 UTC 2012 x86_64 GNU/Linux

Every day cron executes backup script as root:

#crontab -e

0 5 * * * /root/sites_backup.sh > /dev/null 2>&1

#nano /root/sites_backup.sh

#!/bin/bash
str=`date +%Y-%m-%d-%H-%M-%S`
tar pzcf /home/backups/sites/mysite-$str.tar.gz /var/sites/mysite/public_html/www
mysqldump -u mysite -pmypass mysite | gzip -9 > /home/backups/sites/mysite-$str.sql.gz
cd /home/backups/sites/
sha512sum mysite-$str* > /home/backups/sites/mysite-$str.tar.gz.DIGESTS
cd ~

Everything works perfectly, but I notice that Munin's memory graph shows increase of cache and buffers after backup.

Then I just download backup files and delete them. After deletion Munin's memory graph returns cache and buffers to the state that was before backup.

Here's Munin graph:

Externally hosted image was a dead link.

Answer 1

You are experiencing the Linux Ate My Ram issue.

Don't Panic.

This is NOT a problem.

Your system is Working As Designed.

The problem is not your OS -- the problem is your understanding of what "free" memory is.

---

Unix systems use memory for more than just running programs. Memory might be used for:

• Running programs (active/used)
• Buffering data in transit (buffers)
• Caching data recently read from/written to the disk (cache)
• Absolutely nothing (free)

What follows is a brief (and largely incomplete) tour of how modern Unix systems report RAM usage.

What is Free memory (the OS definition)?

When a Unix system is reporting RAM as Free it means "I am not using this RAM for anything".
Free RAM is effectively worthless - It is not making your system faster, it's just sitting there being "free" in case something needs it. That something could be any of the three other items I mentioned above.

What are Cache and Buffer memory?

Cache and Buffer memory are RAM the operating system is using to make your system faster.
This memory is not needed for running programs right now, so your OS is using it to hold data that it needs frequently -- for example the C library (needed by pretty much every program you run) is almost always held in cache memory, so the system doesn't have to go to the disk to find the instructions it needs to print "Hello World" on the screen.
^{It's actually a lot more complicated than that -- there's shared memory, wired memory, etc. -- but for our purposes this simple explanation is adequate.}

What is Active memory?

Active memory is part of we understand as "used" memory -- RAM that applications are using for whatever it is they do -- sorting spreadsheets, serving web pages, editing graphics, etc.
"Active" memory has been "active" recently -- the program claiming it has made use of its contents (reading or writing), and it's not considered a good candidate for swapping out.

What is Inactive memory?

Like Active memory, Inactive memory is RAM that applications are using for whatever it is they do. The difference is this memory hasn't been accessed in a while, so if push comes to shove the OS thinks it can be swapped out to disk and (with a little luck) the program claiming it won't ask for it again so it will never notice.

What is "Used" memory (the HUMAN definition)

What you and I think of as "Used" memory is, essentially, the sum of Active and Inactive memory. All the RAM currently claimed by applications for their use.
As long as you have more installed RAM than the sum of Active and Inactive memory (plus a nice safety margin of say 512-1024MB on top) you're in an OK place: Your OS probably won't be hitting swap and killing performance.

What is "Free" memory (the HUMAN definition)?

What you and I think of as "free" memory is the memory available to run programs.
This is slightly more complicated than just the "Free" figure your OS reports. When a program asks for RAM the operating system will try to get that RAM in the least disruptive way it can:

• If there is Free memory available (sitting around doing nothing) that RAM will be allocated.
• If there is no Free memory available the OS will cannibalize the Cache and Buffer space: The least-recently/least-frequently accessed stuff in the buffer pool will be tossed out, and that RAM given to the program.
• If there is no Buffer/Cache RAM to cannibalize the swapper will look at the inactive memory and pick the regions it thinks are least likely to be accessed. That data will be paged out to swap (disk), and the newly-freed RAM given to the program.
• If all the Inactive RAM has been swapped out the swapper will start putting Active RAM on disk.
  (This is about where performance usually goes to the dogs: Every time a program gets its turn on the CPU its swapped-out bits need to be brought back into RAM, which means some other program's Active memory has to be swapped out -- the high turnover in swap is called thrashing)
• If the system has swapped out everything it can (and filled up the swap partition), or if you're running a system without a swap partition, Bad Things happen. At this point one of two things will occur:
  • malloc() will fail. This is the POSIX-conforming behavior - the operating system will tell the program asking for RAM that it can't satisfy the request.
    The program can either ask for less RAM, or if it can't make do with a smaller chunk of memory it can clean up and exit. (If the program is badly written it will simply crash.)
  • If you're on a Linux box, the OOM-Killer may go on a gang-style drive-by killing spree, terminating other processes to try to free up enough RAM to meet the request.
    ^{In case you can't tell by my description here and my answer on the linked question, I think this is a terrible way to deal with the problem.}

---

Why does Free RAM go up when you delete files?

In the example from the question here you noticed that it's possible to "Free" RAM by deleting the backup file -- the explanation for that is pretty simple: Since nothing is using that file (no open file handles) and it's no longer accessible from the filesystem (unlinked) the OS knows nobody will ever access that data again, and it purges the data from the filesystem cache.
This makes the OS report more Free memory, but has no impact on system performance.

Answer 2

This is the same "problem" as from Server refuses to use swap partition and a few other similar questions on this site. ( High Memory Usage on Linux Server, Memory Usage in LINUX, Web Server Running Low in Memory, etc.)

Pay attention to the fact that the memory consumption is from cache. This means it's keeping a file in memory. Cached memory is "free" memory. Instead of leaving the block of memory empty, your OS is keeping recently read files in that space. If an application does need that memory, it will be used by the application. Until then, it stands a chance to save your from having to read a file from the disk again if it is frequently referenced.

According to this graph, your effective memory consumption hasn't changed at all for the entire duration of the graph.

Answer 3

Something else to check if the above fails:

Check the usage of the Slab cache (Slab:, SReclaimable: and SUnreclaim: in /proc/meminfo). This is a cache of in-kernel data structures, and is separate from the page cache reported by free.

If the slab cache is resposible for a large portion of your "missing memory", check /proc/slabinfo to see where it's gone. If it's dentries or inodes, you can use sync ; echo 2 > /proc/sys/vm/drop_caches to get rid of them.

You can also use the slabtop tool to show the current usage of the Slab cache in a friendly format. c will sort the list by current cache size.

From: https://stackoverflow.com/a/5467207