45

This is a canonical question about how Unix operating systems report memory usage.
Similar Questions:

I have production server that is running Debian 6.0.6 Squeeze

#uname -a

Linux debsrv 2.6.32-5-xen-amd64 #1 SMP Sun Sep 23 13:49:30 UTC 2012 x86_64 GNU/Linux

Every day cron executes backup script as root:

#crontab -e

0 5 * * * /root/sites_backup.sh > /dev/null 2>&1

#nano /root/sites_backup.sh

#!/bin/bash
str=`date +%Y-%m-%d-%H-%M-%S`
tar pzcf /home/backups/sites/mysite-$str.tar.gz /var/sites/mysite/public_html/www
mysqldump -u mysite -pmypass mysite | gzip -9 > /home/backups/sites/mysite-$str.sql.gz
cd /home/backups/sites/
sha512sum mysite-$str* > /home/backups/sites/mysite-$str.tar.gz.DIGESTS
cd ~

Everything works perfectly, but I notice that Munin's memory graph shows increase of cache and buffers after backup.

Then I just download backup files and delete them. After deletion Munin's memory graph returns cache and buffers to the state that was before backup.

Here's Munin graph:

Externally hosted image was a dead link.

3 Answers3

61

You are experiencing the Linux Ate My Ram issue.

Don't Panic.

This is NOT a problem.

Your system is Working As Designed.

The problem is not your OS -- the problem is your understanding of what "free" memory is.


Unix systems use memory for more than just running programs. Memory might be used for:

  • Running programs (active/used)
  • Buffering data in transit (buffers)
  • Caching data recently read from/written to the disk (cache)
  • Absolutely nothing (free)

What follows is a brief (and largely incomplete) tour of how modern Unix systems report RAM usage.

What is Free memory (the OS definition)?

When a Unix system is reporting RAM as Free it means "I am not using this RAM for anything".
Free RAM is effectively worthless - It is not making your system faster, it's just sitting there being "free" in case something needs it. That something could be any of the three other items I mentioned above.

What are Cache and Buffer memory?

Cache and Buffer memory are RAM the operating system is using to make your system faster.
This memory is not needed for running programs right now, so your OS is using it to hold data that it needs frequently -- for example the C library (needed by pretty much every program you run) is almost always held in cache memory, so the system doesn't have to go to the disk to find the instructions it needs to print "Hello World" on the screen.
It's actually a lot more complicated than that -- there's shared memory, wired memory, etc. -- but for our purposes this simple explanation is adequate.

What is Active memory?

Active memory is part of we understand as "used" memory -- RAM that applications are using for whatever it is they do -- sorting spreadsheets, serving web pages, editing graphics, etc.
"Active" memory has been "active" recently -- the program claiming it has made use of its contents (reading or writing), and it's not considered a good candidate for swapping out.

What is Inactive memory?

Like Active memory, Inactive memory is RAM that applications are using for whatever it is they do. The difference is this memory hasn't been accessed in a while, so if push comes to shove the OS thinks it can be swapped out to disk and (with a little luck) the program claiming it won't ask for it again so it will never notice.

What is "Used" memory (the HUMAN definition)

What you and I think of as "Used" memory is, essentially, the sum of Active and Inactive memory. All the RAM currently claimed by applications for their use.
As long as you have more installed RAM than the sum of Active and Inactive memory (plus a nice safety margin of say 512-1024MB on top) you're in an OK place: Your OS probably won't be hitting swap and killing performance.

What is "Free" memory (the HUMAN definition)?

What you and I think of as "free" memory is the memory available to run programs.
This is slightly more complicated than just the "Free" figure your OS reports. When a program asks for RAM the operating system will try to get that RAM in the least disruptive way it can:

  • If there is Free memory available (sitting around doing nothing) that RAM will be allocated.
  • If there is no Free memory available the OS will cannibalize the Cache and Buffer space: The least-recently/least-frequently accessed stuff in the buffer pool will be tossed out, and that RAM given to the program.
  • If there is no Buffer/Cache RAM to cannibalize the swapper will look at the inactive memory and pick the regions it thinks are least likely to be accessed. That data will be paged out to swap (disk), and the newly-freed RAM given to the program.
  • If all the Inactive RAM has been swapped out the swapper will start putting Active RAM on disk.
    (This is about where performance usually goes to the dogs: Every time a program gets its turn on the CPU its swapped-out bits need to be brought back into RAM, which means some other program's Active memory has to be swapped out -- the high turnover in swap is called thrashing)
  • If the system has swapped out everything it can (and filled up the swap partition), or if you're running a system without a swap partition, Bad Things happen. At this point one of two things will occur:
    • malloc() will fail. This is the POSIX-conforming behavior - the operating system will tell the program asking for RAM that it can't satisfy the request.
      The program can either ask for less RAM, or if it can't make do with a smaller chunk of memory it can clean up and exit. (If the program is badly written it will simply crash.)
    • If you're on a Linux box, the OOM-Killer may go on a gang-style drive-by killing spree, terminating other processes to try to free up enough RAM to meet the request.
      In case you can't tell by my description here and my answer on the linked question, I think this is a terrible way to deal with the problem.

Why does Free RAM go up when you delete files?

In the example from the question here you noticed that it's possible to "Free" RAM by deleting the backup file -- the explanation for that is pretty simple: Since nothing is using that file (no open file handles) and it's no longer accessible from the filesystem (unlinked) the OS knows nobody will ever access that data again, and it purges the data from the filesystem cache.
This makes the OS report more Free memory, but has no impact on system performance.

voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • Now it's perfectly clear for me... My Linux experience is somewhat 2 months, it all started with this Debian server. One month ago I've installed Gentoo on my home laptop, and was so amazed by it's performance, that completely changed from Windows user to Linux user even on my work computer. It's great to learn Linux with Gentoo, and still there's a hell of a lot to learn. –  Nov 16 '12 at 05:16
  • @stan31337 You're not the only one it's unclear for - it's especially hard for people coming from a Windows background because unless you drill into it Windows doesn't report memory usage with this level of detail - it's just "Used" (by programs) and "Free" (for programs to take). Mac users have it a *little* easier because Activity Monitor is basically a graphical version of `top` and reports Free/Wired/Active/Inactive RAM. – voretaq7 Nov 16 '12 at 05:27
  • I used to work with **Process Explorer** on Windows, it had a lot of Memory things to show, but most of them were so unclear to me, that I actually didn't look at them... –  Nov 16 '12 at 06:30
  • 2
    Also, it should be noted, that while Windows doesn't usually *show it* in that much detail, it has a *very* similar setup. – Joachim Sauer Nov 16 '12 at 06:34
  • You know, even in Windows the memory readout is confusing. This is something that OS makers might want to put more emphasis on, or rather something that users should care less about than they do currently. – gparent Nov 17 '12 at 19:49
  • @gparent I'm not sure the average user cares *enough* -- it's a no-win scenario for OS makers: If you report "Free, Used" power users & sysadmins are miserable because we can't tune effectively. If you report Free, Wired, Active, Inactive, Cache, Buffers" ordinary users are miserable because they have "no free RAM". Since consumer operating systems don't have manuals anymore it seems to fall to the internet to educate the masses. (Thanks Commercial OS Vendors! We ***LOVE*** being your tech support!) – voretaq7 Nov 17 '12 at 21:14
  • I just think OSes should show "Free" and "Used", where "Free" means anything that can be freed if needed to be used by "real" programs. Rather than something that can be misinterpreted by users. Power users will always write tools to get what info they want, anyway. – gparent Nov 18 '12 at 01:38
  • @gparent: Then people will wonder why their OS is wasting so much of their precious RAM. They'll also think they'll get no benefit from adding more RAM to the system, since it would just be free anyway. – David Schwartz Dec 28 '12 at 20:04
27

This is the same "problem" as from Server refuses to use swap partition and a few other similar questions on this site. ( High Memory Usage on Linux Server, Memory Usage in LINUX, Web Server Running Low in Memory, etc.)

Pay attention to the fact that the memory consumption is from cache. This means it's keeping a file in memory. Cached memory is "free" memory. Instead of leaving the block of memory empty, your OS is keeping recently read files in that space. If an application does need that memory, it will be used by the application. Until then, it stands a chance to save your from having to read a file from the disk again if it is frequently referenced.

According to this graph, your effective memory consumption hasn't changed at all for the entire duration of the graph.

Jeff Ferland
  • 20,239
  • 2
  • 61
  • 85
  • Is it possible to instruct server not to cache these files? I guess I should run `sync; echo 3 > /proc/sys/vm/drop_caches` after backup? –  Nov 16 '12 at 04:49
  • 13
    @stan31337 Yes, it is possible, no you **should not** do that. Having this file in memory costs you nothing. If you clear all your caches, every file that is cached and has to be read again from disk will just slow your system down. Let it do what it was designed to do. – Jeff Ferland Nov 16 '12 at 05:03
  • 2
    @stan31337 When you delete your backup file the OS automatically kicks it out of the cache (that's why your free memory jumps again after you delete it) -- Linux is smart enough to know that a file that isn't open and can't be reached from the filesystem tree will never get accessed again. Like Jeff said you definitely don't want to dump your whole FS cache (the system will just have to re-read and re-cache that data from disk, which will actually make your server slower for a while) – voretaq7 Nov 16 '12 at 05:17
2

Something else to check if the above fails:

Check the usage of the Slab cache (Slab:, SReclaimable: and SUnreclaim: in /proc/meminfo). This is a cache of in-kernel data structures, and is separate from the page cache reported by free.

If the slab cache is resposible for a large portion of your "missing memory", check /proc/slabinfo to see where it's gone. If it's dentries or inodes, you can use sync ; echo 2 > /proc/sys/vm/drop_caches to get rid of them.

You can also use the slabtop tool to show the current usage of the Slab cache in a friendly format. c will sort the list by current cache size.

From: https://stackoverflow.com/a/5467207

Chris S
  • 77,337
  • 11
  • 120
  • 212