Why drop caches in Linux?

Question

In our servers we have a habit of dropping caches at midnight.

sync; echo 3 > /proc/sys/vm/drop_caches

When I run the code it seems to free up lots of RAM, but do I really need to do that. Isn't free RAM a waste?

Answer 1

You are 100% correct. It is not a good practice to free up RAM. This is likely an example of cargo cult system administration.

Answer 2

Yes, clearing cache will free RAM, but it causes the kernel to look for files on the disk rather than in the cache which can cause performance issues.

Normally the kernel will clear the cache when the available RAM is depleted. It frequently writes dirtied content to disk using pdflush.

Answer 3

The reason to drop caches like this is for benchmarking disk performance, and is the only reason it exists.

When running an I/O-intensive benchmark, you want to be sure that the various settings you try are all actually doing disk I/O, so Linux allows you to drop caches rather than do a full reboot.

To quote from the documentation:

This file is not a means to control the growth of the various kernel caches (inodes, dentries, pagecache, etc...) These objects are automatically reclaimed by the kernel when memory is needed elsewhere on the system.

Use of this file can cause performance problems. Since it discards cached objects, it may cost a significant amount of I/O and CPU to recreate the dropped objects, especially if they were under heavy use. Because of this, use outside of a testing or debugging environment is not recommended.

Answer 4

The basic idea here is probably not that bad (just very naive and misleading): There may be files being cached, that are very unlikely to be accessed in the near future, for example logfiles. These "eat up" ram, that will later have to be freed when necessary by the OS in one or another way.

Depending on your settings of swappiness, file access pattern, memory allocation pattern and many more unpredictable things, it may happen that when you don't free these caches, they will later be forced to be reused, which takes a little bit more time than allocating memory from the pool of unused memory. In the worst case the swappiness settings of linux will cause program memory to be swapped out, because linux thinks those files may be more likely to be used in the near future than the program memory.

In my environment, linux guesses quite often wrong, and at the start of most europe stock exchanges (around 0900 local time) servers will start doing things that they do only once per day, needing to swap in memory that was previously swapped out because writing logfiles, compressing them, copying them etc. was filling up cache to the point where things had to be swapped out.

But is dropping caches the solution to this problem? definetly not. What would be the solution here is to tell linux what it doesn't know: that these files will likely not be used anymore. This can be done by the writing application using things like posix_fadvise() or using a cmd line tool like vmtouch (which can also be used to look into things as well as cache files).

That way you can remove the data that is not needed anymore from the caches, and keep the stuff that should be cached, because when you drop all caches, a lot of stuff has to be reread from disk. And that at the worst possible moment: when it is needed; causing delays in your application that are noticeable and often unacceptable.

What you should have in place is a system that monitors your memory usage patterns (e.g. if something is swapping) and then analyze accordingly, and act accordingly. The solution might be to evict some big files at the end of the day using vtouch; it might also be to add more ram because the daily peak usage of the server is just that.

Answer 5

I have seen drop caches to be useful when starting up a bunch of virtual machines. Or anything else that uses Huge Pages such as some database servers.

Huge Pages in Linux often need to defrag RAM in order to find 2MB of contiguous physical RAM to put into a page. Freeing all of the file cache makes this process very easy.

But I agree with most of the other answers in that there is not a generally good reason to drop the file cache every night.

Answer 6

It is possible that this was instituted as a way to stabilize the system when there was no one with the skills or experience to actually find the problem.

Freeing resources

Dropping caches will essentially free up some resources, but this has a side effect of making the system actually work harder to do what it is trying to do. If the system is swapping (trying to read and write from a disk swap partition faster than it is actually capable) then dropping caches periodically can ease the symptom, but does nothing to cure the cause.

What is eating up memory?

You should determine what is causing a lot of memory consumption that makes dropping caches seem to work. This can be caused by any number of poorly configured or just plain wrongly utilized server processes. For instance, on one server I witnessed memory utilization max out when a Magento website reached a certain number of visitors within a 15 minute interval. This ended up being caused by Apache being configured to allow too many processes to run simultaneously. Too many processes, using a lot of memory (Magento is a beast sometimes) = swapping.

Bottom Line

Don't just assume that it is something that is necessary. Be proactive in finding out why it is there, have the guts to disable it if others suggest it is wrong, and observe the system - learn what the real problem is and fix it.

Answer 7

Linux/m68k actually has a kernel bug which causes kswapd to go crazy and eat up 100% CPU (50% if there’s some other CPU-bound task, like a Debian binary package autobuilder – vulgo buildd – running already), which can (most of the time; not always) be mitigated by running this particular command every few hours.

That being said… your server is most likely not an m68k (Atari, Amiga, Classic Macintosh, VME, Q40/Q60, Sun3) system ;-)

In this case, the person who put in the lines either followed some questionable or, at best, outdated advice, or got the idea about how RAM should be used wrong (modern thinking indeed says “free RAM is RAM wasted” and suggests caching), or “discovered” that this “fixes”[sic!] another problem elsewhere (and was too lazy to search for a proper fix).

Answer 8

I can think of one plausible reason to do this in a nightly cron job.

On a large system, it may be useful to periodically drop caches so you can remove memory fragmentation.

The kernel transparent hugepage support does a periodic sweep of memory to coalesce small pages into hugepages. Under degenerate conditions this can result in system pauses of a minute or two (my experience with this was in RHEL6; hopefully it's improved). Dropping caches may let the hugepage sweeper have some room to work with.

You might argue that this is a good reason to disable transparent hugepages; OTOH you may believe that the overall performance improvement from transparent hugepages is worth having, and worth paying the price of losing your caches once a day.

---

I've thought of another reason you would want to do it, although not in a cron job. Right before a virtualization system migrates a VM to new hardware would be a very good time for this. Less memory contents to copy to the new host. You'll eventually have to read from the storage, instead, of course, but I'd probably take that tradeoff.

I don't know if any of the virt software actually does this.

Answer 9

One reason might be the site is running some kind of monitoring, that checks the amount of free ram and sends a warning to administrators when free ram drops below a certain percentage. If that monitoring tool is dumb enough not to include cache in the free ram calculation, it might send false warnings; regularily emptying the cache could suppress these warnings while still allowing the tool to notice when "real" ram gets low.

Of course, in this kind of situation, the real solution is to modify the monitoring tool to include cache in the free ram calculation; cleaning the cache is just a workaround, and a bad one as well, because the cache will refill quickly when processes access the disk.

So even if my assumption is true, the cache-cleaning is not something that makes sense, it's rather a workaround by someone who isn't competent enough to fix the primary problem.

Answer 10

It's old, and refer to the most accepted answers/response for why not to do it, but there is one place that I've seen this as a requirement inside the guest : a hosting provider (name not mentioned) that provides very cheap VMs, but they appears to be heavily oversubscribing, and uses this to keep their systems "usable", ie. you don't have very fast IO all the time, and they clean the caches to not do things like a ballooning driver. This goes that mostly people don't use more than a fraction of the RAM allocated, thus by cleaning the caches (frequently) they keep the actual RAM usage in the guests and thus the host as low as possible. ie. more VMs less physicals, more income

Answer 11

Just to add my two cents: The system knows very well that these memory pages are caches, and will drop as much as needed when an application asks for memory.

A relevant setting is /proc/sys/vm/swappiness, which tells the kernel during new memory allocations to prefer to drop memory caches or to swap "idle" allocated memory pages.

Answer 12

The question is from 2014, but as the problem exists to this day on some hidden centos 6.8 backends, it may still be useful for someone.

https://github.com/zfsonlinux/zfs/issues/1548 describes an issue with zfs. There, disk space isn't freed for deleted files because if nfs is used on top of zfs the file's inodes aren't dropped from the kernel's inode cache.

To quote from the bug thread, behlendorf, Jan 6 2015 wrote:

The current speculation is that for some reason the NFS server is keeping a cached version of the file handle. Until the NFS server drops this file handle ZFS can't unlink this file. Some light testing has shown that dropping caches on the server will cause this reference to be dropped (like the NFS file handle) at which point the space is correctly freed. Memory pressure can also cause it to be dropped.

i.e. a nightly echo 3 > /proc/sys/vm/drop_caches is the easiest fix for that bug if you don't want to have a downtime for restructuring your zfs.

So maybe not cargo cult admining, but some pretty good debugging was the reason.

Answer 13

This may make sense on NUMA (non uniform memory access) systems, where, typically, each CPU (socket) can access all the memory transparently but its own memory can be accessed faster than other socket's memory, in association with parallel HPC applications.

Many simple parallel applications tend to do file I/O from a single process, thus leaving on exit a big fraction of memory on a single NUMA node allocated to disk cache, while on the other NUMA node the memory may be mostly free. In these situations, since the cache reclaiming process in the Linux kernel, as far as I know, is still not NUMA-aware, processes running on the NUMA node which has memory allocated to cache are forced to allocate memory on the other NUMA node, as long as there is free RAM on the other node, thus killing the performances.

However, in an HPC system, it would be wiser to clean cache before starting a new user job, not at a specific time with cron.

For non parallel applications this problem is unlikely to arise.

Answer 14

When your page cache is quite large (a lot larger than your current swap usage), and swap in and swap out happens in turns, this is when you need to drop caches. I have seen cases where memory usage increases in one of my MariaDB database servers running Ubuntu 16.04LTS, and Linux just chose to increase swap usage instead of removing unused page caches. Transparent hugepages already disabled in my system because TokuDB required it to be disabled. Anyway maybe it is not a bug, but linux still doing this behaviour is quite puzzling to me. Various sources stated that Linux would remove page cache when application requested it :

• https://www.linuxatemyram.com/
• https://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics

But the reality is not that simple. The workaround is either :

1. Execute drop cache periodically
2. Execute drop cache when needed (monitor using vmstat 1 for swapping out activities)
3. Advise linux to remove certain files from cache (like apache log files) using utility such as dd or python-fadvise. See https://unix.stackexchange.com/questions/36907/drop-a-specific-file-from-the-linux-filesystem-cache

Example dd run :

dd if=/var/log/apache2/access_log.1 iflag=nocache count=0

Example python-fadvise :

pyadvise -d /var/log/apache2/access_log.1

Answer 15

I have a desktop machine with 16GB of RAM running on PAE kernel. After a an hour or two the disk performance degrades dramatically until I drop the caches so I simply put it into cron. I don't know if this is a problem with PAE kernel or with the cache implementation being so slow if there is plenty of memory.