8

We have a very large (multi-GB) Nginx cache directory for a busy site, which we occasionally need to clear all at once. I've solved this in the past by moving the cache folder to a new path, making a new cache folder at the old path, and then rm -rfing the old cache folder.

Lately, however, when I need to clear the cache on a busy morning, the I/O from rm -rf is starving my server processes of disk access, as both Nginx and the server it fronts for are read-intensive. I can watch the load average climb while the CPUs sit idle and rm -rf takes 98-99% of Disk IO in iotop.

I've tried ionice -c 3 when invoking rm, but it seems to have no appreciable effect on the observed behavior.

Is there any way to tame rm -rf to share the disk more? Do I need to use a different technique that will take its cues from ionice?

Update:

The filesystem in question is an AWS EC2 instance store (the primary disk is EBS). The /etc/fstab entry looks like this:

/dev/xvdb       /mnt    auto    defaults,nobootwait,comment=cloudconfig 0       2
David Eyk
  • 667
  • 1
  • 7
  • 17
  • You should probably also mention the filesystem that you're using and how (mount options). – Cristian Ciupitu Oct 15 '13 at 17:36
  • Updated. Also, in case it matters, this is on Ubuntu 12.04. – David Eyk Oct 15 '13 at 18:24
  • Note that IO performance on Amazon EBS can be pretty bad. See http://perfcap.blogspot.com/2011/03/understanding-and-using-amazon-ebs.html which recommends a long-term maximum of 100 iops, with short-term (1 minute) bursts up to 1000. It sounds like your case is way higher than that in a minute, hence the problem. – Moshe Katz Oct 18 '13 at 00:14
  • Right, that's why we're using an instance store, not EBS, for the cache. See my update comment. Sorry if that wasn't clear. – David Eyk Oct 18 '13 at 13:32
  • Sorry I'm late, but you could investigate cgroups and the blkio controller: https://www.kernel.org/doc/Documentation/cgroups/blkio-controller.txt – AndreasM Oct 22 '13 at 12:15
  • That's tantalizing, and I'm sure there's something there, but those are depths to which I'm not willing to descend right now. :) Can you translate that into bash and/or fstab? – David Eyk Oct 22 '13 at 16:50

3 Answers3

9

Removing files performs only metadata operations on the filesystem, which aren't influenced by ionice.

The simplest way would be, if you don't need the diskspace right now, to perform the rm during off-peak hours.

The more complex way that MIGHT work is to spread the deletes out over time. You can try something like the following (note that it assumes your paths and file names DO NOT contain any spaces!):

while find dir -type f | head -n 100 | xargs rm; do sleep 2; done
while find dir -type d -depth | head -n 100 | xargs rmdir; do sleep 2; done

Also note that you can't use rm -f in the first command because then the loop wouldn't stop (it depends on the error exit code of rm when there is no argument).

You can tweak it by modifying the number of deletes per cycle (100 in the example) and the sleep duration. It might not really work however since the filesystem might still bunch up the metadata updates in a way that you get into trouble with your IO load. You just have to try.

aferber
  • 361
  • 1
  • 2
  • Removing that many files takes a long time, so there's really no "off-peak" period that will encompass it. :( – David Eyk Oct 15 '13 at 18:25
  • The `while` loop seems to do the trick when `head -n 50`. 100 was still slowly raising the load average above critical, which tells me too much resource contention was going on. – David Eyk Oct 15 '13 at 18:46
  • Man, that takes a long time to run! – David Eyk Oct 16 '13 at 14:09
  • The find is still going to list all files in the directory and all subdirectories for every iteration of the while loop. You could probably do better with something like – Randy Orrison Oct 18 '13 at 13:01
  • 1
    The find is still going to list all files in the directory and all subdirectories for every iteration of the while loop. You could probably do better with something like find dir -type f -print0 | xargs -l50 -0 rmwait where rmwait is a script that does rm "$@"; sleep 2. Note the use of -print0 and -0 to handle filenames with spaces. -l50 tells xargs to only do 50 at a time. – Randy Orrison Oct 18 '13 at 13:10
3

All data gathered from this page. Below are some options to delete large directory of files. Check out the writeup for the details of how this was produced.

Command                                 Elapsed System Time %CPU cs1* (Vol/Invol)
rsync -a –delete empty/ a                10.60      1.31    95%  106/22
find b/ -type f -delete                  28.51      14.46   52%  14849/11
find c/ -type f | xargs -L 100 rm        41.69      20.60   54%  37048/15074
find d/ -type f | xargs -L 100 -P 100 rm 34.32      27.82   89%  929897/21720
rm -rf f                                 31.29      14.80   47%  15134/11

*cs1 is context switches voluntary and involuntary

Back2Basics
  • 160
  • 1
  • 7
  • Whilst this may theoretically answer the question, [it would be preferable](http://meta.stackexchange.com/q/8259) to include the essential parts of the answer here, and provide the link for reference. – Tom O'Connor Oct 22 '13 at 20:20
  • Fascinating! I'll try it. – David Eyk Oct 23 '13 at 14:50
  • `rsync` is running right now. Perhaps it's too early to tell, and it could be helped that I'm not running it in the middle of a busy morning, but the server is still responsive and the load average is manageable. – David Eyk Oct 23 '13 at 21:18
  • The exact invocation I'm using: `ionice -c 3 nice -19 rsync -a --delete /mnt/empty/ /mnt/nginx-cache-old` – David Eyk Oct 23 '13 at 21:19
  • Well, that only took 4 hours. ;) I'm going to accept this answer (sorry @aferber) as I like the straightforward invocation and it appears to be susceptible to `nice` and `ionice`, or at least it didn't destroy the server like `rm -rf` did. – David Eyk Oct 24 '13 at 13:34
-1

You can pair it with the "nice" command. ionice -c 3 nice -19 rm -rf /some/folder

This changes the priority of the process on the machine.

Back2Basics
  • 160
  • 1
  • 7
  • Unfortunately, `nice` seems to have about as much effect as `ionice`, that is, nothing appreciable. – David Eyk Oct 15 '13 at 18:29
  • @DavidEyk. If nice and ionice do not have "noticeable" effect, it either means nothing else is contending for resources in any appreciable manner, or you simply aren't noticing the effect with your naked eye. You really should benchmark it using iostat and vmstat to see the real effect. – Michael Martinez Oct 21 '13 at 23:14
  • I believe @aferber addressed this in his answer: "Removing files performs only metadata operations on the filesystem, which aren't influenced by ionice." I've seen the contention--my server processes were starving for read time while the CPU loafed and `rm -rf` had 99% on `iotop`. – David Eyk Oct 22 '13 at 03:01