Doing an rm -rf on a massive directory tree takes hours

Question

We're using rsnapshot for backups. It keeps lots of snapshots of the backuped up file, but it does delete old ones. This is good. However it's taking about 7 hours to do a rm -rf on a massive directory tree. The filesystem is XFS. I'm not sure how many files are there, but it numbers in the millions probably.

Is there anyway to speed it up? Is there any command that does the same as rm -rf and doesn't take hours and hours?

I used `find . -delete -name directory` and it's much more fast than `rm -rf`. — Paolo, Dec 14 '13 at 01:13

score 40 · Accepted Answer · edited Mar 26 '17 at 00:44

40

No.

rm -rf does a recursive depth-first traversal of your filesystem, calling unlink() on every file. The two operations that cause the process to go slowly are opendir()/readdir() and unlink(). opendir() and readdir() are dependent on the number of files in the directory. unlink() is dependent on the size of the file being deleted. The only way to make this go quicker is to either reduce the size and numbers of files (which I suspect is not likely) or change the filesystem to one with better characteristics for those operations. I believe that XFS is good for unlink() on large file, but isn't so good for large directory structures. You might find that ext3+dirindex or reiserfs is quicker. I'm not sure how well JFS fares, but I'm sure there are plenty of benchmarks of different file system performance.

Edit: It seems that XFS is terrible at deleting trees, so definitely change your filesystem.

edited Mar 26 '17 at 00:44

Dej

115
4

answered Jul 28 '09 at 08:41

David Pashley

23,151
2
41
71

1

Some years ago I noticed terrible performance using reiserfs in a similar use case. – knweiss Jul 28 '09 at 08:54
1

Marvelous post! – wzzrd Jul 28 '09 at 09:04
2

It nearly just said "no" :) – David Pashley Jul 28 '09 at 11:39
2

I agree with everything here apart from your statement that unlink speed being dependant on the size of the file. unlink just removes the link to the file and does nothing to the actual contents. There should be no discernable difference between files of different size (you can test this yourself). – Kamil Kisiel Jan 28 '10 at 22:27
@KamilKisiel You are right telling `unlink` does nothing to the actual content but to perform an `unlink` system call, the file system code has nevertheless more work to do if the removed link is the last one to the file and if it is not currently open. This is of course file system dependent but there can then be a very discernible difference when the removed file is huge. – jlliagre Sep 05 '16 at 19:56
As a sidenote: XFS terribility should have been fixed 5+ years ago, and it highly depends on the mount and log options as well. New versions shouldn't cause bigger problems on delete than anything else. – grin Jun 07 '17 at 12:50

score 26 · Answer 2 · answered Jul 28 '09 at 08:45

26

As an alternative, move the directory aside, recreate it with the same name, permissions and ownership and restart any apps/services that care about that directory.

You can then "nice rm" the original directory in the background without having to worry about an extended outage.

answered Jul 28 '09 at 08:45

Greg Work

1,956
12
11

That could work, since a mv is very very quick. – Amandasaurus Jul 28 '09 at 10:15
Yup - it works well. I've used this technique many times to "fix" maildir-based mailboxes where an email client has lost it brain and left a mess on the disk. The biggest (single) directory I've fixed in this manner had around around 1.5 or 2 million files IIRC. Total downtime to the end user was ~3 minutes, most of which was waiting for the mail client and imap processes to die. – Greg Work Jul 28 '09 at 11:25

score 8 · Answer 3 · answered Sep 21 '09 at 20:52

8

Make sure you have the right mount options set for XFS.

Using -ologbufs=8,logbsize=256k with XFS will probably triple your delete performance.

answered Sep 21 '09 at 20:52

James

7,553
2
24
33

2

+1 for this tip... One should also enable lazy counters for another performance boost. – hurikhan77 Sep 21 '09 at 21:36
3

Some explanation on these settings would be helpful for future readers. – Aron Rotteveel Oct 30 '15 at 11:35

score 6 · Answer 4 · answered Aug 19 '09 at 15:02

6

It's good to use ionice for IO-intensive operations like that regardless of filesystem used.
I suggest this command:

ionice -n7 nice rm -fr dir_name

It will play nicely for background operations on server with heavy IO load.

answered Aug 19 '09 at 15:02

ash108

266
1
5

score 5 · Answer 5 · answered Jul 28 '09 at 08:36

5

If you are doing the rm at effectively at the file level then it will take a long time. This is why block based snapshots are so good:).

You could try splitting the rm into separate areas and trying to do it in parallel however I might not expect it to make any improvement. XFS is known to have issues deleting files and if that is a large part of what you do then maybe a different file system for that would be an idea.

answered Jul 28 '09 at 08:36

James

2,212
1
13
19

Block-based snapshots are not uniquely good in this case. A number of file systems---WAFL and ZFS come immediately to mind---also provide good performance for snapshot delete. They treat snapshots as first class file system objects. So rather than iterating (slowly) over millions of files to determine which blocks to free, they only have to look a the block-list associated with the snapshot. – Keith Smith Jul 30 '09 at 01:03
Hmm. I probably came off as being too contrary above. The original poster must be using Linux, and there really isn't a well-proven Linux file system that does snapshots---although btrfs and nilfs look interesting for the future. So as a practical matter, I agree---better to use block-based snapshots. – Keith Smith Jul 30 '09 at 01:30
+1 for the tip to split and parallelize the workload: xfs plays its strength on parallel workloads. – hurikhan77 Sep 21 '09 at 21:38

score 2 · Answer 6 · answered Jan 28 '10 at 21:31

I know this is old, but I thought id toss in a suggestion. You are deleting those files sequentially, executing parallel rm operations might speed things up.

http://savannah.nongnu.org/projects/parallel/ parallel can commonly be used in place of xargs

so if your deleting all the files in deltedir

find -t f deletedir | parallel -j 10 rm

That would leave you with just empty directory structures to delete.

Note: You will likely still hit the filesystem limitations as noted above.

What's advantage of using parallel over xargs? – Amandasaurus Jan 31 '10 at 22:48 — Amandasaurus, Jan 31 '10 at 22:48

score 1 · Answer 7 · answered Jul 28 '09 at 08:50

1

Would an alternative option here be to seperate the data in such a way that you can junk and rebuild the actual filesystem instead of doing the rm?

answered Jul 28 '09 at 08:50

Moo

2,225
19
23

3

I think rsnapshot uses hard-links as part of the maintaining-multiple-snapshots-efficiently feature. So if the questioner is using that feature using separate filesystems won't work (as you can't hard-link over a filesystem boundary) – David Spillett Jul 28 '09 at 10:10

score 0 · Answer 8 · answered Jul 28 '09 at 08:38

0

How about decreasing the niceness of the command? Like:

nice -20 rm -rf /path/to/dir/

answered Jul 28 '09 at 08:38

RainyRat

3,700
1
23
29

5

The bottleneck is not the scheduler, it is the file system, I would say. – Manuel Faux Jul 28 '09 at 08:41
In the unlikely event that the scheduler is the bottleneck, you'd only end up pounding the I/O subsystem harder, making the server even less usable during the rm. – David Mackintosh Jul 28 '09 at 15:44

Doing an rm -rf on a massive directory tree takes hours

8 Answers8

Linked

Related