12

Short version: rm -rf mydir, with mydir (recursively) containing 2.5 million files, takes about 12 hours on a mostly idle machine.

More information: Most of the files being deleted are hard links to files in other directories (the directory being deleted is actually the oldest backup made by rsnapshot; the rm command is actually given by rsnapshot). So it's mostly directory entries being deleted - the file content itself isn't much; it's in the order of some tens of GB.

I'm far from certain that btrfs is the culprit. I recall backup was also very slow before I started to use btrfs, but I'm not certain that the slowness was in the deletion.

The machine is an Intel Core i5 2.67 GHz with 4 GB RAM. It has two SATA disks: one has the OS and some other stuff, and the backup disk is a 1 TB WDC WD1002FAEX-00Z3A0. The motherboard is an Asus P7P55D.

Edit: The machine is a Debian wheezy with Linux 3.16.3-2~bpo70+1. This is how the filesystem is mounted:

root@thames:~# mount|grep rsnapshot
/dev/sdb1 on /var/backups/rsnapshot type btrfs (rw,relatime,compress=zlib,space_cache)

Edit: Using rsync -a --delete /some/empty/dir mydir takes about 6 hours. A significant improvement over rm -rf, but still too much I think. (Explanation of why rsync is faster than rm: "[M]ost filesystems store their directory structures in a btree format, the order [in] which you delete files is ... important. One needs to avoid rebalancing the btree when you perform the unlink.... rsync -a --delete ... does deletions in-order")

Edit: I attached another disk which had 2.2 million files (recursively) in a directory, but on XFS. Here are some comparative results:

                  On the XFS disk      On the BTRFS disk
Cached reads[1]       10 GB/s               10 GB/s
Buffered reads[1]     80 MB/s              115 MB/s
Walk tree[2]         11 minutes            43 minutes
rm -rf mydir[3]       7 minutes            12 hours

[1] With hdparm -T /dev/sdX and hdparm -t /dev/sdX.
[2] Time taken to run find mydir -print|wc -l immediately after boot.
[3] On the XFS disk, this was soon after walking the tree with find. On the BTRFS disk it is the old measurement (and I don't think it was with the tree cached).

It appears to be a problem with btrfs.

Antonis Christofides
  • 2,556
  • 2
  • 22
  • 35
  • 1
    2.5 million files in a single directory? I'm not aware of a filesystem which handles this well. – Michael Hampton Feb 11 '15 at 20:55
  • @MichaelHampton: It's not flat, it contains nested directories. I added the word "recursively" in the short description; I hope this clarifies it. – Antonis Christofides Feb 11 '15 at 20:57
  • 1
    Why are you using the copy-on-write directory trick on a copy-on-write filesystem? – symcbean Feb 11 '15 at 20:58
  • @symcbean: You mean that the hard link trick is redundant on `btrfs`? This is possible, of course, but do you think it may be relevant? Right now I can't recall why I decided to try `btrfs`. – Antonis Christofides Feb 11 '15 at 21:04
  • May be you should try zfs. I once had millions of files in one directory, can't remember I had any troubles deleting them. Though it took some time indeed. – drookie Feb 12 '15 at 05:32
  • 2
    Ah, I remember now. I decided to switch to `btrfs` because I wanted the transparent compression. Now: `rsnapshot` uses hard links. It does not have any option to not use hard links. So the hard links overlap with `btrfs`'s copy-on-write functionality, but I can't do much about that. – Antonis Christofides Feb 12 '15 at 14:02

3 Answers3

3

Well this is still an Btrfs issue, that's well known that deleting many small files does take quite long time compared to other file systems.

If you dislike it, you can either wait until upstream has fixed it or move on to another file system which does it better.

Your main error though is using an ancient kernel (3.16, yes it was already ancient when you posted) with btrfs. Btrfs is a file system which is still under heavy development, so you should always stay with the latest and greatest kernel version to get in touch with the improvements. If your distribution does not do backports, you can either do that yourself or you are screwed.

Btrfs got many performance improvements in kernel version 3.19 - this is the minimum version you should use in production, your kernel version 3.16 plainly sucks without backports.

Also take in mind that according to Chris Mason he does consider Btrfs stable by now, but not production ready yet.

Marc Stürmer
  • 1,894
  • 12
  • 15
  • 1
    How do you define "well-known"? I had searched the web extensively and in vain, and no-one from those who participated in this discussion knew about it. But, anyway, I'm now just staying away from `btrfs`. Too hyped while its development seems to be taking forever. – Antonis Christofides Dec 31 '15 at 09:10
  • 1
    Well, there are for example the people from CoreOS. They used roughly Btrfs one year as the default file system until the beginning of 2015 where they switched back then to Ext4+Overlayfs. Keep in mind though that this was before kernel version 3.19, which brought a lot of improvements for Btrfs. Also take a look at this presentation of October 2015, which take a look at ext4, xfs, zfs and btrfs on database work load conditions, namely Postgres: http://de.slideshare.net/fuzzycz/postgresql-on-ext4-xfs-btrfs-and-zfs-54525451 Another benchmark, though not so good kernel: http://goo.gl/rR3kZ2 – Marc Stürmer Jan 04 '16 at 00:28
  • And like I told, the kernel version of your box (3.16) is known to be plagued by performance issues, at least use 3.19 for serious Btrfs stuff according to Chris Mason. If you want to use Btrfs seriously, always use the latest and greatest kernel - something that doesn't work really well with Debian... and search term "btrfs metadata performance." – Marc Stürmer Jan 04 '16 at 07:41
2

I'm a bit late to this party, but here's a trick to very quickly delete extremely large btrfs trees:

  1. Create a dummy subvolume on the same btrfs filesystem.
  2. Move the top level directory you want to remove into said subvolume - this operation should be really quick if you're doing it on the same btrfs filesystem, even across subvolumes.
  3. Destroy the subvolume.

The kernel is going to start reclaiming space in the background, so you won't have the available space quite immediately, but the process should be way faster than doing any sort of user-land deletion.

Nicolas Noble
  • 121
  • 1
0

You could rename the directory and then delete the renamed directory in a background process. This isn't going to speed up the delete operation. However, this would allow the program to continue forward with an empty directory while the delete operation is happening on the side.

I'm not sure if this is going to work in your use case. It depends if the program can't continue until the disk is idle (i.e. it is going to do some heavy disk operations). It depends if the program is going to fill up the disk with a lot of data.

Nathan
  • 157
  • 7