Understanding metadata usage behavior in a newly converted btrfs filesystem

Question

I have a 5 TB hard drive with 3.3 TiB (as reported by du) of media files in 119k files and directories; the average file size was about 28 MiB. I converted the ext4 partition into btrfs with btrfs-convert. The process took 10.4 hours and was CPU-bound. Right after the conversion:

# btrfs fi df /mnt/btrfs/
Data, single: total=3.03TiB, used=2.21TiB
System, single: total=32.00MiB, used=236.00KiB
Metadata, single: total=1.52TiB, used=1.10TiB

btrfs allocated all space on the hard drive (3.03 + 1.52 TiB ≈ 5 TB), and it apparently put ~1 TiB of file content into metadata chunks (1.10 TiB used). This is unexpected because very few of my files would fit into the leaf nodes.

The standard solution to the huge metadata allocation is to re-balance the metadata. btrfs balance start -m took 8.0 hours and was I/O-bound. Afterwards, it appears that btrfs not only released unused metadata chunks, but also moved file content from metadata chunks into data chunks.

# btrfs fi df /mnt/btrfs/
Data, single: total=3.32TiB, used=3.31TiB
System, single: total=32.00MiB, used=104.00KiB
Metadata, single: total=3.00GiB, used=2.18GiB

Could someone explain what is happening? Why would "metadata" consume 1.10 TiB initially, and why would balancing reduce it to 2.18 GiB? Does btrfs balance move file content from metadata chunks to data chunks?

I used Linux kernel 3.13.0-46. My files have no hard links, xattrs, SELinux contexts, nor ACLs. I have not turned on compression, and have not deleted the ext4 roll back image.

Answer 1

First, the conversion process saves a copy of all the prior system metadata in the new metadata, which can take up a substantial amount of space on large drives.

Second, the conversion process is messy, resulting in extremely large extents, since the extents in EXT4 are also large, and BTRFS will inherent their size.

The allocated size becomes approximately 1.5X the size of the used metadata size. The defragmentation process will reduce the size of the used metadata, but does not change the allocation. There is also a skinny extents option to further reduce metadata, but this is more helpful on systems with large amounts of small files; your metadata allocation less than a 10th of a percent, which is very small.

The balance command IS supposed to reduce the allocation size to a new value based on the current metadata usage, which it seems to have done correctly. The balance command is not supposed to move from metadata to data, but that may have something to do with the original EXT4 metadata image copy being in the original metadata allocation, and now moved into data (ext2_saved subvolume). Check the size of the EXT4 image to see if it is 1.1TB. Regardless, I would defragment the filesystem.

It should be noted that running balance without defragmentation can result in errors. Newer versions of the kernel are recommended to prevent filesystem problems, specifically 3.17 and later.