I am considering which OS and filesystem to use on some new servers I have and am considering either Free BSD with ZFS, or Linux with BTRFS.
The programs I have run on both systems, so the only issue is reliability of the filesystems and performance, etc.
As of April 2010 Btrfs is still in development and not ready for production use (really, it isn't - I have been bitten experimenting with kernel 2.6.32). Feature-wise btrfs contains some but not all of ZFS features. Most notably RAID-Z and data deduplication are missing.
What does work with btrfs is snapshotting, copy-on-write, checksumming and using multiple disks as a single volume. GNU cp recently got a neat "reflink" function to copy files as copy-on-write duplicates.
ZFS-FUSE seems more stable than btrfs, so if you go with Linux that may be an option (http://zfs-fuse.net/). Most features of Solaris ZFS are implemented. Can't find up-to-date performance benchmarks, though.
For a rock solid solution, I'd still say go with Linux and XFS. If you need snapshotting, add LVM2. If you need Software-RAID, add mdadm.
As of today (2012-02-01) btrfs still lacks an fsck utility, meaning that all your data is at risk if your system crashes (think kernel panic, power loss, and many other scenarios). I've experienced this data loss first-hand. If you're looking for data security (and who isn't?) then avoid btrfs. As of right now BTRFS is for experimenting filesystem junkies or non-essential data.
Of course you should keep backups of valuable data anyways, right?
For an ultra-stable alternative use ZFS on PC/FreeBSD 9. ZFS is a trustworthy enterprise filesystem that's been around many years. RAIDZ2 is a life saver.
As others have mentioned, Btrfs is still experimental, so you probably don't want to rely on Btrfs just yet.
People generally seem happy with ZFS on BSD, but you might also want to consider running ZFS on Solaris.
The BSD port of ZFS is probably the most mature port, but it still lags a few versions behind ZFS on Solaris, so you won't get all the latest features right away. As of FreeBSD 8.0, only ZFS version 13 is supported--so, for example, you won't get deduplication (which was introduced in ZFS version 21). You also won't get iSCSI with FreeBSD 8.0.
If you want the latest ZFS features as they're introduced, you may want to opt for OpenSolaris or Oracle Solaris, or Nexenta (a Solaris derivative with a GNU userland).
As others have already pointed out, BTRFS is not really production quality yet, so if you need something that works today, ZFS is a better bet.
For some background on BTRFS, and how it differs from ZFS, see this LWN article.
I'm still a ZFS user (on both Solaris and Linux) however, on the latter, I would now consider btrfs as its latest (March 2012) release includes new features, performance improvements, bug fixes, and the very expected release of a btrfsfsck with repair capabilities.
According to that press release, the Btrfs file system is now production-ready.
This question was asked almost a decade ago, but it still comes up in search engines, so it's probably worth adding an updated answer. ZFS on Linux is now fairly mature: Ubuntu added basic support in 2015 and in late 2019, added installation to a ZFS root. This means it is easier to compare ZFS and btrfs on the same OS, without turning this into a Linux vs FreeBSD discussion. (If you're not on Linux, btrfs is probably not an option.)
I've been using ZFS on a basic home Linux server for the last year with a few mirrored disks, and it works well. However, the need to install the ZFS kernel module complicates things somewhat compared to using btrfs, which is built into Linux. Booting from a ZFS root is not typical, and installing ZFS makes it harder to manage the frequent kernel updates in a rolling release distribution. I am using btrfs for some of my local storage, so I thought I would try to migrate my server to btrfs as well and solve all my problems with no external kernel modules.
ZFS is overall a more mature product. Setting up your first ZFS system is surprisingly easy: it seems like there should be more configuration required, but it's a "batteries included" filesystem. ZFS provides native support for encryption, whereas btrfs requires you to encrypt your own block device using dm-crypt or similar. ZFS can even take care of exporting your filesystem over NFS or SMB. ZFS's documentation is not pitched at a kernel hacker or casual user running a single boot disk (although ZFS can do this), but assumes that the reader is building a serious file server with users, redundancy and uptime.
In contrast, while btrfs is generally stable and many people have been using it happily for years, it still contains some dangerous features like the unstable RAID56 implementation, and its documentation is difficult to locate and understand. The overwhelming list of questions on the FAQ page is typical of the btrfs documentation experience. If your needs are simple, then you can probably safely ignore many of the warnings, but it will still take you longer to reach the level of understanding required to trust the filesystem. If you're still not sure what your needs are, btrfs makes it easier to change plans later, by adding or removing devices or resizing the filesystem while it is mounted.
ZFS and btrfs use the term snapshot in very different ways, which can be confusing at the initial stage of evaluating the two systems. While the two approaches are equally powerful, you may find one or the other more intuitive:
In ZFS, block devices form a zpool which can host an arbitrary number of mountable datasets. Each dataset may have a number of snapshots, which represent efficient copies of the state of that dataset at a particular time.
In btrfs, block devices form a filesystem which contains an arbitrary number of mountable subvolumes. A snapshot of a subvolume is itself another subvolume; "a snapshot is just like a (full) copy, except it's more economical because of the shared storage." The flexibility of this approach means you need to decide on some kind of layout convention if you want to use snapshots to manage incremental backups.
If you're considering both ZFS and btrfs on Linux, and you are interested in a more bleeding edge alternative, you could also look at bcachefs.
While I was looking for the same thing I've stumbled over this piece of information from the FreeBSD wiki on the status of ZFS in FreeBSD. Looks like FreeBSD is catching up.
Source: http://wiki.freebsd.org/ZFSTuningGuide
History of FreeBSD releases with ZFS is as follows:
