2

(DISCLAIMER: This is "time bound" as probably every other question, I just point it out now instead of implicating it)

part of reviewing a few basic decisions for our infrastructure is once again the topic of volume/storage management. I'm mainly talking about about local storage here (or block devices that cannot be handled by the providing servers intelligently).

Scraping the web, blogs, communities it seems to me that BTRFS is basically dead, the promised land still hasn't arrived and what is even worse I can't see any overall progress in the project.

ZFS seems to have a community that is substantially larger but the stability on linux is still something I can't quite trust.

LVM is well known by most people but has it's own set of prolems (snapshots, external handling of RAID, ...)

Ceph is the new kid but isn't really storage management without vast amounts of glue. Also the system is quite a beast in terms of complexity and the most intriguing selling point (CephFS) is still marked as "not ready for production" -- but hey someone got to be the first to try :)

GlusterFS seems nice but we had problems withsplit brain situations so our trust level is quite low -- given

So far for my status on the topic. The question is:

For the next couple of years, if you choose today, what kind of storage management will you use if the constraint is Linux?

My personal preference of importance is this:

  • Strong Community support
  • "Easy" to work with (standard tasks should be easy to accomplish after a relatively short period of getting used to it)
  • Widely supported by different distros (at least.deb and .rpm available)
  • native kernel support (or otherwise low maintenance for on-site administrators)

(It's been a long time since my last post here so please redirect me to a better place)

Martin M.
  • 6,428
  • 2
  • 24
  • 42
  • 2
    I think this question is a little too subjective and you're trying to compare apples with toothpaste. lvm/zfs and btrfs are so different to gluster & ceph that you can't really compare them. – hookenz Oct 03 '13 at 22:38
  • You haven't told us what data you're storing.... – fukawi2 Oct 03 '13 at 23:11
  • As I said it's a basic decision so candidates are: Fileservers, DB Servers, "Spooling Areas" (short term file storage, range: a couple of hours). I strongly prefer stability, correctness and ease of of use over speed – Martin M. Oct 04 '13 at 15:30
  • uh oh... "opinion-based" ... how do I rephrase that so it fits here? (after all isn't any answer opinion based, if this would be a data based decision I wouldn't have to ask :)) – Martin M. Oct 04 '13 at 15:32

1 Answers1

3

I'm primarily using ZFS on Linux for my production storage needs, in several forms.

I use RHEL/CentOS-like systems for the most part, and spent a decade equipping servers with XFS as the main filesytem. In November 2012, there was a change to the XFS shipped in the Red Hat kernel that made new XFS installations unsuitable for my needs. That was around the same time that the ZFS on Linux project improved its delivery/installation process..

I took a leap of faith and used the availability of native Linux ZFS as an opportunity to run it under production workloads. I'm approaching a year in this setup and have been completely happy with the decision. I have ~40 Linux server in varying capacities that use ZFS data volumes.

The main benefits...

  • Compression.
  • L2ARC read caching has helped accelerate some of my applications. The ZFS caching mechanism is far more intelligent than the Linux VM subsystem.
  • ZFS is a slicker volume manager than LVM and offers more flexibility.
  • The data integrity can't be beat...
  • I've done many stupid things with ZFS and still haven't managed to lose data...

A few downsides...

  • I have a few esoteric ZFS configurations that required digging into the source code or relying on the developer community to fully understand/optimize. Example: ZFS on Linux on Fusion-io, or ZFS zvols on hardware RAID with XFS on top.
  • Some of the ZFS best practices in Solaris/Illumos/Nexenta-land don't translate to Linux or are counterintuitive (zvol tuning, for instance).

As for the others...

  • LVM is in wide use, but not a direct competitor to ZFS.
  • BtrFS never hit its stride.
  • Ceph has a place and definite support in certain circles. I probably wouldn't build on it as a direct-attach storage platform, though.
  • Gluster has a community and some mindshare, too. But again, it's for a narrow set of applications.
Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799