20

Is it possible to create a ZFS cluster? Or would you need to go the seemingly ugly (to me at least) route of ZFS with UFS formatted pools governed by GlusterFS?

The idea is to see if it is possible to expand to Multi-Petabyte distributed store which ZFS should be able to handle without a hiccup.

Any pointers, blogs, posts?

ewwhite
  • 194,921
  • 91
  • 434
  • 799
SteveMustafa
  • 419
  • 1
  • 5
  • 11

4 Answers4

5

Why yes, it is possible to build an active/passive ZFS-based cluster using shared DAS and multipath SAS.

Details at: https://github.com/ewwhite/zfs-ha/wiki

The key to this high-availability storage design is a shared SAS-attached storage enclosure, or JBOD. While it's not shared-nothing, this can be a useful way of bringing higher availability to a storage setup that can scale vertically.

To complement the shared JBOD enclosure, we need two servers (head nodes or controllers) to provide client connectivity and compute/RAM resources for the ZFS array.

The servers and JBOD enclosure should be arranged in a SAS cabling ring topology.

enter image description here

This can scale using more enclosures...

enter image description here

ewwhite
  • 194,921
  • 91
  • 434
  • 799
4

You can cluster ZFS with at least two solutions:

"Clustering is now available not only as a commercial product like the RSF-1 solution from www.high-availability.com but also as a OpenSource solution based on PaceMaker. http://www.clusterlabs.org/wiki/Main_Page"

Mike
  • 41
  • 2
  • I did not realize pacemaker worked on FreeBSD. Thank you. With that said, have you tried ZFS on Linux? – SteveMustafa Aug 15 '13 at 22:33
  • 1
    Also, now after all this time, I'd like to clarify that by clustering for ZFS, I mean infinitely expanding storage, just add boxes/disks – SteveMustafa Aug 15 '13 at 22:34
2

Maybe this is of interest to you: Sequoia’s 55PB Lustre+ZFS Filesystem

It is from Lawrence Livermore National Laboratory (LLNL). A native port of ZFS on Linux is in developement at that lab.

kschurig
  • 152
  • 3
  • I know about Lustre, but I didn't think there was a port onto FreeBSD. Interesting, this requires some more investigation on my part. Thank you for the link. Incidentally, KQ Infotech, one of the porters of ZFS to Linux are no longer actively developing it as of 2011 (http://www.osnews.com/story/24853/KQ_ZFS_Linux_No_Longer_Actively_Being_Worked_on) – SteveMustafa Feb 19 '13 at 18:16
2

Not natively. You could have a true clustered filesystem running on top of a ZFS backing store, though...

However, ZFS high-availability (active-active) clusters are available, but that's not what you meant by your question... Maybe the Nexenta Namespace Clustering feature...

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Not the HA, that unfortunately only scales to two machines. I was thinking of something that can span several dozen JBODs, perhaps across data centers, even continents :) I think a bit big.. – SteveMustafa Feb 19 '13 at 18:13
  • @SteveMustafa There's also a [Nexenta Namespace Cluster Plugin available...](http://www.nexenta.com/corp/products/what-is-openstorage/namespace-cluster-plugin) – ewwhite Feb 19 '13 at 18:28
  • Nexenta is great, but their licensing is next to useless for my purposes. I'm trying to base this completely on FLOSS. – SteveMustafa Feb 20 '13 at 18:35
  • 1
    Talk to them about your needs. Nobody pays retail! – ewwhite Feb 20 '13 at 18:36
  • Good point. I'll give them a ring and see how things go. So far, ZFS with UFS governed by Lustre seems like the most feasible way for me so far. That's the one I've yet to try and in theory, its great. – SteveMustafa Feb 21 '13 at 19:32
  • Namespace Cluster has its uses - not sure this is one of them, though. I hesitate to answer this question -- for starters, the current forerunner answers (Gluster, Lustre, etc) have been referenced, and for seconds, my immediate response is not a how-to, but a 'why?' – Nex7 Mar 22 '13 at 14:31
  • All the links are 404'd – mrtumnus Mar 24 '20 at 15:22
  • Correct. It’s been 7 years. Try this, though - https://github.com/ewwhite/zfs-ha/wiki – ewwhite Mar 24 '20 at 19:20