4

How to build a really big ZFS filesystem?

The above question is vague in the 1st reading, so more precise.

I know how to build big ZFS volume with one computer. e.g.

  • get a good computer with ECC RAM and redundant power
  • get some SATA controllers
  • attaching a bunch of drives (e.g. 32x2TB = 64TB)
  • install 1GB memory for every TB of storage (64GB RAM)
  • install fast SSD for L2ARC
  • install freebsd with ZFS
  • tune it
  • done.

Works fine, without any problems. But, how to extend it? Yes, I can setup another server, but the question is: how to join the two separate ZFS pools from two computers into one logical filesystem? I need ONE mount-pount over the network for the whole raidz2 volume.

Or the question in another form: How to build a "homemade" petabyte ZFS storage with plain drives and standard computers?. (ZFS is a must).

Any pointer is welcomed. (I know the Blackblaze solution, but is is done with Linux and not ZFS)...

And here is this question too, 150 TB and growing, but how to grow? , but it is different from my problem.

Thanx in advance.. ;)

Community
  • 1
kobame
  • 257
  • 4
  • 10

2 Answers2

6

ZFS isn't designed to scale, or cluster, across more than one single Solaris (or other) system. It's not a distributed filesystem, so any distributed filesystem involving ZFS will have to have another layer that binds together the various mountpoints exposed by the individual systems in your cluster.

Why the ZFS insistance? Sounds like you want something more like the Hadoop filesystem.

growse
  • 7,830
  • 11
  • 72
  • 114
2

Add another server chassis to hold the drives. Pop drives and SAS expanders in it. For example the Chenbro RM91250 holds 50 drives; add two CK23601 expanders and a UEK for the expanders. Uplink that back to your main server (may need a SAS card with external ports if your server doesn't have one already). Or add a HP MDS600, might find one on eBay cheap-ish, and use it instead of the aforementioned box.

Note: I have no idea what you're using this for, so this might be totally inappropriate for your intended use. Fair warning this is just an idea, no guarantee of performance or suitability for a particular purpose.

Chris S
  • 77,337
  • 11
  • 120
  • 212