8

Which is best ?

Mirroring 2 internal disks with HW raid and then putting ZFS filesystem on top

or

Using the 2 internal disks without HW Raid and creating a ZFS mirror with the 2 disks.

I've read that to get the full benifits of ZFS it needs at least 2 disks to use its error correction and prediction features fully.

Its a question of ZFS ontop of HW RAID , or just ZFS , not HW vs ZFS . Performance is not my issue.

AndyM
  • 901
  • 2
  • 14
  • 26

4 Answers4

12

ZFS works optimally if it knows as much as possible about the underlying physical disk structure. A hardware RAID 1 will present a single volume to the OS which ZFS will then treat as a single disk. You'll therefore miss out on the benefits of error correction / fault detection / checksumming etc.

There's another benefit to using ZFS over HW RAID - if your hardware dies, be it the motherboard, RAID card, CPU etc, you can pull the disks and use them in any other system that is running ZFS, and the entire filesystem is readable. If you use HW RAID, you may have a dependency on that specific RAID card, so if it fails you might need to find a similar model before you can read the disks again. You can mitigate this by having dual RAID cards, but this usually starts to become expensive and complex.

In my experience, for the majority of use-cases that I've used a mirrored disk configuration, the performance difference between ZFS and HW RAID wasn't great. If performance is your #1 priority, HW RAID will offer the best solution. In all other cases, I believe ZFS is the better choice.

growse
  • 7,830
  • 11
  • 72
  • 114
  • How can having to write data twice immediately over the bus come close to the performance of writing once immediately to RAM, regardless of how much RAM and CPU you've got? – womble Aug 11 '11 at 08:59
  • Not sure what you mean - both HW RAID and ZFS will do write caching...? In fact, ZFS lets you specify an entire set of disks as write cache devices if you want to. – growse Aug 11 '11 at 09:06
  • When you write cache in main RAM you lose the data if the machine dies, and write caching to a single disk means you lose the data if that disk dies. Hardly comparable to battery-backed cache. – womble Aug 11 '11 at 09:09
  • Ah, I see your point. True, battery-backed caches can provide significant benefit in the HW RAID space. I guess ZFS's counter to that would be to use a physical disk as a ZIL rather than RAM. If the machine dies, you can playback the write cache because it's on a persistent disk. – growse Aug 11 '11 at 09:13
  • And if the disk dies, you're *still* up the creek, and wasn't the whole point of running RAID because disks are unreliable? – womble Aug 11 '11 at 09:15
  • So you mirror the ZIL disk? – growse Aug 11 '11 at 09:18
  • And now you're back to where we started, making two writes to disk immediately rather than one write to RAM. – womble Aug 11 '11 at 09:19
  • 1
    If write latency is *that* important to you, then use HW RAID. In all other cases, I believe that the advantages of ZFS outweigh those of HW RAID. YMMV. Obviously. – growse Aug 11 '11 at 09:22
  • I'm only questioning your statement that "the performance differences between something like ZFS and hardware RAID1 are becoming negligable". I agree that ZFS has other benefits. – womble Aug 11 '11 at 09:27
  • You're right, I've badly worded. Will edit. :) – growse Aug 11 '11 at 09:30
  • Hmm I'm still not sure which to use then. The disks would be the root mirrors in a Oracle server. I'd like the ZFS ease of use features, but dont want to compromise on availability. Performance is not an issue for me , resiliance is. I suppose the only thing I'm worried about is ZFS going wrong if it uses a HW raid device rather than just 2 disks. – AndyM Aug 11 '11 at 09:57
  • 4
    @AndyM: If you want availability then ZFS is a *better* option, hardly any HW RAID solution checks both mirrors when reading let alone adds checksums to written data. – Hubert Kario Aug 11 '11 at 11:12
  • OK thanks, Im leaning towards the ZFS only option. Some people think its strange not to use the built-in HW RAID on a server, I suppose they dont know the benfits of ZFS :-) – AndyM Aug 11 '11 at 12:07
  • 2
    @growse: You no longer HAVE to mirror the ZIL disks. In the more recent Zpool vers (26+ IIRC) you can safely remove a ZIL disk from a pool. The pool simply reverts to using the pool disks itself for the ZIL. If you're doing a lot of sync writes then you'd take a performance hit until you replace the ZIL disk. – Tim Aug 11 '11 at 12:42
  • @Tim, I think the concern is that if the ZIL device fails, you'll lose whatever's been written to the ZIL but not yet committed to disk. Sure, you can safely remove, but a catastrophic removal is more the worry. – growse Aug 11 '11 at 13:51
  • I had did some more reading and you're 100% right. I totally missed that part in the reading I did. I've had to rethink my own plans for an unmirrored SLOG. – Tim Aug 17 '11 at 19:45
  • @growse Isn't it rather like if you lose the ZIL device, then you lose outstanding transaction groups if that loss *coincides* with a system crash? Remember the ZIL in normal operations is write-only; it's only read back on an unclean import. Under normal operations, data is written to ZIL and then committed to storage from ARC (the ZIL not taking part except for disaster recovery). – user Mar 14 '15 at 20:40
8

Although I love ZFS if you're doing a LOT of writing then take into consideration that any software RAID 1/10 will need to send every write twice over the bus and through the controller - meaning you can get into congestion and queue issues.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • This is a good point - from a performance point of view, HW RAID has some significant advantages in certain situations. – growse Aug 11 '11 at 09:14
  • 2
    On the other hand, if you have close to 150MiB/s random writes (it's a db server) you're a madman for running it on a single bus. Interface bandwidth is hardly a problem. – Hubert Kario Aug 11 '11 at 11:15
2

ZFS has lots of built in optimizations that work best when used without Raid controllers. On a ZFS Raid1 you get read performance that is usually superior to a hardware Raid, also with RaidZ you can get superior performance to Hardware Raid5 Just take a look at Anadtech.

What you absolutely need to take into consideration is how ZFS caches though. You should basically have 4GB of Ram to use it at all, 8GB to get some decent performance, but more is always good. There are lots of cases where more Ram helps with ZFS. The best ways to improve performance though are seperate devices (SSDs) for the ZIL and L2ARC, as those generally tend to be bottlenecking performance. This explains the idea behind L2ARC a little.

The principal problem with Hardware Raid here is, that ZFS does, what the Raid Controller does, so when using a Hardware Raid below ZFS, there is a Raid Controller that expects itself and only itself to be doing Queueing, Caching, IO Optimization and so on and there is ZFS on top of it also expecting the same of itself. This has side effects of course and they aren't necessarily pretty and will most likely have negative performance effects due to the way ZFS does I/O Parallelization. Also its predictive "self healing" features will in part be kind of useless that way. Therefor it is specifically recommended that when you use a Raid controller, you should expose the Disks JBOD are as Single-Disk Raid0 devices, so that ZFS can do the housekeeping by itself.

juwi
  • 573
  • 5
  • 14
-1

Well, you need to define "best". I'd expect that a good hardware RAID controller with BBU would provide superior performance, but perhaps performance isn't your primary consideration.

womble
  • 95,029
  • 29
  • 173
  • 228
  • And my answer is ZFS ontop of HW RAID or just ZFS. – womble Aug 11 '11 at 10:08
  • 1
    That is actually a very bad idea. It will basically not be good for ZFSs I/O Parallelization and for exactly this reason its probably not good for the disks either. – juwi Jan 26 '12 at 01:07