15
  • HP 2012i SAN,
  • 7 disks in RAID 5 with 1 hot spare,
  • took several days to expand the volume from 5 to 7 300GB SAS drives.

Looking for suggestions about when and how I would determine that having 2 volumes in the SAN, each one with RAID 5, would be better??

I can add 3 more drives to the controller someday, the SAN is used for ESX/vSphere VMs.

Thank you...

Teddy
  • 5,134
  • 1
  • 22
  • 27

9 Answers9

25

I've wrestled with this question for a while. There are a number of factors determining how many disks should go into a RAID5 array. I don't know the HP 2012i, so here is my generic advice for RAID5:

  • Non-recoverable read error rate: When a non-recoverable read error occurs, that read fails. For a healthy RAID5 array this is no problem since the missed read can be found in the parity information. If one happens during a rebuild, when the entire RAID5 set is read in order to regenerate the parity info, it can cause the entire RAID5 array to be lost. This rate is measured like this: "1 per 1014 bits" and is found on the detail tech-specs for drives. You do not want your RAID5 array to be any more than half that size. Enterprise drives (10K RPM SAS qualifies) can go longer than Desktop drives (SATA).
  • Performance degradation during rebuilds: If performance noticeably sucks during rebuilds, you want to make sure your array can rebuild quickly. In my experience write performance tends to suck a lot worse during rebuilds than reads. Know your I/O. Your tolerance for bad I/O performance will put an upper limit on how large your RAID5 array can get.
  • Performance degradation during other array actions: Adding disks, creating LUNs, changing stripe widths, changing RAID levels. All of these can impact performance. Some controllers are very good about isolating the performance hit. Others aren't so good. Do some testing to see how bad it gets during these operations. Find out if restriping a 2nd RAID5 array impacts performance to the first RAID5 array.
  • Frequency of expand/restripe operations: Adding disks, or on some controllers creating new LUNs, can also cause the entire array to redo parity. If you plan on active expansion, then you'll be running into this kind of performance degradation much more often than simple disk failure-rate would suggest.

RAID6 (double parity) is a way to get around the non-recoverable read error rate problem. It does increase controller overhead though, so be aware of the CPU limits on your controllers if you go there. You'll hit I/O bottlenecks faster using RAID6. If you do want to try RAID6, do some testing to see if it'll behave the way you need it to. It's a parity RAID, so it has the same performance penalties for rebuilds, expansions, and restripes as RAID5, it just lets you grow larger in a safer way.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
  • 13
    +1: The rebuild is a sticking point. It is not at all uncommon to lose a drive while you're rebuilding. I once lost 19 drives in a 24 disk cabinet (8 RAID5 arrays, no hot spares), one after the other. The rebuilds stressed them, and backing up the data to neighboring arrays stressed THOSE, and THEY started failing...Scary stuff. – Satanicpuppy Mar 16 '10 at 19:52
  • 1
    +1 to sysadmin + SatanicPuppy. Especially Satanicpuppy's point - if you buy a big array and fill it with disks in one go/ bulk purchase, they will stand a higher chance of being the same age and possibly the same batch. When one goes the others may well not be far behind. – Rob Moir Mar 16 '10 at 20:31
  • There's a very good discussion of this here: http://blogs.zdnet.com/storage/?p=162 – Whisk Mar 30 '10 at 16:23
6

There isn't really a clear limit on the number of disks in RAID 5 per se. The limits you'll meet are typically related to RAID 5's relatively poor write performance, and limitations imposed elsewhere (RAID controller, your own organization of data etc).

Having said that, with 7-8 disks in use you're close to upper bound on the common RAID 5 deployments. I'd guesstimate that the wast majority of RAID 5 deployments are using <10 disks. If more disks are wanted one would generally go for nested RAID levels such as RAID "50".

I'm more puzzled by your choice to keep one big array for all this. Would your needs not be better served by 2 arrays, one RAID5 for slow, mostly read data, and one RAID 10 for more I/O intensive data with more writes?

  • +1, General rule of thumb 7 or 8 dirves for RAID5; and about double that for RAID6 (though your card's performance will likely become a bottleneck). After that go RAID 50 or 60. – Chris S Mar 16 '10 at 20:41
3

For my money, I'd do two three-disk arrays, with one shared hot spare.

If you don't have an need for a single block of space larger than a 3 disk array, then there is no reason to cram all 6 disks into 1 raid. You're not going to gain anything in performance or over-all space, and, given a two disk failure, you're probably going to be in a better place.

@Dayton Brown: The total space will be the same...3 1TB drives in a RAID5 is what, 1.8TB? 6 will be 3.6 by the same measure, so in that sense you'll have more space in that particular RAID volume, even though the total space will remain the same. The difference is, RAID5 only allows for 1 parity drive, whether there are 3 drives in the RAID or 300, so splitting the drives into manageable groups adds protection against multiple failures. Even if you lost 3 disks, for example, you'd only lose half of your data.

If you moved to RAID6, a six disk array would make more sense, because you could lose two and be okay. But most people jump straight to RAID10, and skip 6.

Satanicpuppy
  • 5,917
  • 1
  • 16
  • 18
  • Why do no more than 3? Aren't you increasing the amount of lost storage significantly with each new volume? With a hot spare (and monitoring) you will have plenty of time to replace any bad drive once it bites the dust. There is going to be a performance hit, but that is only if you have a dual channel raid controller that can hammer the throughput... I'm more curious than anything. – Dayton Brown Mar 16 '10 at 17:26
  • 1
    I suspect that the poster recommends no more than 3 because if you lose another disk (or encounter disk errors of almost any kind) during the rebuild, *the entire array may be lost*. That's why I said below that it is preferable to keep the disk count as low as possible, while offering the necessary space. – Joe Mar 16 '10 at 18:39
  • 1
    A 6 x 1TB raid5 array will be closer to 4.6 TB, not 3.6TB – ITGuy24 Mar 16 '10 at 20:54
  • +1: Yep yep. You're right. 1 parity disk out of 6 instead of 1 out of 3. My bad. – Satanicpuppy Mar 17 '10 at 01:26
  • @Joe. Thanks for the explanation. That's a very good reason. – Dayton Brown Mar 17 '10 at 14:13
3

How much space do your NEED? You currently have ~1.995TB, could you live with ~1.140TB? if so then I'd strongly suggest you move to RAID 10 - not only will it be faster but you'll be able to lose half your disks without user impact. If you choose to go this way I'd order the extra 4 x 300GB disks now and build it with all 12 disks on day one rather than add them later - this will also handily give you ~2.280TB available too.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • 5
    You can lose UP TO half the disks. If you lose two disks from the same mirror, you're SOL. – Nic Mar 30 '10 at 15:18
2

With RAID5, you want to keep your disk count as low as possible, while still providing the necessary space. A hot spare, per RAID5 array, is recommended to ensure reliability.

RAID6 is really a much better idea for a future addition. Fully populate the rest of your array with disks, and make a new LUN using RAID6 using all available disks, with one hot spare. Migrate your VMs to the new LUN and convert that RAID5 array to RAID6.

Lots of folks, in fact, recommend this, especially in this day and age.

Community
  • 1
Joe
  • 1,765
  • 15
  • 23
2

Any number of disks in a RAID 5 array is too many disks. The only possible exception maybe if you want the maximum storage/$ for something where speed and a reasonably quick recovery from failure is not an issue.

You've invested in expensive hardware to build a SAN; arrays, switches, fibre, training etc. Then you've put in pretty fast (relatively) expensive disks. Furthermore you want to put VMs on it. With this investment and usage it looks like you want to make it as fast as you can so your VM users don't complain about performance. So why touch RAID 5 at all?

BAARF is old, but still relevant as using RAID5 is as bad an idea now as it was a decade ago.

goo
  • 2,838
  • 18
  • 15
0

Technically? I think my adaptec 5805 limits a single array to 32 discs. I would, though, NEVER put a RAID 5 over so many discs ;)

TomTom
  • 50,857
  • 7
  • 52
  • 134
0

I don't have a suggestion as to the array size but I would suggest adding a second hot spare or making this RAID 6. The time required to rebuild an array of that size increases the risk of a drive failing before the array has been rebuilt.

John Gardeniers
  • 27,262
  • 12
  • 53
  • 108
-1

If you have any doubts about the RAID 5 array and number of disks needed for the setup, then go through this link.

Lucas Kauffman
  • 16,818
  • 9
  • 57
  • 92
null_pointer86
  • 1