3

This is hopefully a simple question. Right now we are deploying servers which will serve as data warehouses. I know with RAID 5 the best practice is 6 disks per RAID 5. However, our plan is to use RAID 10 (both for performance and safety). We have a total of 14 disks (16 actually, but two are being used for OS). Keeping in mind that performance is very much an issue, which is better - doing several RAID 1's? Do one large RAID 10? One large RAID 10 had been our original plan, but I want to see if anyone has any opinions I haven't thought of.

Please note: This system was designed for using RAID 1+0, so losing half of the raw storage capacity is not an issue. Sorry I hadn't mentioned that initially. The concern is more whether or not we want to use one large RAID 1+0 containing all 14 disks, or several smaller RAID 1+0's and then stripe across them using LVM. I know the best practice for higher RAID levels is to never use more than 6 disks in an array.

masegaloeh
  • 17,978
  • 9
  • 56
  • 104
Matthew
  • 2,666
  • 8
  • 32
  • 50
  • 1
    What kind of writes are you expecting? Would these be GB's of sequential writes? Or smaller/random writes? – Univ426 Jun 25 '12 at 14:05
  • It's actually a mix of both. They're acting as datastores for splunk, so they'll be streaming in several hundred GB a day of data (sequential writes), but there will also be scheduled searches firing off every 2-3 minutes which will be reading against random data on disk. I can't give you actual utilization numbers though, sorry. The vendor describes it as incredibly write-heavy, and recommends raid 1+0. – Matthew Jun 25 '12 at 14:10
  • The vendor's recommendation is correct. If you have the space, RAID 1+0 is probably the right choice. – ewwhite Jun 25 '12 at 14:21
  • We do have the storage capacity (design was meant for raid 1+0). The concern is more whether or not we want to use one large Raid 1+0 containing all 14 disks, or several smaller raid 1+0's and then stripe across them using LVM. – Matthew Jun 25 '12 at 14:28
  • Is this on a SAN or are the disks directly attached to the server? – pauska Jun 25 '12 at 14:29
  • These are direct attached. Using HP DL380's with modified front bezel and upgraded controller to support 16 15k SFF disks. – Matthew Jun 25 '12 at 14:30

5 Answers5

5

Take a look at this discussion detailing the disk layout for a RAID 1+0 setup on an HP ProLiant server:

6 Disk Raid 1+0

A Smart Array controller configured in RAID 1+0 is a stripe across mirrored pairs. Depending on how you've arranged your drive cages and which controller you're using, the disks will likely be paired across controller channels.

E.g. in a 4-disk setup:

     Logical Drive: 1
     Size: 558.7 GB
     Fault Tolerance: RAID 1+0

     Logical Drive Label: AB3E858350123456789ABCDE6EEF
     Mirror Group 0:
        physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
        physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 300 GB, OK)
     Mirror Group 1:
        physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
        physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)

physicaldrive 1I:1:1 pairs to physicaldrive 1I:1:3
physicaldrive 1I:1:2 pairs to physicaldrive 1I:1:4

With that number of disks, there's no downside to leaving them in a single logical drive. You'll get the benefits of more (MOAR) spindles for sequential workloads and increased random workload capabilities. I'd recommend tuning the controller cache to bias towards writes (lower-latency) and possibly make some choices at the OS level regarding filesystem choice (XFS!), I/O elevators (deadline) and block device tuning.

Which operating system distribution will this be running on?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Thank you! We're using RHEL 5.6, and the plan had been ext3 (we don't have the entitlement for XFS/xfs-utils, and the volumes will never be larger than the ceiling of ext3). These boxes will be our unit of scale so when we outgrow them, we just add more boxes, not more storage. Otherwise, only other tweak will be increasing the stripe size of the raid to whatever the highest available is. – Matthew Jun 25 '12 at 14:55
3

Matthew - I'm a BIG Splunk client, we use R10 exclusively - whether it's on SAS disks for our low end boxes, Enterprise SSDs for medium sized systems or FusionIO cards for our busiest machines. You've been smart and sized for R10, trust your instincts, you're on the right path.

We just create one big PV/VG/LV for all of /splunkdata, leaving /opt/splunk on the boot disks by the way.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • Fantastic, thank you! My background with R5 just gets me nervous when I see that many disks in a particular raid :) – Matthew Jun 25 '12 at 15:15
  • Couldn't agree more, there's some big issues with large R5 array. By the way I'd imagine your array with spend >95% of its time writing, Splunk's not a database *at all* (it's actually very hard ensuring that everyone knows this), it's just a bin to throw things into, if there's no scheduled jobs and no manual searching going on then there's very few reads happening. – Chopper3 Jun 25 '12 at 15:22
1

400GB/day is ~270 MB/min.

RAID5 might work. RAID 50 is probably best in terms of storage efficiency and write performance. RAID 10 will give you the best write performance but at the cost of 50% storage efficiency. I worry about heavy writes plus the random reads... that is going to cut into overall performance.

The type of disk you use will be critical. If you can use 10k or 15k drives performance will greatly increase, but of course these disks are more expensive and lesser capacity than enterprise SATA/SAS 7.2k drives, which go up to 3+TB at current time.

Ultimately, no one can tell you what is best for your application, so you need to test it yourself. My recommendation is to a RAID card that has a large write cache (512MB or above) and a decent sized read cache as well. Then, test various RAID combinations (I would suggest RAID5-6 and RAID5-10, as well as RAID50 and RAID10). See what performs best. Tweak the RAID card settings. You will find the optimal configuration.

Jeremy
  • 938
  • 2
  • 7
  • 18
  • I appreciate your feedback but the system was actually designed with RAID 1+0 in mind, so we don't mind losing the capacity. The question is more whether having 14 disks in a single raid 1+0 is a good or bad idea based on best practices. – Matthew Jun 25 '12 at 14:33
1

"The concern is more whether or not we want to use one large Raid 1+0 containing all 14 disks, or several smaller raid 1+0's and then stripe across them using LVM."

Well. If you think about it, you're basically asking us wether you should let your raid controller do the RAID0-part of RAID10, or if you should let LVM do it.

I guess if you have the worst raid controller in the world then LVM could probably outperform it.. otherwise I think you're safe letting the raid controller do all the work.

pauska
  • 19,532
  • 4
  • 55
  • 75
  • lol, I don't think that's an issue. So, with Raid 5/6, the more disks you have in the array, the higher your exposure to losing the array in the event of multiple disk failures. Raid 10 I would expect has lower exposure. But is it still a good idea to have such a large array? If not I'd make several small ones and stripe across them using lvm, on top of hardware raid (the system will see it as one large volume). – Matthew Jun 25 '12 at 14:38
0

RAID-10 will cut your available space in half. I'd recommend RAID-50 instead, which requires at least 6 drives; it will give you great fault tolerance and performance.

You'll want to check out and benchmark several RAID cards, as they don't all perform the same. In case you don't know this already, make sure to use enterprise SATA drives, not desktop ones, and not "green" drives.

gekkz
  • 4,219
  • 2
  • 20
  • 19
  • Make that enterprise SAS drives, 2.5", and a SUperMicro rack case that holds 72 of them ;) – TomTom Jun 25 '12 at 14:26
  • I wanted to upvote this answer because of the suggestion of RAID50, which is excellent for random data. But, as TomTom pointed out, the suggestion for S-ATA makes me cringe.. – pauska Jun 25 '12 at 14:28
  • I appreciate your feedback but the system was actually designed with RAID 1+0 in mind, so we don't mind losing the capacity. The question is more whether having 14 disks in a single raid 1+0 is a good or bad idea based on best practices. – Matthew Jun 25 '12 at 14:33
  • I assumed for some reason SATA drives were going to be used which is why I made the recommendation for enterprise ones. If you've got the cash - SAS all the way. – gekkz Jun 25 '12 at 16:11