3

I just got a HP DL180 G6 that has 25X 146GB 15K SAS drives, 36GB RAM, 2X 2.0GHz Xeon 1333Mhz FSB. For fun I configured them all in a single RAID 0 and installed Ubuntu on it to see how fast it could get with 25 drives on a HP Smart Array P410 Controller w/ 512MB RAM.

When I ran hdparm -tT /dev/mapper/concorde--vg-root I get

Timing cached reads:   5658MB in  1.99 seconds = 2834.13 MB/sec
Timing buffered disk reads: 1192 MB in  3.00 seconds = 397.13 MB/sec

When I run the same command on my other server (HP DL360 G5 - 32GB RAM - 2X 2.66GHz 667Mhz FSB) that only has 4X 15K drives I get:

Timing cached reads:   13268 MB in  1.99 seconds = 6665.18 MB/sec
Timing buffered disk reads: 712 MB in  3.00 seconds = 237.17 MB/sec

I would have expected this to run 5 times faster than the old one, not slower. The server is intended to deliver streaming media and so I need super fast access and transfer to keep up with 2 1Gb network ports I hope to max out at times along with performing it's other tasks.

I just put together a bunch of copies of a 400MB MP4 file to get 45GB to copy from one directory to another and it took 96 seconds, which just seems wrong for everything I have ever heard of the performance boost of RAID 0.

It is setup as a hardware raid, is there anything I need to do in Linux to take advantage of the extra speed that should be there? Does it matter which flavor of Linux I use? I am comfortable with CentOS and Ubuntu but could do others if needed.

Is there a different command I should use to measure performance? I tried using iotop and iostat yesterday to monitor the RAID usage and couldn't get it to report any usage while copying 2GB files over FTP, so kind of stuck trying to set a benchmark, comparing it's performance across servers, and monitoring it so I know when the hard drives are maxing out and need to be replaced with SSD.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
Alan
  • 543
  • 2
  • 6
  • 18
  • The buffered disk read *is* better on the new array (as you'd expect.) You are attributing too much weight to the [cached read](http://serverfault.com/q/654271/50647). – Aaron Copley Dec 31 '14 at 20:29
  • Sorry, you are right, but with 25 drives in the RAID shouldn't it be 5X faster on the buffered disk read, not marginally better? – Alan Dec 31 '14 at 20:36
  • 1
    RAID 0 isn't going to be the final configuration, is it?! – Michael Hampton Dec 31 '14 at 20:40
  • I was thinking of doing it like this: 5 drives in RAID 5 - OS Drive, 10 drives in RAID 0 - Content disk 1, 10 drives in RAID 0 - Content disk 2. I really just wanted to see how fast it "could" go. – Alan Dec 31 '14 at 20:48
  • 1
    No, I don't think so. What are you basing this 5x figure on? In my experience, throughput (read or write) does not increase linearly for RAID 0 despite popular lore. – Aaron Copley Dec 31 '14 at 20:49
  • You really don't want to use RAID 0 for anything here. If you lose a drive, then your server is down until you replace the drive _and_ re-upload _all_ your data, since you will have lost it all. – Michael Hampton Dec 31 '14 at 21:45
  • @MichaelHampton - there are lots of use cases for RAID0 where I/O is more important than data availability. Like on a cluster where other nodes have a copy of the data. Or as another example, company uses RAID0 arrays extensively to cache datasets for processing that are read from an external data store. If a drive fails and the node is lost, it's not a problem, since it checkpoints results periodically back to the permanent data store and another node can pick up where it left off. In this case, I think Alan was just trying to benchmark his new server, so there's no problem with using RAID0. – Johnny Jan 01 '15 at 02:12
  • 1
    @Johnny Since I answered his _last_ question, I know exactly what his use case _actually_ is. :) – Michael Hampton Jan 01 '15 at 02:18
  • This particular server will end up being the permanent data store that other servers will pull from to cache, more than anything though I am just trying to learn more about the hardware capabilities under different configurations so as I go to buy more servers I van make better choices about where my money is best spent. For instance I can get a LOT more data storage with 7200RPM drives than 15K, and if I get a server like this with 25 bays, it might be just fine to have 15 drives in the RAID. Or if I do a 1U with only 6 bays, do I need SSD, or should I just put in 196GB RAM. – Alan Jan 01 '15 at 16:25
  • I am trying to approach my very real problems using an academic approach learning what is most important for my use case. As Michael knows, we are spending way too much on CDN hosting ($6K/month) and can do it ourselves for 10% of the cost if we have the right servers. – Alan Jan 01 '15 at 16:28

2 Answers2

8

Wow... there's a lot to address here.

  • Disk performance isn't just about throughput. There's the notion of IOPS and latency and service times to contend with. Most workloads are a bit random in nature, so 25 enterprise disks in an array will always trump 4 disks from an IOPS perspective.

  • hdparm is not the right tool to benchmark enterprise storage. Look into purpose-built programs like iozone and fio.

An example iozone command that could be helpful for you is (run from a large directory on the disk array you wish to test): iozone -t1 -i0 -i1 -i2 -r1m -s72g

  • The design of this server means that your disk backplane is oversubscribed. There's an expander chip on the server and those 25 disks are sharing a 4-lane 6Gbps connection to the RAID controller. That means that you have a theoretical maximum throughput of 24Gbps (or 3,000 Megabyte/second) to the array. That's a ceiling, and you won't see performance beyond that point.

  • Ubuntu is almost never the best choice when hardware drivers and support are concerned.It's not officially supported by the server. CentOS or RHEL would be a better fit for this hardware.

  • HP Smart Array controllers have the ability to carve a group of disks (an array) into multiple logical drives of varying capacity and RAID levels. The example below shows a 4-disk array carved into three logical drives. One of the logical drives is configured with a different RAID level than the others.

Smart Array P420i in Slot 0 (Embedded) (sn: 0014380296200A0)

  logicaldrive 1 (72.0 GB, RAID 1+0, OK)
  logicaldrive 2 (1024.0 GB, RAID 1+0, OK)
  logicaldrive 3 (869.1 GB, RAID 5, OK)

  physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS, 900.1 GB, OK)

  • At no point should you use RAID 0 for a logical drive here. If you can spare the space, RAID 1+0 will perform very well with this hardware combination.

  • You have LVM in place. That's not the best approach when working with these HP Smart Array RAID controllers. It's an additional layer of abstraction, and you won't see the best performance (although it can be tuned to work well).

  • Firmware. You'll want to update the firmware of the server and related components. Functionality improves with each HP RAID controller firmware revision.

  • RAID cache configuration. Make sure the RAID battery is healthy and that the cache read/write balance is right for your workload.

  • Filesystem choice. XFS is a good option for streaming media. But the rates you're asking for are relatively low. Any modern Linux filesystem should be fine.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • I disagree with a blanket statement saying that RAID-0 should not be used, there are lots of use cases for RAID0 where the increase in physical I/O (or storage capacity) outweighs the inconvenience of the loss of an array or an entire node. – Johnny Jan 01 '15 at 02:14
  • With this setup, a RAID 0 logical drive will have diminishing returns due to controller and backplane limitations. In addition, recovering a RAID 0 logical drive on an HP controller is a hassle. Not worth the *perceived* speed increase. – ewwhite Jan 01 '15 at 02:25
  • Whether the vendor is willing to provide contractual support for one distro or not has no bearing at all on how well it can be expected to perform. Same kernel, same drivers, same performance can be expected. – psusi Jan 01 '15 at 05:23
  • When I get back to the office I will install new firmware, divide up the RAID into multiple sub sets, install CentOS and check battery. I didn't have a voltmeter with me yesterday to check it. Raid 1+0 should be just fine, I was just doing it this first way as a fun test - which didn't work out like I thought. – Alan Jan 01 '15 at 16:30
  • @psusi Vendor support *does* matter in the form of drivers and monitoring tools. HP ProLiant servers have some value-added features that are often not available or intended for Ubuntu. [Sometimes it's just a non-starter](http://serverfault.com/a/526227/13325). – ewwhite Jan 01 '15 at 16:32
  • That sounds like a fakeraid, in which case, you are better off using it in AHCI mode like they said works, and sticking with `mdadm` instead of their B.S. closed source fakeraid driver. – psusi Jan 01 '15 at 16:57
  • What about replacing the RAID controller? I am not sure where I fall in the line of prosumer vs enterprise user and am not sure what I should spend on one. I see the controller I have is around $10 used on ebay, I certainly don't mind upgrading, but not sure how much to spend to get something that will work well without wasting money. – Alan Jan 02 '15 at 01:00
  • @Alan The controller you have is fine. It's the proper device for the server you have. Don't change it. The performance you'll have with this combination is fine. Don't change it. Please ignore the bad advice the other user is posting. – ewwhite Jan 02 '15 at 01:01
1

One thing that jumps out at me right away is the low cached througput on the new server, which indicates you have a ram bottleneck. You mention it has 36 GB of ram in it, which is a bit of an odd number. I am betting that you failed to install the ram in pairs ( or 3x if this cpu/board is capable of triple interleaving, as I suspect it is ), and so the ram is not interleaved properly. You might run memtest86+ to get a good measurement of the memory throughput.

As for the actual IO throughput, for an array this large you are probably not throwing enough requests at it. hdparm only reads 2 MiB at a time, so if the array is using a 512k stripe factor, that will only keep 4 of the drives busy at a time. For other commands like dd, it depends on the readahead size and the default readahead size is only 128 kb. Try using dd instead of hdparm ( without iflag=direct ) and pump up the readahead factor in /sys/block/dm-X/queue/read_ahead_kb to make sure the kernel sends large enough requests to keep all of the drives busy, or if you do use iflag=direct with dd, then make sure to use a very large value for bs=, like 32MiB, and then the kernel readahead setting doesn't matter.

psusi
  • 3,247
  • 1
  • 16
  • 9
  • Thanks psusi, I have more ram on order and will be filling all banks with 8GB sticks. I thought the ram arrangement seemed wrong, but where I bought it from was having trouble getting it all to show in other configurations. When I get back to the office I will try a few of those other commands like you suggest. One thing is you mentioned that hdparm only does 2MiB at a time, but I tried copying 45GB which took 96 seconds. Now this may have been a faulty test because I was reading and writing to the same raid, I can try setting up several other raids to try more stuff. – Alan Jan 01 '15 at 16:19
  • There is only one way to install/situate RAM on this model. Please refer to the **DIMM Population Guidelines** section in the [DL180 G6 QuickSpecs document](http://www8.hp.com/h20195/v2/getpdf.aspx/c04286375.pdf?ver=12). – ewwhite Jan 01 '15 at 16:35
  • @Alan, best test is to `dd` to read from the drive and write to /dev/null. – psusi Jan 01 '15 at 16:51
  • Just a quick update @ewwhite, @psusi. Back in the office today. I reconfigured the RAM and added 32GB from another computer and got a little better performance on hdparm (3431MB/s and 419MB/s). I checked the battery and it was completely dead, trying to see if I can pick one up locally where I bought it but they may be closed today. Using the `dd` command I was able to copy a 439MB file in 2 seconds for a rate of 220 MB/s. I am in the process of downloading and installing CentOS and will configure the filesystem, stripe and stuff from your suggestions and try again. – Alan Jan 02 '15 at 17:14