3

We are doing some tests on a new database server with 4 x 240 GB SSD disks. From what I have read RAID 10 should be faster than RAID 5 with the same "one-disk loss ok" redundancy.

However when testing with bonnie++ it seems the RAID 10 isn't any quicker than RAID 5. Any idea why?

RAID5 (all 4 disks):

# cat /proc/mdstat
md2 : active raid5 sdd3[4] sdc3[2] sda3[0] sdb3[1]
      688730112 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        647G  1.6G  613G   1% /

# bonnie++ -d /tmp -u root

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
db1a           252G  1113  99 474860  26 327393  16  5943  99 1192788  23 +++++ +++

Sequential write:   0.474 G/s
Sequential rewrite: 0.327 G/s
Sequential read:    1.192 G/s

RAID10:

# cat /proc/mdstat
md2 : active raid10 sdd3[3] sdc3[2] sdb3[1] sda3[0]
      459153408 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

 # df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        431G  1.6G  408G   1% /

# bonnie++ -d /tmp -u root

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
db1a           252G  1221  99 492972  27 323392  15  5688 100 1178194  23 +++++ +++

Sequential write:   0.492 G/s
Sequential rewrite: 0.323 G/s
Sequential read:    1.178 G/s

Update

I ran the RAID 10 test with iozone to see if a multithreaded benchmark would perform any better on the assumption that the 99%-100% CPU reported by bonnie++ might indicate a bottleneck:

# iozone -R -i 0 -i 1 -l 12 -u 12 -r 8k -s 22G
(12 threads, 8k block size, total file size of 264G)

"  Initial write "  538817.21  0.538 G/s
"        Rewrite "  511450.04  0.511 G/s
"           Read " 1087437.45  1.087 G/s
"        Re-read " 1201127.73  1.201 G/s
"    Random read "  576435.70  0.576 G/s
"   Random write "  400612.46  0.400 G/s

The results are slightly better than bonnie++ but not much.

iozone results for RAID 5:

"  Initial write "  516469.10   0.516 G/s
"        Rewrite "  489970.21   0.489 G/s
"           Read " 1116074.84   1.116 G/s
"        Re-read " 1116666.97   1.116 G/s
"    Random read "  611738.43   0.611 G/s 
"   Random write "  199486.44   0.199 G/s

So as explained in the answers RAID 10 random write performance is twice as fast as RAID 5 but all the other stats are similar or slightly better.

David Tinker
  • 557
  • 1
  • 8
  • 16
  • 1
    1.178M/s are 1.1 megabyte? Your problem is not the RAID, your problem is that you have a speed on those "ssd" that is about 1% of what it should be. Heck, not even 1%. A Raid 10 of 4 SSD should read more than 500 mb/second. – TomTom Nov 26 '14 at 16:25
  • Oops sorry wrong units, have updated – David Tinker Nov 26 '14 at 16:31
  • Yeah. That makes a lot more sense. How large is th test? – TomTom Nov 26 '14 at 16:35
  • 252G and the machine has 128G RAM – David Tinker Nov 26 '14 at 16:36
  • Bonn++ has already warned you BTW, but you shouldn't do that test as root :) – Halfgaar Nov 26 '14 at 19:00
  • Also, for valid test results, make sure the file system is `fstrim`ed and let the drive's garbage collector clean up the data. – Halfgaar Nov 26 '14 at 19:01
  • Have you performed your tests against a single drive? The single drive stats might be interesting. Did you notice that your CPU is hitting 99-100% on most of the tests? I am not completely familiar with bonnie++ what does your CPU usage look like during the test? Are you maxing out a CPU core? – Zoredache Nov 26 '14 at 21:29
  • Are these disks all on one controller/channel, looks like you're hitting a single-SATA-channel limit to me – Chopper3 Nov 28 '14 at 08:54
  • Hmm looks like the machine only has one controller so I guess they are all on it. – David Tinker Nov 28 '14 at 21:07

2 Answers2

2

From what I have read RAID 10 should be faster than RAID 5 with the same "one-disk loss ok" redundancy.

I don't agree.

Let's look at reads -- here, there's no reason there should be any difference. Both let you read data from all four drives and use their full bandwidth. With RAID 5, no parity is read unless it's needed, so no difference there.

Now, let's look at writes. For RAID 10, bandwidth is halved since each write has to be done twice. With RAID 5, it's not quite so bad. We have to write out parity, but only 1/4 of the data is parity (for every 3 bytes of data we write, we have to write one byte of parity). So RAID 10 halves the bandwidth, RAID 5 has a 33% penalty. So RAID 10 is a tiny bit worse here.

Why should RAID 10 be better? (Assuming no device failures.)

David Schwartz
  • 31,215
  • 2
  • 53
  • 82
  • I would expect **random** writes in RAID 5 to be considerably slower. For RAID 10 you have to write to two disks only. For RAID 5, you have to write the block to one disk, write out the parity to another, but additionally you also have to read out the rest of the data blocks from the same stripe in order to recalculate that parity. – chutz Nov 27 '14 at 08:16
  • Hmm. That makes a lot of sense. So the only reason to use RAID 10 for a db server would be for faster writes? – David Tinker Nov 27 '14 at 08:33
  • @chutz He's not testing small random writes. But in any event, the same thing applies to RAID 10. If you change just one byte, it has to read the entire block containing that byte. – David Schwartz Nov 27 '14 at 08:38
  • @DavidSchwartz For RAID 10 you still only need to work with **two** disks, while for RAID 5 you need to work on **all** disks. – chutz Nov 27 '14 at 09:35
  • @chutz Why do you think that makes any difference? – David Schwartz Nov 27 '14 at 10:03
  • RAID-5 must update parity for every write which means reading every block on other data disks while also writing the original data and updated parity block. It's called the RAID-5 write hole problem. Linux has a "stripe cache" in md raid5 that allows deferring this work for a while, coalescing parity updates into large sequential operations which are much faster. As noted by @chutz the worst enemy of the RAID-5 stripe cache is lots of random writes which will seriously hurt overall array performance. RAID-10 has no parity to update, so no extra reads and writes are involved. – Jody Bruchon Jul 30 '18 at 11:30
0

I don't thing striping RAID is of any use with an SSD. Striping distrubutes work over several disk heads, but SSDs already have excellent random access.

Halfgaar
  • 7,921
  • 5
  • 42
  • 81