3

I have an HS22 blade configured with two 600GB 10K 6Gbps SAS drives configured in RAID1 using the onboard LSI Logic controller. Running on it is VMWare ESXi 4.0u2, and on top of that are a couple VMs. (Yes, I'm aware that we should be providing storage via SAN, but this was a budgetary constraint) I'm seeing poor read/write performance

  • Host A: RHEL 5.5, 8GB RAM, 2 vCPUs
  • Host B: CentOS 5.5, 1GB RAM, 2 vCPUs

Both kernels are configured to boot with elevator=noop

Result of ~8GB dd on Host A to a 350GB thin-provisioned disk, ext3 formatted:

# dd if=/dev/zero of=fullram bs=1K count=8388608
8388608+0 records in
8388608+0 records out
8589934592 bytes (8.6 GB) copied, 467.934 seconds, 18.4 MB/s

The maximum write performance I've seen is ~30MB/s (monitored through vSphere client)

Result of ~8GB dd on Host B to a 40GB thin-provisioned disk, ext3 formatted:

# dd if=/dev/zero of=fullram bs=1K count=8388608      
8388608+0 records in
8388608+0 records out
8589934592 bytes (8.6 GB) copied, 478.192 seconds, 18.0 MB/s

The maximum write performance I've seen for this VM however is about 50MB/s (monitored through vSphere client)

I've tested read performance on Host A in the following way:

  1. dd a 1GB file
  2. dd a second file to the same size as RAM (8GB)
  3. Read the 1GB file with dd

Result was:

# dd if=testfile of=/dev/null bs=1K
2097152+0 records in
2097152+0 records out
2147483648 bytes (2.1 GB) copied, 190.255 seconds, 11.3 MB/s

I'm at a loss for what could be causing this issue

brent
  • 3,481
  • 3
  • 25
  • 37

2 Answers2

3

Thin is slower, have you tried thick for comparison?

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • From my testing there seems to be a slight performance increase, about 20MB/s write. – brent Oct 22 '10 at 18:08
  • hmmm...here's a thought - if you can, try ripping out one of the disks - the controller won't be trying to mirror the data, see if that makes a difference, you can plug it back in for a rebuild right after. I've got lots of very similarly-spec'ed HP blades that I can try this on next week if you like, they're FC-connected usually but I could recreate the local datastore for a test. – Chopper3 Oct 22 '10 at 18:22
  • I believe there's some outstanding firmware fixes that I'll apply before I get into it any further. Do you know what SAS controller your blades have? The HS22s are LSI 1064e – brent Oct 22 '10 at 19:05
  • it's called a HP P410i but I don't know who actually makes it sorry – Chopper3 Oct 22 '10 at 19:57
3

Seriously - what do you expect? 19k SAS drives are not exactly stelar in perforamnce and you put up a LOT of stuff onto one blade. Get a storage blade or a SAN behind it.

To give you a comparison - I run a dual Opteron Hyper-V server with 64gb RAM.... Using Velociraptors 10k RPM drives 300gb earch (roughly -comparable to your discs - lets sqay a little slower).

But I use 6, soon 8 of those to get the IOPS budget I need for that server, and db intensive stuff (a database) uses ANOTHER 86 discs in RAID 10 mode.

Basically a RAID 1 is physically limited. You will need to put in a lot more discs, and blades are NOT well suited for that, unless your blade can get what IBM calls storage blades (basically a blade with just discs, no processor, but quite some of them).

TomTom
  • 50,857
  • 7
  • 52
  • 134
  • 1
    I don't think 2 VMs with 2 vCPUs each, allocated 9GB RAM on a blade that's 2x6 core (12 physical cores, 24 logical cores) with 70GB RAM is "a LOT". I didn't expect stellar disk performance, but I expected better than 18MB/s. We're waiting on the DC PSU DS3500 to come out (Q12011) before we can get a SAN behind it, so we needed an "in the meantime" solution. – brent Oct 22 '10 at 17:46
  • @TomTom - The question isn't about whether the kit he has CAN give him the performance he WANTS, it's about whether the kit he has IS giving him the performance is SHOULD. A R1 pair of modern 10k's on an otherwise quiet raw linux box should be able to give at least 30-40MBps on sequential writes, he's not getting that. – Chopper3 Oct 22 '10 at 17:49
  • it is a LOT for the - sorry - piss pure disc layout. Turn whatever you want, but you only have a RAID 1 here of a 10k disc - make the calculation of your IOPS budget yourself. This is a piss poor dubget for any virtualization. AT LEAST get a storage blade (39R7563, "Expansion blade"). Using 2 discs in a RAID 10 just will not give you the budget you need. You run a LOT of IO compared to what is a glorified home computer disc subsystem at the moment. – TomTom Oct 22 '10 at 17:51
  • @Chopper: Sort of agree, could be a driver problem for the raid controller. But sequential write performance is exactly one thing in virtualization: a totally (!) useless number anyway, because multiple VMs always interfere with the sequential access. – TomTom Oct 22 '10 at 17:52
  • Not always, what if one VM never touches the disk while the other does nothing but? I know it could be a driver thing, @brent - have you applied all patches? but the dude's using thin disks, they're slower, period - you need to test both to compare first. – Chopper3 Oct 22 '10 at 18:04
  • Totally unrealistic. Remember all the log files being written. Even small IO disrupts. – TomTom Oct 22 '10 at 18:44
  • I'm going to accept this answer simply because it's given me a bit of perspective on the configuration. – brent Oct 28 '10 at 17:58