3

I have a three server Hyper-V 2012 cluster which points at LUNs based on an HP MSA P2000 G3 SAN for the virtual hard disk storage. I am experiencing speed issues and one thing that has been highlighted is latency in the read/write speed to and from the SAN. I spoke to HPE who checked the SAN for hardware faults and found nothing. However, to me the performance data over the last four days looks to be spiking well above the 10-20ms range deemed acceptable. I don't have anything to compare my charts to as a baseline was never taken.

This storage system has worked well in the past and there haven't been any recent changes. If anyone can provide me with any insight into this i'd be very grateful.

Many thanks,

Steve

P2000 Perfromance Graph

Steve Gould
  • 49
  • 4
  • How many disks are installed? How are the vdisks arranged? Is this a SAS, Fiber Channel or iSCSI unit? – ewwhite Sep 05 '16 at 16:40
  • Adding to @ewwhite What kind of disks you have in the SAN (SATA, Nearline SAS, SAS, flash)? What RAID level? Can you pull performance reports per virtual machine, at the same time you pull the performance reports of the SAN? Do you have an accurate inventory of what each virtual machine/physical machine (attached to the SAN) does? – CIA Sep 05 '16 at 16:54
  • It's a low-end SAN. He won't get per-VM stats or any granular info. – ewwhite Sep 05 '16 at 16:56
  • He should be able to pull performance monitor data from the HyperV servers at least. – CIA Sep 05 '16 at 17:48
  • There are two arrays of six SAS disks each. The first array is made up of 6 x 1TB 2.5inch 7.5k drives. The second is 6 x 600GB 2.5inch 10k drives. Both arrays are Raid10 and it is an iSCSI unit. We are running 10VMs in total which do a mixture of things: File storage, SQL, Exchange and user applications (RDS). It is a low end SAN - you're right. The amount of performance data is limited on the console aside from the graph I have attached. These is a CLI but my experience of this is limited. Performance data from the Resouce Monitor in Hyper-V seems to reflect the performance stats on the SAN. – Steve Gould Sep 06 '16 at 07:55
  • If you can do a performance monitor capture on individual VMs in your HyperV environment, you can get a better idea of what's going on. I recommend you capture about a week work of data for each server, then graph performance for % Processor, Memory Used, DISK I/O, Disk Avg Read/s, Disk Avg Write/s, and Network Utilization, then compare it to the data you get from the SAN. Usually, disk latency means your storage is queuing up requests, because there are too many I/O requests from one (or all) of your servers. Pulling specific performance data for each VM will help you identify which – CIA Sep 06 '16 at 13:31

1 Answers1

1

This model of HP SAN array had a problem with the write cache modules failing. These are 4GB Compact Flash cards installed on the controller modules.

If this is sudden behavior, that may be worth a check.

I'd suggest controller failover, but need the details about what type of P2000 G3 array this is (iSCSI, SAS, FC) and how it's connected to your hosts.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • That is an excellent call. We actually had one of these compact flash cards fail not long after we got the SAN up and runnning. HP replaced it under warranty. The problem was flagged up in the web management console. Unfortunately nothing like this is showing up this time and HP have checked the system logs and confirm all hardware performing normally. – Steve Gould Sep 06 '16 at 07:57
  • Have you failed over to the other controller to test performance? – ewwhite Sep 06 '16 at 09:33
  • The arrays are already split between the controllers. The 7.5k array is on controller A and 10k on B. The graphs seem to show equally poor response on either array, slightly worse on B. This was my rational for not testing all the load on one controller. – Steve Gould Sep 06 '16 at 09:38