Determining VM doing I/O on a Hyper-V host

Question

How do I find out which VM is responsible for the majority of the I/O I see with Hyper-V?

The situation: I do have Hyper-V hosts with a number of VMs (around 30 per host) where I occasionally see prolonged periods of high disk utilization. I would like to know which VM is causing this.

I tried using Process Explorer, but it reported all Hyper-V related I/O to be handled by the "System" process, so no help there:

It already would help to have a breakdown by files accessed. The full GUI install of Windows Server has the Resource Monitor which is providing this information. However, Resmon seems to be unavailable in Core and Hyper-V installations.

Answer 1

Open Performance Monitor (run as Admin) on your local workstation. Add counter, select counters from the hyper-v machine, Hyper-V Virtual IDE Controller or Hyper-V Virtual Storage Device, select counters and instances (VMs) as seems appropriate. You might have to fish around a bit to find the counter that makes the most sense to you.

There are some good hints as to which Counters to look for in Monitoring Hyper-V Performance

Answer 2

Syneticon-dj, I wrote something for you this afternoon. I thought this problem was interesting, so this simple script will give you the read and write IO stats on each running VM on the Hyper-V host. As an added bonus it associates each VM to its vmwp.exe's Process ID.

You can run this on your Hyper-V server, because it doesn't need a GUI.

The downside is that while I was working on this, I noticed that the performance counters were working great for a while, and then for no discernible reason they decided to all stay on zero. Well maybe it's not a bug, like Chris S says... but these counters might unfortunately not be very useful after all. Regardless, it would be very easy to modify the script to use the Virt. Storage Device counters instead.

The output looks like this:

PID     VMName               ReadBytesPerSec             WriteBytesPerSec
---     ------               ---------------             ----------------
5108    DC02                          483.90                            0
2796    DC01                               0                            0
3348    ECA01                     4782668.27                            0

---

#Requires -Version 3
function Get-VMPidAndIO
{
<#
.SYNOPSIS
    Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
.DESCRIPTION
    Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
    Currently only works for VMs using virtual IDE controllers.
    Requires Powershell 3 at a minimum.
.LINK
    http://myotherpcisacloud.com
.NOTES
    Written by Ryan Ries, June 2013.
    ryan@myotherpcisacloud.com
#>
    BEGIN
    {
        Try
        {
            $VMProcesses = Get-CimInstance -Query "Select ProcessId,CommandLine From Win32_Process Where Name ='vmwp.exe'" -ErrorAction Stop
        }
        Catch
        {
            Write-Error $_.Exception.Message
            Return
        }
    }
    PROCESS
    {

    }
    END
    {
        Foreach($_ In $VMProcesses) 
        {
            $VMName = $((Get-VM | Where Id -EQ $_.CommandLine.Split(' ')[-1]).Name)            
            [PSCustomObject]@{PID=$_.ProcessId;
                              VMName=$VMName; 
                              ReadBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Read Bytes/sec").CounterSamples.CookedValue, 2);
                              WriteBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Write Bytes/sec").CounterSamples.CookedValue, 2); }
        }

    }
}

Answer 3

Both of the other answers are useful. But I find that I get what you're looking for much more easily (when the VHDs are stored on local or locally shared disks instead of a file server) by opening up "Resource Monitor" and looking at the Disk tab. Look at "Disk Activity" and sort by "Total (B/sec)." You'll see the VHDs listed in order of their activity.

Obviously my strategy is a loose approximation for machines that you're locally logged into, which makes it less useful in a big, headless environment.

Answer 4

I wrote a PowerShell GUI tool gathering Guest performance data for all VM's found on selected Hyper-V hosts. Data gathering and matching is done with get-counter and Hyper-V WMI information. Run from any member server, no modules required. I hope this will help in quickly troubleshooting performance problems on Hyper-V hosts and VM's.

Show Hyper-V Virtual Machine Guest Performance Statistics (PowerShell) https://gallery.technet.microsoft.com/Show-Hyper-V-Virtual-652fdd54

Answer 5

You may want to try a 3rd party tool that can help with detecintg virtual machine that is causing high I/O, and also has the ability to alert on such high I/O’s. You can check it out here - ApexSQL VM Monitor.

This tool can monitor large spectre of disk counters for host and virtual machine, and beside predefined counters, it can be set up to monitor and show graphical presentation of counters from Performance monitor.

Hope I helped.