14

I have seen on numerous virtualization services (Azure) and products (vmware,kvm,hyperv) I/O and system stalls under heavy I/O workload.

My questions are:

  • Is it ever sane to use a Virtualized solution when performing I/O heavy workloads?
  • What are the best practices around this sort of stuff?
  • What causes these problems, are there well known system bottlenecks, or is just a question of excessive contention?
bryan hunt
  • 257
  • 1
  • 4
  • Relevant [IBM research paper](http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf). I'd also consider using containers, e.g. Docker. From the paper: *"Containers can also eliminate the distinction between IaaS and “bare metal” non-virtualized servers [...] since they offer the control and isolation of VMs with the performance of bare metal."* – oleksii Jun 10 '15 at 12:49
  • 1
    1. What research have you done? We expect you to do a significant amount of research before asking. Showing your research helps others, and helps people give you better answers. See http://serverfault.com/help/how-to-ask and the [help]. 2. This question is quite broad, possibly too broad for this site. Standard advice is: one question per question. 3. This site is not a good fit for questions that call for opinion. Can you ask something more specific and more objectively answerable than "best practices around this sort of stuff"? – D.W. Jun 10 '15 at 20:42
  • 1. 10 years of running various virtual machines, kvn, xen, virtualbox, amazon, azure, hyperv. I've found the performance to be generally disastrous but there is so much hype and venture capital floating around that current IT orthodoxy is that everything must be virtualized. Wondering if it were just me, or if other ops professionals are feeling a degree of skepticism too. – bryan hunt Jun 10 '15 at 21:42
  • 2
    If you found the performance to be GENERALLY disastrous then - after 10 years of working with virtualization - it is time to accept defeat in being an admin and look for something you can do, when the rest of the world has no problems doing that. Seriously, I think you dramatize here. Or never realizes that virtualization needs buying a decent IO subsystem. I always find if horrific when people load 10+ virtual machines - on a desktop disc and complain about IO speed. – TomTom Jun 11 '15 at 05:33
  • It's more of an observation. I don't always have a say in the solution that is chosen. I'm ops. I just keep whatever gets thrown at me up and running. – bryan hunt Jun 11 '15 at 13:22

3 Answers3

19

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads?

Yep, very sane indeed, in fact for most organisations now virtual is the default and doing things on physical boxes is the very much the exception. We have over 100k VMs of all forms and many of them are >40k IOPS with no issue at all.

What are the best practices around this sort of stuff?

The key thing here isn't whether it's virtualised or not - it's understanding your IO needs well and matching the virtual storage resources. It's that simple, if you know what you need/want and have the budget to match that with your storage systems then the virtualisation layer really plays little or no part - unless you're REALLY pushing things of course (I'm talking tens/hundred of millions of IOPs).

What causes these problems, are there well known system bottlenecks, or is just a question of excessive contention?

Lack of understanding or trying to do too much with too little storage resources, that's what normally causes people problems.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
10

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads?

Does a database server regularly pulling 1gb/second random IO count? Have one here.

Or a virtual file server delivering up to 600mb/second to a HPC cluster. That one is running off 8 Velicoraptors in a Raid 10, dedicated.

What are the best practices around this sort of stuff?

Provide plenty of IO. I think this SQL VM has around 8 or 10 dedicated SSD.

What causes these problems, are there well known system bottlenecks,

People not doing basic math. If the IO subsystem is not capable of handling the load, it also will not do so under virtualization. Need a LOT of IO - then provide a dedicated storage subsystem of appropriate size.

TomTom
  • 50,857
  • 7
  • 52
  • 134
2

Besides the basic math & concept that you still need the same IOs as non-virtualised, there is also QOS/prioritisation. Most virtualisation platforms offer at least a basic support for this, will help out a lot to prevent the misbehaving dev VM stalling your prod DB.

Milo de Vries
  • 79
  • 1
  • 3
  • 1
    Yeah. Need 5000IOPS then a hard disc with 350 will not cut it. Load more vms on the same disc and no, it will not magically work. I am always astonished how many people ignore basic math. – TomTom Jun 10 '15 at 12:45