4

Recently our MySQL server has been "going away" (ie. the client connection drops out). After weeks of trying different things (like adjusting packet size), we've discovered that it's our Veeam imaging backups which use the VMWare API to snapshot and copy the vmdks etc.

We are using ESXi 5 with a Centos 6.4 guest, running (pretty much) only MySQL 5.1.69-log.

The change which seemed to initiate this problem was increasing the physical disk size to 300GB, from about 100, and resizing the guest filesystem to use most of the new capacity. Ever since the disk was increased, we've been getting these problems during backups - presumably due to the increase time it takes to perform snapshot related functions.

The new disks are 2x300GB Gen8 15k SAS in RAID1. The old disks would have been similar only smaller. The target of the Veeam process is a ReadyNAS over a 1Gb dedicated ethernet (i.e. separated from general office traffic).

The host is an HP DL380P tower:

==server spec (BASE CHASSIS)==
SERIES DL380P GEN8
PROCESSOR TYPE Intel Xeon E5-2609 v2 (2.5GHz/4-core/10MB/6.4GT-s QPI/80W)
NUMBER OF PROCESSORS 2 
MEMORY 80GB
INTERNAL DRIVE BAYS 8 SFF HDD Bays
COMPATIBLE HDD SFF SAS/SATA
HARD DISK CONTROLLER SMART ARRAY P420I/ZERO MEMORY CONTROLLER (RAID 0/1/1+0)

My "IT guy" has made a few tweaks to the Veeam config including skipping empty blocks (the majority of the new disk is empty), but this didn't seem to help at all.

Veeam haven't been much help either, saying "reboot the target" or "we just use VMWare APIs".

I believe the "stun" means the virtual machine simply freezes for a time (around 30s) then continues normally.

VMWare.log example:

Line 7411: 2016-06-08T17:11:44.910Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 21068381 us
Line 7556: 2016-06-08T17:22:24.608Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 19819322 us
Line 7700: 2016-06-08T17:22:30.140Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 1130044 us
Line 7929: 2016-06-08T17:23:08.616Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 30197618 us

So my problem has two likely solutions:

  1. Is there a way to prevent or reduce the "stunning" of a VMWare guest during imaging.

  2. Is there a way to reduce the impact of the stun onto MySQL or the virtual network or Centos.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
scipilot
  • 201
  • 2
  • 7
  • What's your underlying storage? – ewwhite Jun 09 '16 at 09:32
  • Good point - I've updated the question: 15k SAS RAID1 in the host. – scipilot Jun 09 '16 at 11:28
  • So, two disks? Hardware RAID? Do you have a write cache on the RAID controller? – ewwhite Jun 09 '16 at 11:29
  • Yes two disks, hardware RAID, but looking at the specs it sounds like no cache... if "zero memory controller" means that. I'd need to query this to confirm. – scipilot Jun 09 '16 at 11:41
  • Which disks did you increase? The virtual hard disk for the VM or the physical hard disk in the host? – joeqwerty Jun 09 '16 at 11:43
  • @joeqwerty Both, question updated. – scipilot Jun 09 '16 at 11:48
  • Do the backups take a snapshot of the guest memory? Do the backups quiesce the file system of the guests? – joeqwerty Jun 09 '16 at 11:52
  • @joeqwerty I don't believe so, (other than any swap which will be in the vmdk). I don't know the answer to the second question - I'd have to find out. – scipilot Jun 09 '16 at 11:56
  • @scipilot Is this an HP server? – ewwhite Jun 09 '16 at 12:04
  • @ewwhite Yes, HP DL380P tower, I updated the question with specs. – scipilot Jun 09 '16 at 12:07
  • @scipilot so did you ever come up with which of these actually fixed the problem? An answered question is very useful to the stack. – Rowan Hawkins Jan 04 '18 at 23:55
  • @RowanHawkins sadly I never got a clear result nor solution. Ordering the cache got put on hold, then my IT support/supplier changed and now someone else has taken over managing the system both making various changes. It's difficult to track. We still have problems at certain regular times at night, from network dropouts to application level freezes. But honestly I give up trying to get a straight answer from the support company. We're in control of our software so it is now resilient enough to cope. So I don't feel I can accept either answer honestly. I understand the need. – scipilot Jan 05 '18 at 04:18

2 Answers2

7

This is an HP ProLiant server running with a Smart Array RAID controller without a Flash-backed cache module.

As a result, you have no write cache (or read cache), and operations like snapshots of virtual machines will suffer. You've experienced the effect of this. The current configuration is unsuitable for most workloads, especially virtualization.

Your best option is to simply buy a cache module and battery/FBWC; HP parts 631681-B21, 631679-B21, or 631069-B21.

This will accelerate performance and eliminate the problem you're seeing.

Also see:

FBWC and Zero Memory (ZM) RAID Controller on HP DL360p

BBWC: in theory a good idea but has one ever saved your data?

What is the memory module on a RAID card needed for?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Ok that sounds quite likely, seeing as we pushed it "over the edge" with the expansion, it feels like a performance issue. I'll check with my guy... I will accept your answer if this solves it - it may take some time. – scipilot Jun 09 '16 at 12:10
1

Answering my own question from research. (I will only accept my own answer if one of these approaches actually works and it's before someone else's suggestion.)

This (older) article WHAT ARE THE DANGERS OF SNAPSHOTS AND HOW TO AVOID? mentions a few possible causes and three preventative measures. Interestingly it mentions how the issue similarly affects MS SQL Server and other server products.

If you do not want to stun / pause the virtual machine you can set snapshot.maxIterations to 20 (or higher). This means vSphere will do more tries (iterations) to commit the snapshot files. More information in this KB article.

It then goes on to describe the risks and downsides of this approach.

Secondly it suggests:

Alternatively you can set snapshot.maxConsolidateTime to 60 seconds. This means you can accept a pause of the virtual machine for 60 seconds to do a synchronous consolidate. This is often a better option than wait for the snapshot file grow so big the virtual machine will require to be stunned for a much longer time.

But I do not know the different between "stun" and "pause".

And lastly:

ESXi 4.1 has a update which added parameter snapshot.asyncConsolidate.forceSync = “FALSE” which needs to be added to the VMX file. This setting disables synchronous consolidate and the virtual machine will never be stunned. More info in this KB.

It doesn't describe the potential drawbacks with these solutions, but I'd presume there are some, else they'd be default.

I haven't yet checked if these parameters or solutions are still relevant in v5.

UPDATE: Veeam have recommended we make the above mentioned changes as listed in this KB which is relevant to v4 and v5 of ESXi.When removing a snapshot virtual machines become unresponsive for over 30 minutes (2039754)

UPDATE2: We are making these configuration changes tonight and rebooting the host, as it's cheaper and quicker than waiting for the cache. We will then monitor for a few days to see if this alone resolves it for us.

scipilot
  • 201
  • 2
  • 7
  • 1
    What version of ESXi are you running? – mfinni Jun 09 '16 at 12:17
  • 1
    Version 5. I'll confirm the minor and update the question... (can't believe I missed so many details!) – scipilot Jun 09 '16 at 12:28
  • 1
    The higher the version of ESXi, the better performance you'll see around snapshots. Don't spend too much time reading KB articles and blog posts that refer to major versions behind yours because they likely won't apply. That being said, Ed is almost definitely on the money pointing out the deficit in your IO performance. – mfinni Jun 09 '16 at 12:30
  • Good point. When I read it, I thought the parameter might still be around, but you're right - those posts might be talking about problems which have been resolved entirely in major updates. – scipilot Jun 09 '16 at 12:38
  • 1
    @mfinni it's a shame that VMware has a lot of "internet pollution", with so much information pointing to outdated releases and versions still being top search engine results. – ewwhite Jun 09 '16 at 13:20
  • Come on Ed - I'm sure you've got your share of customers still on 4.1 :-) – mfinni Jun 09 '16 at 15:00