5

Since upgrading to Backup Exec 2012 we have been experiencing a significant increase in backup times. The data throughput is about where it was before, the problem seems to be random delays between each server's backup.

When the delay occurs, we'll see the snapshot creation event on the ESX host, then there is an indeterminate amount of time where nothing is happening, then we see the "Symantec Backup" event on the host and the IO starts between the media server and the SAN. The delays can happen at any time, with any server, and there can be more than one server which experiences a delay during any given backup.

The delays seem to be independent of VM server, their config or load, or the host load, or media server activity (this is the only backup job active at the time).

Yes, I've raised this issue with Symantec technical support; let's just say they've been less than helpful so far (I'll resist the temptation to rant). I'm just wondering if anyone else had seen, and hopefully resolved, this issue.

EDIT:
I've had some activity from Symantec support on this recently. No resolution as yet, but they are now maintaining a dialog

Systems Overview
Media server:

  • HP ProLiant DL360 G5 (E5420, 6GB RAM)
  • Tape Drive - Ultrium 1840 SAS
  • 2 x 1Gb NICS for iSCSI to the SAN
  • 2 x 1Gb NICS teamed for the network
  • Windows 2008 R2
  • Backup Exec 2012 with VMWare agent, and RAWS on the VM servers for GRT.

VMWare:

  • VMWare vSphere 4.0

Storage:

  • Equalogic PS6000
  • Connection to ESX hosts & backup server via multipathed 1Gb

The backup was created by selecting the servers in vCenter so that it is one job backing up multiple servers sequentially.

Here is a visualization of the delays: The green areas are where the data is actually going from the SAN to media server (and are fairly consistent), the red areas are the delays where there is no activity.
Note: The shorter backup time of Utility1 on 1/12 was because it was cancelled. Also, there are more servers after these but are omitted for clarity.

Backup Delays

Chris
  • 945
  • 7
  • 17
  • How was the graph generated/where did the data come from)? Is it from backup exec report? Of base on network/disk/san activities? – John Siu Jan 22 '13 at 02:55
  • It's actually made with Excel. I exported the job timings from the backup exec database via SQL and then ran a macro to set the row heights proportional to the time. It was the best way I could come up with to visualize it. – Chris Jan 22 '13 at 13:04
  • Am I correct in assuming that the backup traffic is sent from a kernel port on the ESXi host over the media server? How are the vmware servers communicating with the media server? – SpacemanSpiff Jan 22 '13 at 17:09
  • @Chris: VSphere has excellent performance charts. Why not screen shot one of those for a week period on storage/disk? Here is an example: http://i.imgur.com/MI4Rkwg.png – Greg Askew Jan 22 '13 at 17:30
  • @SpacemanSpiff - I don't know about the kernel port on the host. There are three hosts total, only one of which will be communicating with the media server at any one time. – Chris Jan 23 '13 at 17:32
  • @GregAskew - I'm not sure what that would show me in this context as there is no I/O. – Chris Jan 23 '13 at 17:35
  • @Chris are you have to use symantec backup exec as a backup server for VMs? because symantec backup exec is not very useful for VMs backup? have tried Veeam? this software is specific for VMs and more compatible than backup exec – Hamed Shahinnejad Feb 05 '14 at 06:28

2 Answers2

0

First do you have Backup Exec updated to all the recent patches and RAWS pushed out to your servers?

Does Backup Exec reside on a Virtual machine? In the VMware job properties is the backup running under the SAN transport mode?

Lenora (Backup Exec Support Tech)

Lenora Moss
  • 24
  • 2
0

do you have a support case number? If you do I will review the case notes to get more info and not replicate questions/answers.

Lenora Moss
  • 24
  • 2
  • I think it best not to have two lines of communication with symantec open about this, with one of them being via serverfault; don't you agree? If you wanted to review the case, that's fine, but this is not the forum for us to communicate about it. – Chris Jan 30 '13 at 16:02
  • I agree Chris, I want to assist in resolving this. Please email me your contact information. – Lenora Moss Jan 30 '13 at 22:41