13

If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH.

I'm trying to understand why most (all ?) backup of vmware guests is not done this way.

If the VM is running, you can simply use 'vim-cmd vmsvc/snapshot.create' to create a snapshot and then rsync that snapshot to the remote host. (there's even an option to "quiesce" the snapshot)

OR, if you want a more robust backup, you can gracefully halt the VM and rsync over the vmdk file(s).

So ... it seems like I am a simple shell script away from all the backups I ever wanted to do, simply and easily, using plain old rsync.

What am I missing here ?

user227963
  • 199
  • 1
  • 2
  • 11
  • 1
    Because if a single file changes in the VM you will have to backup the whole vmdk? – faker Jun 25 '14 at 22:14
  • No, rsync will update a single file efficiently with just the changes since the last transfer. Certainly the operations of the VM could produce a LOT more changes than you expect it to, but it's not going to make you resend the entire vmdk ... – user227963 Jun 25 '14 at 22:47
  • Other than the fact you shouldn't use the esxi shell for anything other than maintenance, the esxi OS is not made to work in that way, and you would be unsupported, I think you are misunderstanding the concept of a snapshot. The snapshot in this case is a delta. So if you take a snap and copy it straight away, it would be tiny, and contain almost no information. You are thinking of a backend storage snapshot, and yes people back up VMs in this way – Rqomey Jun 27 '14 at 12:08
  • 1
    @Rqomey - there are different kind of "snapshots" in ESXi. You're talking about the one kind that is visible via the vSphere Client - but using the API you got other options, e.g.: full clone. – omni Jun 27 '14 at 15:59
  • @MASI Do you mean a clone then as opposed to a snapshot? ;) – Rqomey Jun 30 '14 at 10:27
  • Ok, what about using the command 'vim-cmd vmsvc/snapshot.create' ... isn't that a "full" snapshot ? That is to say, regardless of how slow or painful or inconvenient it might be to get that snapshot off of the ESXi system, that *is* a usable backup, yes ? – user227963 Jul 01 '14 at 17:15

4 Answers4

35
  • Because the transfer speeds out of the ESXi console are purposefully limited.
  • Because this isn't scalable in any way.
  • Because you'd have to drop a statically-compiled rsync binary onto the ESXi host.
  • Because the VMs, the VMDKs, their ramdisk files and other components can change enough to make rsync a losing proposition... do you really want to re-sync a 200GB VM that was rebooted and had a small number of files change?
  • Because of CPU/memory resource requirements on the source or destination. Rsync isn't free.
  • Because there are other products on the market, both third-party and VMware-provided. Look up Changed Block Tracking.
  • Because ESXi is NOT a general-purpose operating system.

Also see: Install rsync on VMware ESX 4.1 server

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • 1
    Outstanding answer. – EEAA Jun 25 '14 at 22:24
  • Ok, trying to get a better understanding here ... transfer limits out of ESXi console are definitely an issue - where is that set and can it be changed ? – user227963 Jun 25 '14 at 22:27
  • 1
    @user227963 Not a problem... but VMware kinda went out of their way to make ESXi unfriendly to this. In the early days before ESXi (ESX), I *did* do things like this. I'd drop `rsync` from CentOS 4 onto my hosts and try to treat ESX like a Linux box. That's just not the accepted practice anymore. – ewwhite Jun 25 '14 at 22:30
  • Ok, but I am confused as to why ghettoVCB is still a viable, popular option ? It basically does what I am asking, but simply copies the files out to NFS share, instead of rsync over SSH ... how are people still making ghettoVCB work ? – user227963 Jun 25 '14 at 22:31
  • 4
    They aren't... I mean, it's in the name: ***ghettoVCB***. There are better solutions out there. Veeam, vSphere Data Protection, etc. – ewwhite Jun 25 '14 at 22:34
  • Yes, but it's ridiculous to add all of that bloat (and expense) onto what is, essentially, nothing more than copying some files from one unix filesystem to another. This should be the simplest thing in the world... – user227963 Jun 25 '14 at 22:35
  • 2
    You could certainly use the rsync method if you switch over to xen/kvm. – Zoredache Jun 25 '14 at 23:01
  • 9
    @user227963 Rsync is also rather inefficient on both - large number of files as well as large files. And while it might not have to *resend* the entire file over the wire, it will have to *reread* it on source and destination alike. CBT will help you here, but rsync knows nothing about CBT. – the-wabbit Jun 25 '14 at 23:04
  • 2
    @user227963 copying files is simple. Now make it fast and not a resource hog on large files with small constant changes. rsync is decent but nowhere near the performance of anything with insider info on which blocks changed. – JamesRyan Jun 26 '14 at 09:29
  • @jamesryan Ok (and excuse my learning curve here) ... what you are saying is, if rsync had knowledge of CBT, it would be a viable option ... is that the inside information on the changed blocks you're referring to ? Thanks. – user227963 Jun 26 '14 at 20:06
  • @user227963 depending on how well that is implemented, yes – JamesRyan Jun 27 '14 at 08:55
  • 1
    Some of the answers ewwhite gives do not apply for the average DIY/beigebox experiments, namely the fact of dropping the static binary or the limits from the console (which you can workaround setting it up as a daemon, which I did). The real problem is the one @the-wabbit noticed: the fact that it has to re-read all files again. I did set up ghettoVCB (directory names VMName-0 and VMName-1 instead of VMName-timestamp) and rsync as daemon, and the actual speed achieved was 9MB/s, as if I just transferred the whole backup files. – Alex Mazzariol Oct 09 '14 at 10:41
4

I used to do just this a few years back. (edit: with VMWare running on CentOS hosts, not ESXi admittedly)

Every night I had a script that would suspend a VM, rsync the files from disk to the backup server and then start the VMs again. It worked quite well except...

Rsync doesn't work very well with a 2GB file.

Its not because rsync isn't brilliant, it more that each 2GB vmdk file changes in ways that are very opaque to rsync, even small changes to the enclosed filesystem produce changes in the vmdk (or all vmdks for some reason) which I blamed on Windows, either automatically defragging or otherwise doing all the other things it does that don't matter if you're running a real system, but show up when you are trying to rsync a VM!

I think the rsync mechanism for detecting changes don't work very well on a 2GB file, whilst it quite often skipped chunks of the start of the vmdk, once it started to find a difference it would simply copy the rest of the file. I don't know if that's an issue with rsync not being able to detect a moved chunk of binary data, or with a lack of memory on the source box, or whether the vmdk just got updated all the way through. It doesn't matter as the result wasd the same - majority of the vmdk got copied.

In the end I simply copied any changed files and overwrite them, still using rsync. I also had better performance simply overwriting the backup file instead of letting rsync copy and replace what was there.

Our backup server wasn't the fastest either and it got to the point where overnight wasn't long enough to back up all running VMs.

However, when we did need to restore a VM, it was really easy and worked beautifully.

gbjbaanb
  • 3,852
  • 1
  • 22
  • 27
  • Ok, that's very helpful. I know a bit about how rsync works, and I can tell you it has nothing to do with the size of the file - but what you are describing is that much more of the file changes than you expect it to ... that is to say, you run the VM for a day, and you only do a few little things with it, and then you stop it ... but the vmdk file changed by 30-40% (even though you did very little). So rsync would do just fine, it just has a lot of work to do ... more than you expected. Thanks! – user227963 Jun 26 '14 at 16:11
  • 1
    But then ... the question this raises ... how do the "professional" tools do it ? What kind of magic are they doing that is somehow more optimal than what rsync (or scp, or even cp) would do ? At the end of the day, you have a unix environment (the ESXi console) and you want to move a file in or out of it ... what secrets could there be involved with that ? – user227963 Jun 26 '14 at 16:14
  • @user227963 The professional tools leverage features like the changed-block-tracking or have access to other vSphere or ESXi APIs. – ewwhite Jun 27 '14 at 12:44
3

Rsyncing a single file is not a backup solution,

what do you do when something happened to the vm and files were deleted, but you only noticed this after your rsync has run again? You will have overwritten the good 'backup' of your files with the bad image now.

If you want backup you need to keep the old versions somewhere, or the diff's. Rsync will only copy over the diffs for you, but it will not store only the diff's, but overwrite the previous file.

There might be options for you here, with rsync, and a copy-on-write filesystem with versioning information, which will in effect store the diffs every time your rsync script runs. This solutions starts to get a bit more complicated already, so this is why people resort to known working solutions imho.

Jens Timmerman
  • 866
  • 4
  • 10
  • There is certainly a lot more complexity involved here than I originally thought, but what you're mentioning is not a problem. Certainly if you blindly ran rsync over and over you would run into trouble, as you suggest, but there are plenty of simple ways to clone/rotate rsync-created backups (even single-file ones) ... that problem was solved a long time ago, thankfully. – user227963 Jul 01 '14 at 17:17
1

There isn't any reason why you can't use Rsync in an ESXi server. We offer a statically compiled version here https://33hops.com/download-form.html that works very well (just unfold the software list). There's information on how to compile your own too in the link below.

Nevertheless anybody willing to use it must take on account that Rsync and its Delta algorithm weren't thought to backup huge fixed lenght sparse files, like VM hard disks, but to sync variable length smaller files. So, it does work, but it takes a lot of time and CPU to calculate the diff data. In fact it's just a way to exchange bandwidth by CPU. In any case, it's still quite workable, specially if your virtual disks are in the order of a few tens of gigabytes.

I have published a full post on the topic here, detailing all pros and cons Rsync Considerations copying ©ESXi VM

Daniel J.
  • 214
  • 1
  • 5
  • 1
    That's very kind, but that version crashes with `out of memory in receive_sums [sender]` on large files… 3.1.1 seems to fix it. – Pierre-Luc Paour Apr 13 '20 at 07:44
  • Well, as stated, Rsync wasn't though to sync huge files, we offer other binaries specially designed for that purpose. You have all the details to compile a static binary from some more recent source code, in any case using Rsync to sync virtual disks over some tens of gigabytes in size can be really slugish – Daniel J. Dec 06 '20 at 17:22