2

This is a follow up to things I've written down in an answer to another question:

How does Windows Image Backup decide which files to backup?

Windows/NTFS manages change journals on each volume, keeping track of which file has changed, how it has changed and because those are part of all NTFS volumes by default, they are available in the VHD(X) of the backup target as well. From my understanding, wbengine simply compares those journals to know which files need backup. This makes it easy to support different backup targets without caring about archive bits of files or stuff like that, because those change journals are bound to volume-unique file-IDs and those IDs are exactly the same in the backup target:

C:\>fsutil file queryfileid "C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
Datei-ID: 0x000000000000000000080000000ea5f1

C:\>fsutil file queryfileid "D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
Datei-ID: 0x000000000000000000080000000ea5f1

In the above example, C:\ is my current system volume while D:\ is the mounted last backup on one of the two available backup targets. Even file sizes, timestamps etc. are all the same:

C:\>dir /A "C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
 Datenträger in Laufwerk C: ist System
 Volumeseriennummer: 266B-2863

 Verzeichnis von C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin

03.02.2016  17:08           560.640 perlapp.exe
               1 Datei(en),        560.640 Bytes
               0 Verzeichnis(se), 1.337.040.216.064 Bytes frei

C:\>dir /A "D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
 Datenträger in Laufwerk D: ist System
 Volumeseriennummer: 266B-2863

 Verzeichnis von D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin

03.02.2016  17:08           560.640 perlapp.exe
               1 Datei(en),        560.640 Bytes
               0 Verzeichnis(se), 1.281.302.233.088 Bytes frei

By using this approach, backup can decide at any time with any older VHD(X) which files need to be backed up, as long as the current journal and the one in the image have something in common, which are entry-IDs in my understanding. Without such a shared ID, e.g. because too many I/O happend productive and the backup is too old, wbengine would simply do a full backup instead of an incremental one.

Using those journals makes it as well pretty fast to know which files to backup, because one doesn't need to iterate the whole tree of files. That's what one actually sees in practice as well: Backup seems to know pretty soon which files to backup and starts processing those instead of iterating a file tree.

Behaviour in case of network shares as backup target for Windows Server 2012+.

In older versions of Windows, wbengine seems to have always recreated VHD files, making it pretty incompatible with snapshots supported by some underlying file system like BTRFS or ZFS. Additionally, each and every backup was a full one, simply because wbengine didn't have access to any former backup to compare with.

Things seem to be a little bit different with newer versions of Windows: I'm somewhat sure that one customer of mine ran into troubles with wbadmin and Windows Server 2012 and during debugging those using Process Monitor, I verified that existing VHDX files were kept instead of deleted and recreated. I've tested this right now with Windows Server 2019 again and multiple invocations of wbadmin led to successful backups while KEEPING inodes of VHDX files the same:

root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-02 200024"
total 549063256
 271         0 d---------+ 1 user group          594 Nov  2 20:58 .
 266         0 d---------+ 1 user group          108 Nov  2 20:58 ..
 273 507813736 ----------+ 1 user group 520061190144 Nov  2 22:02 165c4b13-8376-4c55-9643-a8c1b2c17d6b.vhdx
3814         4 ----------+ 1 user group          776 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
3816       440 ----------+ 1 user group       450488 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_Components.xml
3813         8 ----------+ 1 user group         6212 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_RegistryExcludes.xml
3815         4 ----------+ 1 user group         1192 Nov  1 22:47 BackupSpecs.xml
 272  41249064 ----------+ 1 user group  42289070080 Nov  2 22:03 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx

root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-03 132201"
total 549318872
 271         0 d---------+ 1 user group          594 Nov  2 20:58 .
 266         0 d---------+ 1 user group          108 Nov  3 14:19 ..
 273 507813736 ----------+ 1 user group 520061190144 Nov  3 14:19 165c4b13-8376-4c55-9643-a8c1b2c17d6b.vhdx
3814         4 ----------+ 1 user group          776 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
3816       440 ----------+ 1 user group       450488 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_Components.xml
3813         8 ----------+ 1 user group         6212 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_RegistryExcludes.xml
3815         4 ----------+ 1 user group         1192 Nov  1 22:47 BackupSpecs.xml
 272  41504680 ----------+ 1 user group  42289070080 Nov  3 14:21 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx

root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-03 133308"
total 41262660
 271        0 d---------+ 1 user group        2504 Nov  3 14:44 .
 266        0 d---------+ 1 user group         108 Nov  3 14:30 ..
3851        4 ----------+ 1 user group         776 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
3853      440 ----------+ 1 user group      450488 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Components.xml
3850        8 ----------+ 1 user group        6212 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_RegistryExcludes.xml
3840        4 ----------+ 1 user group        3138 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer0bada1de-01a9-4625-8278-69e735f39dd2.xml
3848        4 ----------+ 1 user group        2284 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer2707761b-2324-473d-88eb-eb007a359533.xml
3844        8 ----------+ 1 user group        5386 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f.xml
3846        4 ----------+ 1 user group        1488 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer542da469-d3e1-473c-9f4f-7847f01fc64f.xml
3839        4 ----------+ 1 user group        1628 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer75dfb225-e2e4-4d39-9ac9-ffaff65ddf06.xml
3842       24 ----------+ 1 user group       21686 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writera65faa63-5ea8-4ebc-9dbd-a0c4db26912a.xml
3847        4 ----------+ 1 user group        1484 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writera6ad56c2-b509-4e6c-bb19-49d8f43532f0.xml
3845        4 ----------+ 1 user group        2940 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerafbab4a2-367d-4d15-a586-71dbb18f8485.xml
3849        4 ----------+ 1 user group        1850 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerb2014c9e-8711-4c5c-a5a9-3cf384484757.xml
3843        8 ----------+ 1 user group        6048 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerbe000cbe-11fe-4426-9c58-531aa6355fc4.xml
3838        4 ----------+ 1 user group        1746 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerd61d61c8-d73a-4eee-8cdd-f6f9786b7124.xml
3841    13068 ----------+ 1 user group    13379228 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writere8132975-6f93-4464-a53e-1050253ae220.xml
3852        4 ----------+ 1 user group         758 Nov  3 14:44 BackupSpecs.xml
 272 41249064 ----------+ 1 user group 42289070080 Nov  3 14:44 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx

So in theory this should allow incremental backups replacing files in-place and efficient snapshots of e.g. underlying BTRFS or ZFS at the same time. Looking at storage of the snapshots of the tested Windows Server 2019, at least some of those really share some space:

    Total   Exclusive  Set shared  Filename
424.41GiB   417.69GiB     6.72GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.29-00.00.09
446.53GiB   400.16GiB    46.37GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.30-00.00.08
483.05GiB   448.36GiB    34.70GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.31-00.00.08
553.78GiB   209.26GiB   344.52GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.02-00.00.09
204.68GiB   204.63GiB     0.05GiB  /volume1/@sharesnap/[...]/GMT+02-2019.10.01-00.00.07
410.24GiB   405.37GiB     4.87GiB  /volume1/@sharesnap/[...]/GMT+02-2019.10.27-00.00.06

It's not as much as I would expect, though. The thing is that it might be that not much space is shared even though VHDX files are reused, because when I subsequently invoke backing up C: of that server, backup doesn't seem to get faster.

The first backup might take longer as much data has changed, but after that is finished and with the server doing nothing, a second backup only a few minutes later should be a lot faster, because of only backing up differences of the files. But that doesn't seem to be the case, instead it seems to take the same time like before. Additionally, while BTRFS can share differences in created snapshots between different invocations of wbadmin with the exact same command line, those are much smaller than expected when really only backing up changed files:

root@[...]:~# btrfs filesystem du -s --gbytes /volume1/@sharesnap/[a-zA-Z]*/GMT*
     Total   Exclusive  Set shared  Filename
 446.53GiB   400.20GiB    46.33GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.30-00.00.08
 483.05GiB   448.36GiB    34.70GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.31-00.00.08
 553.78GiB   546.54GiB     7.24GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.02-00.00.09
  39.35GiB    37.68GiB     1.67GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-15.36.52
  39.35GiB    31.18GiB     8.17GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-15.49.03
  39.35GiB    37.98GiB     1.37GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-16.03.06
 410.24GiB   409.72GiB     0.52GiB  /volume1/@sharesnap/[...]/GMT+02-2019.10.27-00.00.06

That is different to what I see when backing up to my USB-disks, subsequent backups are much faster if nothing has changed. What is interesting is that others seem to be not so sure about how things behave on network shares as well:

If you create a scheduled backup job to network shared folder or a mapped network drive, all the backups will only be performed by full backup because network location is not a volume. If you need to create differential backup or incremental backup to network folder, you need to third party backup software.

https://www.ubackup.com/windows-server/windows-server-backup-differential-4348.html

While the same people write the following, which doesn't make much sense:

Tip: You can also use WBADMIN to create incremental backup to network share, like this:

wbadmin start backup –backupTarget: \backupshare\backup1 -allCritical -include:C: -vssFull –quiet

https://www.ubackup.com/articles/wbadmin-incremental-backup-5740.html

In case of backing up to shares, if VHD is recreated always, like seems to be the case for older versions of Windows, one doesn't get incremental backups. But even though VHDX files are kept in newer versions of Windows, it seems like only backing up changed files still doesn't work the same way like it does when using USB-disks.

Influence of -vssFull and -vssCopy.

Using -vssFull vs. -vssCopy didn't make any difference in my tests and according to my understanding is not even intended to do so. Those arguments are only relevant to 3rd party software and don't influence which files are backed up how. Reasons to believe so are documented in my former answer:

Influence of -vssFull and -vssCopy.

https://serverfault.com/a/990394/333397

Backup behaviour of Server 2016.

One of my customers is using Server 2016 using wbadmin as well and that shows a slightly different behaviour in backups running once each day. Looking at log files and how long wbadmin runs, it seems that in this case really incremental backups of C:\ are created, because most of the backups are fast and snapshots of BTRFS share a lot of space:

root@[...]:~# btrfs filesystem du -s --gbytes /volume1/@sharesnap/[a-zA-Z]*/GMT*
     Total   Exclusive  Set shared  Filename
 388.12GiB     4.29GiB   383.83GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.01-00.00.05
 389.34GiB     1.60GiB   387.73GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-00.00.05
 389.64GiB     1.39GiB   388.25GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.04-00.00.04
 330.22GiB     0.00GiB   330.22GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.05-00.00.06

most is the keyword here, because it seems that pretty reliably every ~14 days backups take around 3 times longer, which seems to be a full backup again. That would be in line with what I read somewhere that Server Backup does full backups every 14 days automatically and afterwards only incremental ones. But that wouldn't explain why my Server 2019 doesn't seem to behave that way.

The only difference in the command line used I can see is -allCritical -systemState being used for Server 2016, while Server 2019 uses -include:c:,d: instead for some reasons. In any way, both were using -vssCopy, so like expected, using that or -vssFull shouldn't make any difference.

Questions

Does wbengine of Windows Server 2019 support incremental backups when targeting network shares? If so, under which conditions, like command line arguments? Does anyone see any improvement in backup times and storage allocations in case of snapshots?

Community
  • 1
Thorsten Schöning
  • 643
  • 5
  • 16
  • You mentioned "fsutil file queryfileid" but it works [only on Windows Server 2008 R2 and Windows 7](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/fsutil-file#parameters) – Swisstone Nov 04 '19 at 08:05
  • @Swisstone Not sure what you are trying to tell, `fsutil file queryfileid` works on Windows 10 as well, that's where I tested the mentioned invocations yesterday. The statement in your linked docs surely is meant to be like `Windows Server 2008 and Windows 7 and above`. The docs of MS are full of sentences like these, you can easily try it yourself. – Thorsten Schöning Nov 04 '19 at 09:28
  • Not sure if this is coincidence, but I recognized some days ago that (automatic) snapshots on the Server 2019 have NOT been enabled for the internal volumes. So I did that and starting with the next backup those are a LOT faster now for the last few days already. I've simply enabled those automatic snapshots with default quotas, but changed to be created once a day instead of once per weekday. That's it, `wbadmin` needs far less time now, log messages about copied data are a lot shorter now. I wouldn't have expected any influence of those scheduled snapshots on `wbadmin`. – Thorsten Schöning Mar 13 '21 at 09:30

1 Answers1

0

Yes, it does, though one might need to enable creation of automatic snapshots on the server. I've setup multiple of those since creating my question and all of those behave as expected with long initial backup and pretty fast ones afterwards, depending on the amount of changed data. In all cases I'm backing up to some Synology NAS using SMB.

Not sure if this is coincidence, but I recognized some days ago that (automatic) snapshots on the Server 2019 have NOT been enabled for the internal volumes. So I did that and starting with the next backup those are a LOT faster now for the last few days already. I've simply enabled those automatic snapshots with default quotas, but changed to be created once a day instead of once per weekday. That's it, wbadmin needs far less time now, log messages about copied data are a lot shorter now. I wouldn't have expected any influence of those scheduled snapshots on wbadmin.

Thorsten Schöning
  • 643
  • 5
  • 16