Robocopy /xd with wildcards and sub-paths

Question

I don't think this exact question has been asked before (I looked).

My backup server runs a script using Robocopy, and I want to exclude files that don't really need to be backed up, specifically:

*\AppData\Local\Google 
*\AppData\Local\Microsoft\Windows 
*\AppData\Local\NVIDIA 
*\AppData\Local\Temp 
*\AppData\LocalLow\Google

(all those have a million tiny cache files that take lots of time to parse but aren't needed for restore; most of the incremental backup run time is processing these)

The path prior to "\AppData..." varies, depending on the machine name, user name (multiple users per machine) and folder structure on that particular machine (not everyone has all their data on c:).

To keep things simple, I've got an environment variable called _exclude that I use in each call to Robocoy:

SET _exclude= /XD *AppData\Local\Google *AppData\Local\Temp [...etc]

I don't want to exclude every directory that contains "AppData" or "Local" or "Google", as that would exclude valuable data. Just those particular sub-paths.

I've tried it with and without leading '\', with and without double quotes around the paths, with/without '*', and permutations of those.

Is there any way to make this work with Robocopy, without hard-coding the particular path for every user on every machine?

Answer 1

Robocopy definitely won't do what you're looking for on its own.

Initially I thought about telling you to write a script that dynamically generates the exclusion list, but you're likely to run into command-line length limitations.

With that in mind, I think you'd be best off writing a script to create a Robocopy "job" file. Here's a script to create a simple job dynamically based on the contents of C:\Users:

@echo off
set DEST=\\SERVER\SHARE
set SRC=C:\Users
set RCJ=%RANDOM%.RCJ

echo /DD:%DEST%>%RCJ%
echo /SD:%SRC%>>%RCJ%
echo /IF>>%RCJ%
echo *>>%RCJ%
echo /S     :: copy Subdirectories, but not empty ones.>>%RCJ%
echo /E     :: copy subdirectories, including Empty ones.>>%RCJ%
echo /COPY:DAT  :: what to COPY for files (default is /COPY:DAT).>>%RCJ%
echo /PURGE     :: delete dest files/dirs that no longer exist in source.>>%RCJ%
echo /MIR       :: MIRror a directory tree (equivalent to /E plus /PURGE).>>%RCJ%
echo /R:1000000 :: number of Retries on failed copies: default 1 million.>>%RCJ%
echo /W:30      :: Wait time between retries: default is 30 seconds.>>%RCJ%
echo /XD        :: eXclude Directories matching these names>>%RCJ%

for /d %%i in (%SRC%\*) do call :_de "%%i">>%RCJ%

robocopy /job:%RCJ%
del %RCJ%
goto :EOF

:_de
echo %~1\AppData\Local\Google
echo %~1\AppData\Local\Microsoft\Windows
echo %~1\AppData\Local\NVIDIA
echo %~1\AppData\Local\Temp
echo %~1\AppData\LocalLow\Google

I got very cavalier in this script, going ahead and running robocopy and deleting the job file. I'd comment those lines out while you're testing, and probably add some checking of the errorlevel following completion of robocopy.

Answer 2

I'm sorry, but according to the documentation at https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy:

/xf <FileName>[ ...] Excludes files that match the specified names or paths. Note that FileName can include wildcard characters (* and ?).
/xd <Directory>[ ...] Excludes directories that match the specified names and paths.

So this specifically means that wildcard can be used in the /xf flag but not in the /xd flag.

Answer 3

However, Microsoft's documentation might be wrong.

And in this case, it apparently is. I like to hide folders from being copied by RoboCopy, so I prefix their names with '_'.

Then I add '_*' (without the quotes) after the '/XD' parameter to the Remote Job (.rcj) file.

That works! Microsoft apparently decided it was implied by the comments for the '/XF' parameter, & didn't think it was necessary to restate it.

Well, Micosoft, in a reference document, it IS necessary!