compare 2 directories and copy differences in a 3rd directory

Question

Running ubuntu 12.04, I want to compare 2 directories, say folder1/ and folder2/ and copy any files that are different to folder3/. There are also nested files, so matching subdirectories should be copied as well

Is there a single command that would help me? I can get the full list of changed files running:

rsync -rcnC --out-format="%f" folder1/ folder2/

But rsync doesn't seem to have the ability to "export" these files on a different target directory. Can I pipe the list to cp or some other program, so that the files are copied, while the directories are created as well? For example, I tried

rsync -rcnC --out-format="%f" folder1/ folder2/ | xargs cp -t folder3/

but that wouldn't preserve directories as well, it would simply copy all files inside folder3/

I'd probably use some combination of `diff -qr` and `awk` to build the list, then `rsync`... — ewwhite, May 08 '13 at 13:58
I'd either run a for-loop or use the `rsync --include-from=FILE` option. — ewwhite, May 08 '13 at 14:22
(don't be lazy...) But the [rsync solution](http://serverfault.com/a/508272/13325) below is elegant. — ewwhite, May 16 '13 at 22:37

score 49 · Accepted Answer · edited May 15 '14 at 11:56

49

Use --compare-dest.

From the man page:

--compare-dest=DIR - This option instructs rsync to use DIR on the destination machine as an additional hierarchy to compare destination files against doing transfers (if the files are missing in the destination directory). If a file is found in DIR that is identical to the sender's file, the file will NOT be transferred to the destination directory. This is useful for creating a sparse backup of just files that have changed from an earlier backup.

first check your syntax with --dry-run

rsync -aHxv --progress --dry-run --compare-dest=folder2/ folder1/ folder3/

Then once you're satisfied with the output:

rsync -aHxv --progress  --compare-dest=folder2/ folder1/ folder3/

this link has a good explanation of --compare-dest scope.

edited May 15 '14 at 11:56

Federico Paolantoni

103
4

answered May 16 '13 at 13:17

smithian

1,746
14
15

2

Thanks for this, I hadn't understood the man page explanation of this parameter. For anyone interested, I did it with `rsync -rcC --compare-dest=folder2/ folder1/ folder3/` – periklis May 17 '13 at 07:50
quick note for OsX users. I have `rsync version 3.1.1 protocol version 31` on my local and rsync doesn't understand `~/` paths for `--compare-dest` param. – Jevgeni Smirnov Jul 24 '14 at 14:41
6

For me, `--compare-dest` required an absolute path. – palswim Oct 13 '15 at 22:27
Didn't work for me with either author's or OP's options. I.e. it outputs `fonts/Makefile.in` file, which is identical in both `folder2` and `folder1` dirs. – Hi-Angel Dec 18 '16 at 08:18
1

@Hi-Angel My first attempt was similar, realized I missed the trailing `/` forward slashes. After adding them it worked as described. – dtmland Sep 09 '20 at 23:52

Laurentiu Roescu · Answer 2 · 2013-05-14T11:04:45.967

This little script should work:

folder1="firsts_folder/"
folder2="second_folder/"
folder3="target_folder"
for x in `rsync -rcnC --out-format="%n"  $folder1 $folder2`
    do
        if [ -d "$folder1/$x" ]; then
            mkdir -p "$folder3/$x"
        else
            cp -frv $folder1/$x $folder3/$x
        fi
done

But I also think that parsing output of diff -qr would be a better solution because it gives you path and name for each file consistently.

EDIT

This would be one command, your command slightly modified. But it will create folder1 under the folder3. And it does not work with full paths but also the script above does not work with full paths.

rsync -rcnC --out-format="\"%f\"" folder1/ folder2/ | xargs cp --parents -rfvt folder3/

I used \" in rsync to make it work with files and directories containing blanks. Tested on Ubuntu 10.04. Hope it helps.

thanks for this, I had hoped that a single command would exist though — periklis, May 08 '13 at 16:42

score 2 · Answer 3 · answered May 16 '13 at 22:24

rsync -rcnC --out-format="%f" . ../folder2/ |xargs cp --parents -rt ../folder3/

That's the simplest form I could come up with, if you're willing to first descend into folder1 before issuing the command.

This is the slower option, but I believe it meets your requirements, and it could be executed using non-relative directory names:

rsync -rcnC --out-format="%n" folder1/ folder2/ |grep -vP "/$" |xargs -I{} rsync -R folder1/./{} folder3/

I left your original rsync command alone, because you said it was giving you the correct file list (and my test files are obviously different than yours, so I couldn't say for certain that I'd be getting the correct file list if I messed with it).

The grep is there to remove directory names from the output of rsync.

The output of grep is then piped to another rsync using xargs. In the param folder1/./{} the extra ./ is used to tell rsync to create the relative directories starting with ./{}, ignoring folder1/. In other words, it avoids creating folder3/folder1/.

I was only able to test this on CentOS 6.4, but AFAIK it should work the same in Ubuntu 12.04.

EDIT: I should note that smithian's answer is undoubtedly better, but I wanted to take a stab at doing it with your original rsync command in tact since it was giving you the desired output.

compare 2 directories and copy differences in a 3rd directory

3 Answers3

Linked