How can I intelligently sync directories on Linux?

I have two directories on my Linux system that I want to sync. Most of the contents of these directories are the same, however some files differ:

Directory A contains:

File 1
File 2
File 4
File 5

Directory B contains:

File 1
File 2
File 3
File 5

After synchronization, I want both directory A and directory B to contain:

File 1
File 2
File 3
File 4
File 5

This is an example, the real directories contain thousands of files and their size is several GBs.

How can I do this on Linux?

Update: What if File 1, 2, 4, 5 are not in directory A but in a subdirectory AA that is in directory A? The files in directory B remain the same. After syncing, I don't want to end up with multiple copies of File 1, 2, 5.

Alex

Posted 2017-07-16T10:41:59.907

Reputation: 1

rsync, making sure you don't use --delete? – dirkt – 2017-07-16T10:49:44.643

Answers

What you want is rsync:

$ cd -- "$(mktemp --directory)
$ mkdir A B
$ touch A/File\ {1,2,4,5}
$ touch B/File\ {1,2,3,5}
$ rsync -a A/ B
$ rsync -a B/ A
$ ls A B
A:
'File 1'  'File 2'  'File 3'  'File 4'  'File 5'

B:
'File 1'  'File 2'  'File 3'  'File 4'  'File 5'

rsync will transport only the files necessary to sync up the directories. Unlike other *nix tools the slash after the first directory name is significant - it indicates that the content of the source directory rather than the directory itself should be copied.

l0b0

Posted 2017-07-16T10:41:59.907

Reputation: 6 306

What if File 1, 2, 4, 5 are not in directory A but in a subdirectory AA that is in directory A? The files in directory B remain the same. After syncing, I don't want to end up with multiple copies of File 1, 2, 5. Is rsync smart enough to avoid this? An option that hashes the files synced to prevent duplicates should work. – Alex – 2017-07-16T12:22:43.620

1Do you mean if you have A/AA/1 and B/1, both with the same content, you don't want to copy 1? If so, that's a completely different problem from the original. – l0b0 – 2017-07-16T12:41:05.503

Yes. I only want to make sure File 1 is both in A and B somewhere (can be A/1 or A/subdir/1, I don't care). As long as it is in both A and B it's fine. If I have A/AA/1 and B/1 as you said, I guess rsync would also create A/1 and B/AA/1 which is unnecessary because I would have File 1 in A both as A/1 and A/AA/1 and in B as B/1 and B/AA/1, which is redundant. I will create a new question if this is complicated and completely different problem. – Alex – 2017-07-16T12:49:19.243

Assuming both directories A and B are in the same level

$ rsync -a A/ B/ & rsync -a B/ A/

zarqos.os

Posted 2017-07-16T10:41:59.907

Reputation: 25