linux merge folders: rsync?

Question

I have two copies of a folder

src/
dest/

I want to merge them, doing the following:

If a file is only in src, I want it to be moved to dest

If a file is only in dest, I want it ignored IE left alone.

If a file is in both and has identical contents (IE same size and date), delete from src

If a file is in both and does not have identical contents, leave behind in src so I can manually merge them.

Only a very small number of files (between 0% and 5% of total files) should be in this last category, but I don't know how to separate the the in both and the same from in both, but different.

I've tried to figure out how to do this with rsync but to no avail so far.

Answer 1

I've only performed limited functionality testing, so please be careful with this command (--dry-run):

rsync -avPr --ignore-existing --remove-source-files src/ dest

Please note the trailing / as this will recurse into src instead of copying src itself, this should maintain your existing paths.

By using the --ignore-existing flag in combination with the --remove-source-files flag you will delete only files from src that are sync'ed from src to dest, that is files that did not previously exist in dest only.

For deleting non-sync'ed files, that is those that already existed in dest/ as in src/, you can use:

for file in `find src/ -type f`; do diff $file `echo $file | sed 's/src/dest/'` && rm $file || echo $file; done

or

find src -type f -exec bash -c 'cmp -s "$0" "${0/#src/dest}" && rm "$0"' {} \;

if filenames could contain whitespace/new lines/… Regarding Gilles' comment concerning special characters, that is certainly something to be mindful of and there are many solutions, the simplest would be to pass an -i to rm which will prompt before all deletion. Provided that src/, or its parent path, is provided to find, however, the fully qualified path should result in all file names being handled properly by both the diff and rm commands without quoting.

Answer 2

unison is the tool you're looking for. Try unison-gtk if you prefer a gui. But I don't think it will delete similar files: unison try to have both directories identical. Nevertheless it will easyly 1) identify which files are to copy; 2) which ones needs manual merge.

Answer 3

The following script should do things reasonably. It moves files from the source to the destination, never overwriting a file and creating directories as necessary. Source files that have a corresponding different file in the destination are left alone, as are files that are not regular files or directories (e.g. symbolic links). The files left over in the source are those for which there is a conflict. Beware, I haven't tested it at all.

cd src
find . -exec sh -c '
    set -- "/path/to/dest/$0"
    if [ -d "$0" ]; then #  the source is a directory 
      if ! [ -e "$1" ]; then
        mv -- "$0" "$1"  # move whole directory in one go
      fi
    elif ! [ -e "$0" ]; then  # the source doesn't exist after all
      :  # might happen if a whole directory was moved
    elif ! [ -e "$1" ]; then  # the destination doesn't exist
      mv -- "$0" "$1"
    elif [ -f "$1" ] && cmp -s -- "$0" "$1"; then  # identical files
      rm -- "$0"
    fi
  ' {} \;

Another approach would be to do a union mount one directory above the other, for example with funionfs or unionfs-fuse.