Rsync, how can I delete remote duplicates?

Question

I'm trying to delete remote duplicate files without transferring the non-existing files, and without updating the existing files.

If I specify both --existing and --ignore-existing (along with -av --remove-source-files) the operation is a no-op and nothing will be transfered, but nothing will be deleted either.

The best I have so far is to make a local copy of destination, use rsync without --ignore-existing, then rsync my local copy on top of the destination