Duplicate Images / Tags - sorting and removing

Question

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s).

I want to organise and sort them, and remove duplicates, so I have used a Canon Utility to sort them into folders based on the date/time taken (EXIF information), and a utility called Ant Renamer from PortableApps to rename some of them to the date/time in yyyy-mm-dd hh-mm-ss .jpg format.

Unfortunately I have tagged some of the images with the camera name, or location I took them, but did this after importing them to the computer, and copying to a hard drive. So I have one copy of the image with the tags, and another without.

I'm using Duplicate File Finder v1.1.0.3 by Rashid Hoda but unfortunately this says some are different because of the tags. Ant renamer won't rename images in the same folder if there's one with the same date/time - which is why I kept the original in the filename.

I suppose I COULD make the assumption that any taken at the same time are duplicates because they are only from two cameras, and I MOSTLY kept the date and time set on my camera.

But, I would like to be more certain.

Suggestions for scripts / utilities / or even ideas I can take on board are most appreciated.

Answer 1

There's a utility called identify that comes with the ImageMagick package. It provides a checksum (MD5, I seem to recall) of the image data (not including tags) of the image. This can be used to find duplicates. I go into this in some depth in my post about de-duplicating iPhoto libraries--but the concepts are applicable to other situations too.

I don't know of any tool that does this automatically, but the post I linked to above shows most of what you need. All that is required is to call identify -verbose and extract its checksum rather that use the call to the md5sum utility I use in my script in the post. If you need help with that it's probably a post for StackOverflow :-)