How can I copy files to an external drive and verify their integrity in OS X?

I'm moving large amounts of data from one external drive to another larger one. The files are important and the smaller drives need to be cleared and reused (HD camera). Is there some utility for moving files and verifying their integrity?

I've been using this command

find . -type f -exec md5 '{}' \; > md5list.txt

in the terminal to create a list of MD5s for each file then using diff to compare the two. However, I am moving 320GB at a time, which takes a while by itself. Computing the checksums takes another hour or so. It would be much more efficient to do this on the fly, during the copy. I'm just hoping someone has already written the software...

jedavis

Posted 2011-01-08T18:20:49.657

Reputation: 105

Answers

rsync

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum...

Ignacio Vazquez-Abrams

Posted 2011-01-08T18:20:49.657

Reputation: 100 516

I'd like a GUI for this (so I can delegate). I see there are multiple ones. Which is the best? – jedavis – 2011-01-08T19:18:41.287

Not a clue. I don't actually run OS X (and I've never used a GUI for rsync). – Ignacio Vazquez-Abrams – 2011-01-09T00:29:24.857

This is not correct or rather taken out of context. Rsync only verifies that the transfer was done correctly. In the case of a local copy there is no transfer over the network, so the actual file-writes are not verified. – user329023 – 2014-06-03T15:40:35.800

We use ShotPut Pro for this exact task - it copies media from SxS (and other) HD camera media, puts it into more than one place at a time if you want, and verifies the data.

It's made the process of keeping on top of footage so much easier for us.

If it's for cameras, then this is the best bet I think. If it's for copying other data, I'm not sure (I found this question while trying to find a decent filemanager for OS X!)

ShotPut Pro™ for Macintosh is an automated copy utility application for HD video and photo files. ShotPut Pro is the industry de-facto standard offloading application for professionals. The simple user interface and robust copy speeds make it indispensable for todays tapeless HD workflows.

Security Verify file-by-file sizes, MD5 or CRC checksums, or byte-to-byte comparisons of your media copies to ensure they exactly match the originals.

enter image description here

Jack

Posted 2011-01-08T18:20:49.657

Reputation: 11

Rsync only verifies that the transfer was done correctly. In the case of a local copy there is no transfer over the network, so nothing is verified. The actual file-writes would not be considerer transfer. The other poster noted this when he compared the times vs using cp. You must re-read the file from the disk. The fastest way to do this is CRC (less CPU intensive).

crc32 * > ~/Desktop/crc1.txt
crc32 * > ~/Desktop/crc2.txt

then run crc32 on those files and you will see if the copy was exact.

See the quote from the other answer at http://manpages.ubuntu.com/manpages/lucid/man1/rsync.1.html.

user329023

Posted 2011-01-08T18:20:49.657

Reputation: 111