Computing a checksum of a huge file (30 GB)

Question

I have a huge data set consisting of several files, with size of some of the files exceeding 30 GB. After compressing, transferring the data set over the network, and decompressing again I am suspecting something went wrong (that a file might be corrupt). One quick way to confirm this would be to compute checksums of both copies of a file---but there is a 2 GB limit on the size of the file for MD5 tool in Windows PowerShell.

Is there an alternative for computing a checksum of a file whose size exceeds 30 GB? (The tool should be implemented for both Linux and Windows.)

Answer 1

You can use openssl for that. It's available for Windows and Linux.

In a linux terminal use:

openssl md5 /path/to/your_file

In a windows command line:

openssl.exe md5 C:\path\to\your_file