What is a tool for getting a complete directory/file listing with detailed information including hash(es)?

Once in a while, I take a complete snapshot of my drives with a command like the following.

> for %i in (%drives_hd%) do @dir %i:\/s/a/o>>File_List.txt

This gives detailed information (path, filename, date, and size) for all files on my system and is great for keeping tabs on the files without expending too much space (~50MB for ~500,000 files). The problem is that it does not contain hashes.

Obviously there are file-hashing tools, but they don’t give the other details, and there is no practical way to do the hash(es) separately and combine it/them with the directory listings.

I’m looking for a tool that can create a text file with complete details like with the dir command, but also include file hashes (at least CRC(32), MD5, and SHA1). It should also be well written though so that it only reads each file once no matter how many hashes you ask it to do (ie., don’t read the whole file once for each hash type).

I’d prefer a CLI program, but a GUI one is okay, so long as it can be run from a script.

I’m even open to (Windows ports of) Linux tools.

I could find nothing with Google (though I am surprised such a useful tool is not more common), and have considered writing such a tool myself, but I’m hoping something already exists.

To make clear what I am looking for, see the below sample outputs.

Default dir output:

 Volume in drive C is C-WINDOWSXP
 Volume Serial Number is 1234-5678

 Directory of C:\

2007.07.05  04:05p    <DIR>          Documents and Settings
2011.05.04  07:38p    <DIR>          Program Files
2010.04.02  11:35p    <DIR>          WINDOWS
2011.10.09  10:45p               454 BOOT.INI
2002.08.28  10:08p            47,580 NTDETECT.COM
2002.08.29  02:05a           233,632 NTLDR
               3 File(s)      281,666 bytes

 Directory of C:\Documents and Settings

2003.11.12  03:08p    <DIR>          .
2003.11.12  03:08p    <DIR>          ..
2007.07.05  10:36p    <DIR>          Administrator
2007.07.05  04:21p    <DIR>          All Users
               0 File(s)              0 bytes

 Directory of C:\Documents and Settings\All Users

2003.11.12  03:08p    <DIR>          .
2003.11.12  03:08p    <DIR>          ..
2007.07.05  04:23p    <DIR>          Application Data
2011.06.23  03:23p    <DIR>          Documents
2011.01.09  12:56p           262,144 ntuser.dat
               1 File(s)      262,144 bytes

...

Desired output:

 Volume in drive C is C-WINDOWSXP
 Volume Serial Number is 1234-5678

 Directory of C:\

2007.07.05  04:05p    <DIR>          Documents and Settings
2011.05.04  07:38p    <DIR>          Program Files
2010.04.02  11:35p    <DIR>          WINDOWS
2011.10.09  10:45p               454 BOOT.INI                 d1183b26 fad47d7d255e1189dbef3003fba96868 39c9bbe3edad58a5bd091ea1db8f9b6cf03f9566
2002.08.28  10:08p            47,580 NTDETECT.COM             a709deed 28a3ac957be5d239a3dd4f3d4cdbf3b8 f5625a158d92478c814df3b33a9ad5fcd5f8a956
2002.08.29  02:05a           233,632 NTLDR                    0d7e47bd 9896e483e211b8cd1fa7bb32572f02ec c57426135d0419985681a674149c88e652c8ec63
               3 File(s)      281,666 bytes
               3 Dir(s)

 Directory of C:\Documents and Settings

2003.11.12  03:08p    <DIR>          .
2003.11.12  03:08p    <DIR>          ..
2007.07.05  10:36p    <DIR>          Administrator
2007.07.05  04:21p    <DIR>          All Users
               0 File(s)              0 bytes
               2 Dir(s)

 Directory of C:\Documents and Settings\All Users

2003.11.12  03:08p    <DIR>          .
2003.11.12  03:08p    <DIR>          ..
2007.07.05  04:23p    <DIR>          Application Data
2011.06.23  03:23p    <DIR>          Documents
2011.01.09  12:56p           262,144 ntuser.dat               fc3d370a b3ea06755f614e2c18fc1de875b60126 8264549330d9dbef494264227be9fadffe653556
               1 File(s)      262,144 bytes
               2 Dir(s)

Synetech

Posted 2011-12-26T00:19:48.393

Reputation: 63 242

Answers

WinHasher:

WinHasher is a free, Open Source cryptographic hash or digest generator written in C# using Microsoft's .NET 2.0 Framework. It can be used to verify file download integrity, compare two or more files for modifications, and to some degree generate strong, unique passwords.

CommandLine Hash Generator:

cmdhashgen is a Command Line Utility that can be used to generate various hashes for a given String or File.

Supported Hashes are CRC32, MD5, SHA-1, SHA-256, SHA-384 and SHA-512.

WinHasher has command-line utilities including "Hash", which can be tied into a batch file or script. It looks like the more stable of the two packages.

Usage: hash [-md5|-sha1|-sha256|-sha384|-sha512|-ripemd160|-whirlpool|
       -tiger] [-base64|-hexcaps|-bubbab] filename1 [filename2 ...]

WinHasher is a command-line cryptographic hash generator for files.  It
runs in one of two modes:  single file hashing and multi-file comparison.

In single file mode, WinHasher computes the cryptographic hash of the
given file and prints it to the screen.  With no command-line switches,
it computes the SHA-1 hash and displays it in hexadecimal format.  Various
switches allow you to change to other hashing algorithms, such as MD5,
the SHA family, RIPEMD-160, Whirlpool, and Tiger.  The "-base64" switch
causes WinHasher to output hashes in MIME Base64 (RFC 2045) format rather
than hexadecimal, "-hexcaps" outputs hexadecimal with all capital letters,
and "-bubbab" uses Bubble Babble encoding.

Ƭᴇcʜιᴇ007

Posted 2011-12-26T00:19:48.393

Reputation: 103 763

Like I said, I already know there are countless hashing tools (I’ve got dozens and even wrote a PHP script years ago). Also like I said, that is not what I am looking for because these programs you linked to do not list every file and directory and all their details like dir, nor can their outputs be easily integrated into a directory listing. Also, they do not perform multiple hashes with a single read. (I’d down-vote for not bothering to read the question, but it would only decrease my rep and have zero effect on yours.) – Synetech – 2011-12-26T05:47:38.783

I did read your question... Windows does not include any built-in utilities for cryptographic hashes, so in practice, you could expand your script to use a utility like this (unless someone else knows of one that does EXACTLY what you want), and format the output as you'd like. Perhaps consider using Powershell and the System.Security.Cryptography.HashAlgorithm namespace from .Net? – Ƭᴇcʜιᴇ007 – 2011-12-26T16:26:01.463

I know there is no built-in command; that’s why I am asking if anyone knows of a tool. To repeat, yes, I can script something to incorporate the hashes from a separate hashing tool into the dir output, but like I said, that is not feasible. It would require significant processing of two or more files (which is a nightmare for 50MB files with 900K lines). Worse, doing it your way would require hashing each file once for each hash type (you hash 500GB of files, three times each!) As for PowerShell, if I were going to go to all that trouble, I would just write my own EXE like I said. – Synetech – 2011-12-27T20:52:45.660

Sounds good, just keep in mind that each of the hashes will need to be calculated at some point, regardless if it's a script or an EXE. – Ƭᴇcʜιᴇ007 – 2011-12-28T04:25:28.143

1> keep in mind that each of the hashes will need to be calculated at some point Except that they shouldn’t. A hash takes an input stream and performs calculations on it. A poorly designed program would read the file (or other source), hash it, read the file again, hash it, and so on. A well-written program would read a block of data from the file and run it through each designated hashing function, read another block, hash, and so on until it reaches the end-of-file. – Synetech – 2011-12-29T20:34:24.810

One possibility is NirSoft HashMyFiles.

This includes a GUI which can search a folder (and subfolders) for files and calculate multiple hashes of all files it finds. The displayed output can be copied to the clipboard either as a simple list of items (one file per line, spaces separating fields) or as a HTML report.

The program can also be called from the command line, with various options for what to hash and how to format the results. Several text formats including CSV, two html formats and an xml formats are available.

Sadly, I can't see command-line options for sorting the results - the order appears to be whatever order the files were found in during the traversal, which can sometimes accidentally look like it's sorted by the full pathname, but you can't assume that. However, you can sort by clicking a column heading in the GUI as normal, and that seems to affect what you get if you then copy to the clipboard.

For command line use, I guess you can always use a separate utility or script to do any sorting you need. With the XML output option, it might even be an excuse to learn some basic XSLT.

In any case, this is closer to dir /s/b than to dir /s, but it might solve your problem (if you still have it).

Steve314

Posted 2011-12-26T00:19:48.393

Reputation: 1 569

That’s not really what I’m looking for, but it does include other attributes of the files (path, dates, size, etc.) so it is certainly the closest. And, based on Nir’s body of work, I would not surprised if he designed it to be efficient when calculating multiple hashes for the same file. – Synetech – 2012-06-18T02:18:17.997

http://sourceforge.net/projects/quickhash

Has all of the above features (see 'Recursive Hash' tab), plus cross-platform and open-source and currently maintained, v2.1 released a few weeks ago. Uses MD5, SHA1, SHA256 or SHA512.

Gizmo_the_Great

Posted 2011-12-26T00:19:48.393

Reputation: 493

Thanks, but that’s not quite what I’m looking for. For one thing, it doesn’t include timestamps or directory summaries like dir does. Also, its output is CSV, not human-readable(r) plain-text. And it is GUI only, so you cannot automate its usage in a script. (Plus the GUI is a bit defective at this time.) However it is the closest I have seen so far, and it is open-source, so I suppose someone (hopefully not me; I don’t have the time) could expand/improve it. +1 for now and a possible accept if it improves. – Synetech – 2013-07-19T19:34:07.733