2

I am using md5deep to try take a list of md5 hashes from a list of files (approximately 100,000 hashes) and I'm comparing them to the NSRL hash list (15gb text file).

I'm trying to accomplish 1 thing:

Output hashes from my 100,000 that do not match hasehs in the NSRL hash list.

I am having trouble figuring out how to pass the 100,000 hashes in correctly. Here is what I have:

md5deep -e -x NSRLFile.txt -c md5.csv

md5.csv is just the hashes, separated by breaklines. Its not actually a csv.

I could theoretically cat md5.csv and pipe each line into 

md5deep -e -x NSRLFile.txt -a $line

but I'd like to avoid that, if I can use some functionality of the program.

I get that this is more of an "RTFM" question, but I've gone through the manual, and I'm still not seeing how to do it.

Sugitime
  • 415
  • 1
  • 4
  • 12
  • What's the format of those files? If each file is a list of hashes, then you just need to concatenate the files, sort the result and look for duplicates. Or sort each file independently and process the pair of files using the `comm` command. – kasperd May 21 '14 at 20:32
  • the md5.csv file is just 1 hash per line. NSRL is "SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode" – Sugitime May 21 '14 at 21:03

1 Answers1

0

If I understand the format correctly, this command should give the desired result:

comm -23 <(cat md5.csv | tr A-F a-f | sort) <(cat NSRLFile.txt | cut -f2 -d, | tr -d '"' | tr A-F a-f | sort)

This will output hashes that are in the first file but not in the second file. The -2 for the comm command will skip those lines that are in the second file only, and -3 will skip those lines, that are in the first file only.

cat md5.csv | tr A-F a-f | sort will convert all the hashes to lower case and sort them.

cat NSRLFile.txt | cut -f2 -d, | tr -d '"' | tr A-F a-f | sort will take the second column of the file separated by ,, then remove any " characters, convert all the hashes to lower case and sort them.

kasperd
  • 29,894
  • 16
  • 72
  • 122