As possible solution you may use any tool to convert file into uncompressed stream (pcm
, wav
) without metadata info and then compare it. For conversion you may use any software you have like ffmpeg
, sox
or avidemux
.
For example how I do that with ffmpeg
Say I have for that example 2 files with different metadata:
$ diff Original.mp3 Possible-dup.mp3 ; echo $?
Binary files Original.mp3 and Possible-dup.mp3 differ
Brute force comparison complain they are differ.
Then we just convert and diff body:
$ diff <( ffmpeg -loglevel 8 -i Original.mp3 -map_metadata -1 -f wav - ) <( ffmpeg -loglevel 8 -i Possible-dup.mp3 -map_metadata -1 -f wav - ) ; echo $?
0
Off course ; echo $?
part is just for demonstration purpose to see return code.
Processing multiple files (traverse directories)
If you want try duplicates in collection it have worth to calculate checksums (any like crc
, md5
, sha2
, sha256
) of data and then just find there collisions.
Although it is out of scope of that question I would suggest some simple suggestions how to find duplicates of files in directory accounting only it contents without metadata consideration.
- First calculate hash of data in each file (and place into file for next processing):
for file in *.mp3; do printf "%s:%s\n" "$( ffmpeg -loglevel 8 -i "$file" -map_metadata -1 -f wav - | sha256sum | cut -d' ' -f1 )" "$file"; done > mp3data.hashes
File will be looks like:
$ cat mp3data.hashes
ad48913a11de29ad4639253f2f06d8480b73d48a5f1d0aaa24271c0ba3998d02:file1.mp3
54320b708cea0771a8cf71fac24196a070836376dd83eedd619f247c2ece7480:file2.mp3
1d8627a21bdbf74cc5c7bc9451f7db264c167f7df4cbad7d8db80bc2f347110f:Original.mp3
8918674499b90ace36bcfb94d0d8ca1bc9f8bb391b166f899779b373905ddbc1:Other-dup.mp3
8918674499b90ace36bcfb94d0d8ca1bc9f8bb391b166f899779b373905ddbc1:Other.mp3
1d8627a21bdbf74cc5c7bc9451f7db264c167f7df4cbad7d8db80bc2f347110f:Possible-dup.mp3
Any RDBMS will be very helpful there to aggregate count and select such data.
But continue pure command-line solution you may want do simple steps like further.
See duplicates hashes if any (extra step to show how it works, does not needed for find dupes):
$ count.by.regexp.awk '([0-9a-f]+):' mp3data.hashes
[1:54320b708cea0771a8cf71fac24196a070836376dd83eedd619f247c2ece7480]=1
[1:1d8627a21bdbf74cc5c7bc9451f7db264c167f7df4cbad7d8db80bc2f347110f]=2
[1:ad48913a11de29ad4639253f2f06d8480b73d48a5f1d0aaa24271c0ba3998d02]=1
- And all together to list files duplicated by content:
$ grep mp3data.hashes -f <( count.by.regexp.awk '([0-9a-f]+):' mp3data.hashes | grep -oP '(?<=\[1:).{64}(?!]=1$)' ) | sort
1d8627a21bdbf74cc5c7bc9451f7db264c167f7df4cbad7d8db80bc2f347110f:Original.mp3
1d8627a21bdbf74cc5c7bc9451f7db264c167f7df4cbad7d8db80bc2f347110f:Possible-dup.mp3
8918674499b90ace36bcfb94d0d8ca1bc9f8bb391b166f899779b373905ddbc1:Other-dup.mp3
8918674499b90ace36bcfb94d0d8ca1bc9f8bb391b166f899779b373905ddbc1:Other.mp3
count.by.regexp.awk is simple awk
script to count regexp patterns.
Not very fast, but work perfectly to have a unique checksum on mp3 files to check duplicates. Thank you. – fred727 – 2016-11-11T16:13:25.127
A faster alternative if you can use php is getid3 library : http://www.getid3.org/phpBB3/viewtopic.php?f=3&t=1936
– fred727 – 2016-11-11T23:15:12.4503@fred727 I checked the
avconv
man page and realized that thecrc
option decodes the audio and computes the CRC of the decoded audio. But you can avoid that by setting the audio codec tocopy
. Now, on my system, the command runs in 0.13 seconds instead of 1.13 seconds. I updated the answer, so now you can avoid using PHP. :) – blujay – 2016-11-14T06:59:04.693