Remove Duplicate Messages from Maildir

Question

I've got a bunch of duplicate messages in my IMAP server's Maildir. What's the best way to remove them?

Some relevant points:

Shared Message-ID is usually a good enough definition of duplicate. A tiny script that removes all but one of the duplicate messages would work.
Sometimes it's necessary to find duplicates based on shared message bodies. What's a reasonable definition of shared here? Bitwise equivalent? What about weird differences in line wrapping, escaping, character encoding?
Sometimes there's some meaningful difference between 'duplicate' messages. What's the best way to review the differences in sets of 'duplicate' messages? Diffs?

Adam Spiers · Answer 1 · 2014-02-03T12:29:57.130

10

I've made some significant improvements to Kevin's script mentioned above, and he was kind enough to accept my pull requests. Eventually we split this off into a dedicated project which you can find here:

https://github.com/kdeldycke/maildir-deduplicate

edited Feb 03 '14 at 12:29

answered Apr 28 '11 at 02:55

Adam Spiers

570
1
4
13

This script [moved to a new location](http://kevin.deldycke.com/2013/06/maildir-deduplicate-moved/) so I updated the link. – Adam Spiers Feb 03 '14 at 12:32
AFAIU you always can install it by `pip install mail-deduplicate`. – 0andriy Jan 20 '21 at 10:30

score 3 · Answer 2 · answered Nov 17 '12 at 15:58

3

for generic files in linux, I use fdupes utils to remove duplicate files. I found it also works for Maildir messages.

answered Nov 17 '12 at 15:58

sarabande

51
4

1

[fdupes](https://github.com/adrianlopezroche/fdupes) seems to work for exact duplicates only, while the OP is (implicitly) asking about more complex patterns of duplication. A message delivered twice because of `.forward` or whatever will have slightly different headers, so while the message itself is a duplicate, the two files containing the two copies may not be. – tripleee Feb 06 '17 at 06:55

score 1 · Answer 3 · edited Jun 13 '19 at 11:47

1

Best I've found today is Kevin Deldycke's maildir-deduplicate.

It ignores the X-MIMETrack header by default and compares headers using the SHA224 digest.
It automatically deletes duplicates without asking for confirmation; however there is a dry-run mode which allows previewing which duplicates will be deleted.

I bet someone could make something fancy from Rick Sanders' delIMAPdups.pl, part of his IMAP Tools.

edited Jun 13 '19 at 11:47

Adam Spiers

570
1
4
13

answered Apr 04 '11 at 22:03

Joseph Holsten

263
2
9

`maildir-deduplicate` [moved to a new location](http://kevin.deldycke.com/2013/06/maildir-deduplicate-moved/) so I updated the link. However your information is now out of date. – Adam Spiers Feb 03 '14 at 12:32
I've updated this so that there is no longer misleading outdated info. – Adam Spiers Jun 13 '19 at 11:47

score 0 · Answer 4 · answered Apr 12 '21 at 09:12

0

If you use Dovecot for IMAP access, you can use the following command:

doveadm deduplicate -u user@yourdomain.com mailbox INBOX

It should take care of everything, all duplicate emails should be deleted right away.

answered Apr 12 '21 at 09:12

Gabriel Hautclocq

101
2

score 0 · Answer 5 · answered Nov 11 '15 at 10:06

Gnome's Evolution [a graphical mail user agent] has a built-in feature to remove duplicate mail. As explained on this help page, it boils down to:

Select the suspect messages (or just all messages)
Go to menu Messages, the choose Remove Duplicate Messages.

Voilà.

P.S. Evolution can access your messages locally (MailDir, MH, Mbox) or over IMAP.

Remove Duplicate Messages from Maildir

5 Answers5