0
I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total. I have tried to remove duplicates using Mozilla Thunderbird add-on Remove Duplicate Messages (alternative), but it removed just a small part of them (few hundred). Then, i've used Windows desktop apps, such as Wise duplicate finder, duplicate cleaner free, AllDup, Fast Duplicate finder and Anti-Twin, using Byte by byte (60% comparision) and none of those applications succeded in finding right duplicates (again, i have managed to removed just a part of them, few thousands this time).
I've attached example of two emails that i have, although they have slightly different source code (and different file names), they are basically the same - they have been sent from same email address, in the same time, and they have same file size:
First email - message-1-34437.eml
Received: from e11mailgw02.isp.com ([212.200.12.195]) by mtain3.isp.com (Sun Java(tm) System Messaging Server 6.3-4.01 (built Aug 3 2007; 32bit)) with ESMTP id <0KKM00B5BQ1TKV40@mtain3.isp.com> for user@com; Tue, 02 Jun 2009 22:53:58 +0200 (CEST)
Received: from unknown (HELO vps.mafiascene.com) ([69.73.156.173]) by e11mailgw02.isp.com with ESMTP; Tue, 02 Jun 2009 22:53:57 +0200
Received: (qmail 24030 invoked by uid 48); Tue, 02 Jun 2009 16:53:51 -0400
Date: Tue, 02 Jun 2009 16:53:51 -0400
From: "Mafia Scene" <no-reply@mafiascene.com>
Subject: Mafia Scene Registration Confirmation
To: <user@com>
X-Priority: 3
X-MSMail-Priority: Normal
Importance: Normal
Message-ID: <20090602205351.24028.qmail@vps.mafiascene.com>
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Au0JAFEuJUpFSZyt/2dsb2JhbACOFhEBsRIRCAMEj2iCMR4IBAwEgSAF
X-IronPort-AV: E=McAfee;i="5300,2777,5634"; a="7766158"
X-MimeOLE: Produced By Microsoft MimeOLE V14.0.8089.726
Old-X-EsetId: 4FAA1F2928B4776950AC1F7F23E634
X-EsetId: 745B6128E6F033696B5D617DE9A773
X-EsetScannerBuild: 6455
Thank you for registering with Mafia Scene!
The details you registered your account with at 4:53pm EDT Tuesday - 2nd June 2009 are as follows:
Username: username
Password: password
To active your account you MUST visit the following link WITHIN the next 24 HOURS.
http://mafiascene.com/modules.php?name=users&action=activate&id=c284c0e0a7a7aec0772709511b2b8f3e
Regards,
The Mafia Scene Staff
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4124 (20090602) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4801 (20100124) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
Second email - message-1-54557.eml
Received: from e11mailgw02.com ([212.200.12.195])
by mtain3.isp.com
(Sun Java(tm) System Messaging Server 6.3-4.01 (built Aug 3 2007; 32bit))
with ESMTP id <0KKM00B5BQ1TKV40@mtain3.isp.com> for
user@com; Tue, 02 Jun 2009 22:53:58 +0200 (CEST)
Received: from unknown (HELO vps.mafiascene.com) ([69.73.156.173])
by e11mailgw02.com with ESMTP; Tue, 02 Jun 2009 22:53:57 +0200
Received: (qmail 24030 invoked by uid 48); Tue, 02 Jun 2009 16:53:51 -0400
Date: Tue, 02 Jun 2009 16:53:51 -0400
From: Mafia Scene <no-reply@mafiascene.com>
Subject: Mafia Scene Registration Confirmation
To: user@com
Message-id: <20090602205351.24028.qmail@vps.mafiascene.com>
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result:
Au0JAFEuJUpFSZyt/2dsb2JhbACOFhEBsRIRCAMEj2iCMR4IBAwEgSAF
X-IronPort-AV: E=McAfee;i="5300,2777,5634"; a="7766158"
X-EsetId: 4FAA1F2928B4776950AC1F7F23E634
Thank you for registering with Mafia Scene!
The details you registered your account with at 4:53pm EDT Tuesday - 2nd June 2009 are as follows:
Username: username
Password: password
To active your account you MUST visit the following link WITHIN the next 24 HOURS.
http://mafiascene.com/modules.php?name=users&action=activate&id=c284c0e0a7a7aec0772709511b2b8f3e
Regards,
The Mafia Scene Staff
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4124 (20090602) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
Is there some way to detect such emails as duplicates?
You must formulate programmable criteria of similarity (cut off all headers then compare is one of possible ways) and apply it to each file, then you may use any available software which searches for duplicates or create your own code. – Akina – 2019-09-25T07:58:40.230
In cases like this where there's an automated message, even grep with a catchy string can bring out a lot of emails. That(or similar) can be used in the methods suggested in the answer. – ankii – 2019-09-25T09:07:33.170