10

What is the best way to parse the headers of bounced (undeliverable) email that is sent back to my server and determine whether it is a soft or hard bounce?

I only send opt-in emails to my users, but occasionally some email addresses go stale. When an email bounces back to my server, I would like to find why it bounced (soft/hard). Then I can deal with it appropriately in my database and/or flag the user to update their email when the log in next.

I'm using Ubuntu and Postfix. I have successfully implemented VERP with aliases and virtual aliases. So bounced emails have a return-path of bounce+OrigEmailAddress@example.com, and I can pipe them to a script.

Now that I have VERP setup, I know who the original email was sent to, but I need to parse the returned mail headers to figure out if it is a soft bounce or hard bounce.

What is the best way to handle this? As I understand it, not all mail servers play by the same rules, and headers can have a variety of formats. Is there some open source project that keeps track of these types of things? Something simple I can implement that will categorize the majority of bounces properly?

I'm trying to protect the reputation of my mail server, so any help is much appreciated!

masegaloeh
  • 17,978
  • 9
  • 56
  • 104
Richard
  • 333
  • 1
  • 3
  • 12

2 Answers2

12

As RFC3463 explains, status codes beginning with 5 are used for permanent failures and 4 for persistent transient failures. Instead of trying to parse several messages with different formats you could rely on server logs and try something like this:

grep " dsn=5." /var/log/mail.log | grep -o -P " to=<(.+?)>" | sort | uniq -c

This will find permanent errors from mail.log (Postfix format) and give the addresses and amount of bounces on every address. You can also use " dsn=4." to get addresses with temporary errors.

Esa Jokinen
  • 43,252
  • 2
  • 75
  • 122
  • Thanks Esa! I didn't realize that postfix had that info in the mail log. Is this the solution that you use? Do you find that postfix categorizes the hard bounces dsn=5 correctly? I have read that some mail servers don't comply with RFC. So I thought a more complicated solution might be required. What has your experience been? This seems like a good solution if we can test postfix to get it right :-) – Richard Mar 06 '15 at 16:27
  • 1
    Really useful script - thank you! Recipe here for an alternative to grep's -P flag (for Mac users etc): https://unix.stackexchange.com/a/437694/275762 `grep " dsn=5." /var/log/mail.log | pcregrep -o1 " to=<(.+?)>" | sort | uniq -c` – Peter M. Apr 16 '20 at 21:30
9

Generally there are two types of bounces

  1. The bounces caused by directrejection of remote mail server when your postfix deliver the email.
  2. The bounces caused by remote server (next-hop server after your postfix) fails to deliver the message to final recipients.

The first case was already covered by excellent answer by Esa Jokinen above. Your best bet is parsing maillog.

The second case was special case of bounces. The example scenario:

  • You send email with recipient fakemail@example.com to mail.example.com server.
  • In mail.example.com, fakemail@example.com was aliased to realmail@example.net and must be forwarded to mail.example.net.
  • Someday mail.example.net reject the your message so mail.example.com must send bounces to your server.
  • Unfortunately maillog in your server will have "dsn=2" because mail.example.com already accepted the message but failed to forward it to mail.example.net.

Here the example of second type bounces email. There is forwarding rule Yahoo mail server myuser@yahoo.com -> myuser@example.net. Unfortunately mail server of example.net reject the message :(

From MAILER-DAEMON  Thu Mar  5 05:07:26 2015
Return-Path: <>
X-Original-To: noreply-myuser=yahoo.com@example.org
Delivered-To: noreply-263462085117-1425506829-myuser=yahoo.com@example.org
Received: from nm21-vm7.bullet.mail.gq1.yahoo.com (nm21-vm7.bullet.mail.gq1.yahoo.com [98.136.217.54])
        (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
        (No client certificate requested)
        by mx.example.org (Postfix) with ESMTPS id D6365565FC
        for <noreply-263462085117-1425506829-myuser=yahoo.com@example.org>; Thu,  5 Mar 2015 05:07:25 +0700 (WIT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=bounce; t=1425506842; bh=zk/tWZNl6c36dmlPDmakM9ekK8cHVJANXMmSdsbkcWc=; h=From:To:Date:Subject:From:Subject; b=Im95h1qTg6qN3yUI7vF1fXtJ0SbUnzv8rUPwLbpNwxGPN2p8wfosXJzQgJ3nzr4L4ZQ50P2d9E9U4jEUNtnyi7nlFd5kKbtiVuda4H56h1PFnt+7wSpgHcd5Irs/lLODumb6ZZSEpCOWttcB9+JLaDfEUUPjGcbR+xww4XeH5Eo=
From: MAILER-DAEMON@yahoo.com
To: noreply-263462085117-1425506829-myuser=yahoo.com@example.org
Date: Wed, 04 Mar 2015 22:07:22 -0000
Subject: Failure Notice
X-Yahoo-Newman-Property: bmbounce

Sorry, we were unable to deliver your message to the following address.

<myuser@example.net>:
Remote host said:
550 5.1.1 User unknown
 [RCPT_TO]

For this case, your only method is parsing the bounces message. Unfortunately there are no standard bounces format, so you must parsing the body and determine the rejection caused.

The feature checklist of your postfix bounce parsing:

  1. Check if VERP address was valid. You don't want to parse invalid message.
  2. Parse the body, determine if they are soft or hard rejection.

For the second feature, you can google some common rejection message. The example is this bounce-regex-list.xml by Jakub Liska.

Esa Jokinen made a good point in the comment below about these two bounce types. If your goal is keep the server reputation, then dealing the first bounce type should be enough. The second bounce was about cleaning your lists. So dead email should be erased thus freeing some resources in your server.

Some mailing list managers such as PHPlist and Mailman also deal with this bounce problem with parsing the email body as they have no resources to parsing the maillog.

Community
  • 1
masegaloeh
  • 17,978
  • 9
  • 56
  • 104
  • 2
    This solution is useful and more thorough if it is necessary to handle also mails automatically forwarded to another address. However, if the goal is to protect the reputation of the mail server, handling direct rejections should be sufficient. Administrators of the forwarding MTA should remove outdated forwards and mail lists (to protect their reputation and to avoid unnecessary traffic). After that we are back in the case one. OP should use this solution if the amount of secondary bounces is significant. Which ever takes less effort. – Esa Jokinen Mar 05 '15 at 12:16
  • @masegaloeh, thanks for the info! I hadn't even considered that forwarding situation as a possibility! For now I am mainly concerned with protecting the rep of my mail server, but if bounces increase this may be very useful. – Richard Mar 06 '15 at 16:34