42

I operate a small mail server for my private emails, some friends who have websites and two NGOs. In total my server sends between 60 and 400 messages a day. Now a lot of these emails are personal mails, between two or more people who know each other. Occasionally (usually once or twice a week) there will be a mailing that goes out to "members" of one NGO, informing them what's new etc.

Now I have already moved off the "mass mailings" (about 100 recipients, all personally known and manually subscribed through a paper form) to mailgun.org.

I still get (and increasingly so), rejected messages. Especially big email providers like Gmail, Yahoo or Microsoft (hotmail, live.com, ...) just decide to reject with a 550 or send personal messages to the Spam folder of the recipients. Sometimes this happens:

  • gmail user sends email to user on my system
  • user on my system replies
  • the reply is being rejected or sent to spam

Things I have done:

  • set up DKIM (per-domain signing of all outgoing email)
  • set up SPF, domains usually have ~all, some -all
  • I have a correct PTR for my mail server IP
  • obviously no open relay, users can only send from their own email address after authentication
  • I have DMARC policies for most of the domains
  • I rate limit outgoing messages, for some mail servers down to 1 per minute
  • mail test services report "perfect" scores (all pass) for all of the above
  • I regularily check my IP for blacklisting using http://www.dnsbl.info - it's always all green

Now the paradox comes here: for most of the big mail providers, there is a way to register to monitor rejection rates and IP reputation:

but I do not classify as bulk sender, because of the low volume. So I did register to monitor my reputation and rejection rates, but because I do not send bulk email, there are no reports.

Is there anything else I can do to improve mail delivery rates? Or should I give in and stop trying to operate my own mail server?

In case it is relevant: I use postfix and have very strict rules about incoming mail (i.e. no unknown domains/host names or invalid SPF records, I use spamassassin etc.)

Update

Here is an example, sent from me to my in-laws and it arrived in their SPAM folder: http://pastebin.com/BC6YgjpQ (I replaced the sending address domain with example.com and the receivers address with example@gmail.com)

Since the question came up: Connections to Gmail are Untrusted TLS connection established to gmail-smtp-in.l.google.com[2a00:1450:400c:c0b::1b]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits) encrypted.

Stefan Seidel
  • 732
  • 1
  • 7
  • 20
  • 3
    I gave up quite a while back. Now I send even my personal email through SendGrid. – Michael Hampton Apr 21 '16 at 19:25
  • 3
    I had similar problems, but I have even fewer mail traffic. In my case I added an SPF DNS record, and google allowed my mails again. For me this topic is very important. Running an own mail server is for me a fundamental human right. Maybe you can talk to the EFF they handle big topics like this. – guettli Apr 21 '16 at 19:45
  • 1
    Possible duplicate of [How to send emails and avoid them being classified as spam?](http://serverfault.com/questions/48428/how-to-send-emails-and-avoid-them-being-classified-as-spam) – MadHatter Apr 22 '16 at 07:09
  • I noticed you added an authentication header to the outgoing message. Google may treat this as an attempt to spoof authentication. My reading of the authentication header is that it is intended to be added by MX (border servers) to incoming messages. It is permitted/recommended that the MX remove existing Authentication headers. – BillThor Sep 30 '16 at 17:01

7 Answers7

21

There should be no issues becoming a small mail provider. You seem to be doing the right things. Many large providers don't get things right, and hopefully get most of their mail delivered.

If mail is being sent to the SPAM folder, it is likely you have missed something. There should be a record of why you have delivery issues:

  • For bounced messages read the response. It should specify why the mail was bounced. If you can, make sure bounce messages are logged.
  • For messages that are sent to the Spam folder, examine the message headers on the delivered message. This should (will for GMail or Yahoo) contain details of at some of the checks that were done. This help you determine what the issue is.

A few things you did not specify although some should be caught by the validation report:

  • rDNS validation of your mail servers address succeeds. (Your PTR record should return only one address.)
  • Your server used the name on the PTR record in its EHLO or HELO message.
  • Setup an SPF record for your mail server's domain ("v=spf1 a -all").
  • You have registered with dnswl.org.
  • You have had the DKIM public key(s) published in the correct location. You can use the same key for multiple domains. It may help to have other organizations use CNAME records to DNS records you control.
  • You have used a large DKIM key 1024 or larger.
  • Process outgoing mail through a spam filter (at least log issues).

If you have DMARC you can configure delivery status reports and bounce reports. This will allow you to receive delivery reports. I receive reports from Google, Microsoft and Yahoo. Please note disposition "none" indicates the mail was delivered.

BillThor
  • 27,354
  • 3
  • 35
  • 69
  • 2
    Here's a 550: `host aspmx.l.google.com[2a00:1450:4013:c01::1a] said: 550-5.7.1 [2a02:c200:0:10:3:0:4018:cafe 18] Our system has detected that this message is likely suspicious due to the very low reputation of the sending IP address. To best protect our users from spam, the message has been blocked. Please visit https://support.google.com/mail/answer/188131 for more information. n123si21866589wmb.41 - gsmtp (in reply to end of DATA command)` That doesn't tell me anything. – Stefan Seidel Apr 22 '16 at 07:08
  • Who is the owner of the ip? Is it a residential or a business provider? Would it be an option to relay your outgoing smtp traffic through your internet provider's smtp servers? So you still would have control of the incoming traffic (I know, not exactly what you want, but still, having to explain every time to your in laws why your messages get flagged as spam or rejected could get awkward soon-ish ;-) ) – natxo asenjo Apr 22 '16 at 07:42
  • @StefanSeidel They have provided you with a link to a site where you can get trusted. You will need to as a CNAME or TXT record to each signing domain. It is a fairly quick and painless process. – BillThor Apr 22 '16 at 11:46
  • 4
    @BillThor are you talking about the registration for postmaster.google.com? That has nothing to do with "getting trusted". It allows you to view the spam reporting rates for verified domains. I have done that and it leads exactly to the situation I describe above: since the sending volume is so low, there is no data. I have verified my 4 top domains and none show **any** data. – Stefan Seidel Apr 23 '16 at 17:07
  • @StefanSeidel have you also registered the domain name of your mailhost? – BillThor Apr 24 '16 at 02:33
10

One thing missing in the above (excellent) replies is to set up outbound TLS. Gmail has started to punish senders not using TLS, and other providers aren't saying anything but I'm sure they will follow suit.

Matt Sergeant
  • 504
  • 2
  • 5
  • Thanks. Postfix prefers TLS by default, and I can see that is being used. – Stefan Seidel Apr 22 '16 at 06:57
  • 1
    Are you using a self-signed certificate? I don't know this for sure, but I rather suspect that using a third-party-verified certificate improves matters further, and they're getting awfully cheap. – MadHatter Apr 22 '16 at 07:08
  • Nope, StartCom Level 2 validated. Also, this isn't relevant for outgoing mail, right? – Stefan Seidel Apr 22 '16 at 07:14
  • @StefanSeidel why would it not apply for outgoing email? TLS certificate validation applies to all TLS transactions, and outgoing email from a modern MTA, to a server which advertises `STARTTLS`, will be sent under TLS. – MadHatter May 10 '16 at 05:19
  • @MadHatter correct me if I'm wrong, but I do not think that my server will send _my_ certificate when establishing an outgoing TLS/SSL connection. It will receive and potentially validate the remote servers certificate. Only for incoming connections will it send my certificate (chain). – Stefan Seidel May 10 '16 at 06:47
  • 1
    @StefanSeidel You are wrong. Here is a very typical line from my mail server's log in TLS **server** mode (ie, receiving an incoming connection from another server) showing the validation of the remote server's certificate (`verify=OK`): `May 10 08:01:34 lory sendmail[5884]: STARTTLS=server, relay=mail-bn1bhn0245.outbound.protection.outlook.com [157.56.111.245], version=TLSv1/SSLv3, verify=OK, cipher=AES256-SHA256, bits=256/256` – MadHatter May 10 '16 at 07:08
  • As a note here to anyone who reads this, when the smtp_tls_security_level is ‘may’ (the default), Postfix will try TLS and, should it fail to deliver the message, will then try a non-TLS outgoing connection. So if you send a bunch of mail and GMail delays you, the next connection will be without TLS. Setting a smtp_tls_policy_map for gmail to always use encryption (and even better to use ciphers=high) will ensure that it doesn’t fall back in such a scenario. We found a 80% google SSL rate due to such delays and fixed it with this kind of map. – michaelkrieger Feb 03 '21 at 02:59
4

Nowadays, the spam activities are a real headache. The Big guys like Gmail, Microsoft, Yahoo etc. trying to secure their users from the spams. Hence, they must improve their techniques to filter the spams. And due to the security reason, they never disclose their Spam policies as well. Hence, we could not find a guideline to configure a mail server in such way that we can send mails to the big service providers.

There are no specific rules to be not listed in their bad book, but you should keep your server updated with the new guidelines. Here are some of them.

1) Check the root cause of the bounced back mail. Does it relate to the server IP reputation OR domain's incorrect DNS records.

2) Do not use an SPF record with a default value like ~all. Create a specific SPF record like a MX -all

3) Avoid mail forwarding from your server to Gmail/Yahoo/Microsoft/Comcast. If they detect any spam mail in your forwarded mails, they will not bother to check from where the mail is originated. They simply consider your mail server as a spam origin and you might be added to the blacklist.

4) Install an SSL on your server and use Outbound with TLS connection.

5) Keep Double Opt In list in all the newsletters. And many more...

mohemmadAli
  • 92
  • 4
2
Untrusted TLS connection established to gmail-smtp-in.l.google.com

I think you have a certificate chain error here. Make sure you are sending the intermediate certificate along with your certificate.

longneck
  • 22,793
  • 4
  • 50
  • 84
  • 1
    That just means that *my server* doesn't trust the gmail SMTP. I have added `smtp_tls_CApath` to my postfix conf, so that is a `Trusted TLS connection` now. But since that verification is only local to my mail server, I think it can not be relevant to any scoring. – Stefan Seidel Apr 27 '16 at 15:09
  • smtp_tls_CAPath is the CA which “Postfix SMTP client uses to verify a remote SMTP server certificate”. This is for the outgoing SMTP connection to verify whether Google’s server certificate is valid when using verify client mode. This would not impact google’s interpretation of the SMTP connection or mail. – michaelkrieger Feb 03 '21 at 02:55
2

Answer is no.

I state this mostly because I'm aware that there are hard to imagine number of people with more experience in all aspects of being ESP admin than me and yet I run dozen of production email systems that qualifies as "small" without presented problems.

From what you have posted it looks like you have taught of everything and, presumed you have properly implemented what is listed, there is only one option left - that your IP address has done bad things in it's past life.

I mean (really) bad. Your IP address may be pulled from public blacklists at the ISP's initiative (which is vastly more efficient) but before that served virus and phishing traffic repeatedly for a prolonged period of time.

If that is the case, unfortunately there is little that can be done, to my best knowledge. One option is to remain in attempt to maintain legitimate email server for a long time, paying the price of many emails being rejected before sender reputation - completely different thing to being present or not on public blacklists - is slowly established.

Please keep us informed about your case.

Miloš Đakonović
  • 640
  • 3
  • 9
  • 28
  • Thanks, and I think, yes the big deal after all the other stuff may be IP reputation. I recently added a second mail server, and I got some rejections to Microsoft addresses. Turns out my hosting provider put me in a "bad neighbourhood". I was able to open a ticket with Microsoft though and they took my IP out of the range of bad IPs - subject to continuing good behaviour (which I plan to show, obviously). – Stefan Seidel Nov 05 '17 at 20:13
1

@Stefan

You seem very knowledgeable which is awesome, I looked over your headers, without your domain name it makes it very hard to help troubleshoot. The one thing I did notice in your headers is that you're using "Simple/Simple" to sign your DKIM, you should really switch that over to "Relaxed/Relaxed". A lot of mail servers have trouble with simple.

You also left the mail server name in your pastebin, which does show up on a blacklist. I doubt this is causing your issue, but this tool does scan a lot more then the one you were using.

Send an email to mailtest@unlocktheinbox.com to see how many criticals you have. You might get a few clues.

Henry
  • 910
  • 1
  • 5
  • 17
  • That site is a bit difficult to read for my eyes. I only see green rectangles. I did send this email, and allegedly there are 5 warnings and 3 criticals, but all I can see in between the `Email Authentication Pro Feature - Learn More` links is "pass" and "success". – Stefan Seidel Apr 27 '16 at 15:11
  • Yeah, you must of rolled off that one blacklist. Most public blacklists remove you if they don't detect spam for a certain period of time, usually a day or two. – Henry Apr 27 '16 at 15:22
0

TL/DR : Many non-international companies (NOT hotmail or gmail) use UCEPROTECT as a blacklist qualifier without realising that UCEPROTECT is a conn; requesting money from servers to be removed from their listings awhile failing to inform Hosting providers which specific accounts or IPs are causing spam listings for hosting providers to resolve it internally.

From this, small server providers who happen to be in similar IP blocks as spammers are being caught by over-zealous UCEPROTECT criteria.

To add to all the competent replies here already, our servers have recently had issues with emails not being delivered to destinations, and the issue was to do with the hosting service provider and completely outside of our control.

BACKGROUND

We rent servers from a national (UK) provider -- Specifically Fasthosts. They are in an ASN involving amongst others IONOS, Fasthosts, Arsys, 1&1 Mail and Media, Interdart and others.

Our servers are fully and correctly set up with PTR, DKIM, DMARC, SPF, etc. etc.

At semi-regular intervals our servers come up on the UCEPROTECT-Level3 spam listing, and for a couple of weeks every few months are marked incorrectly as spam.

WHY?

UCEPROTECT is one of the more prominent spam listing services and they are used by large, non-international mailing providers as one of many Blacklisting check services. They have levels 1- 2- and 3- listings. They do not provide a correction service for false listings, but they do allow server companies to pay them to be removed from their listings.

In discussion with our server hosts, Fasthosts, it came up that because neighbouring servers in our ASN blocks (123.456.789.XXX) are marked out on UCEPROTECT-3, then our servers are also marked out, simply by being in a nearby IP range in the same ASN . The offending servers may not even belong to the same Hosting Provider (probably belonging in this case to IONOS).

Fasthosts state they can't resolve the cause of the UCEPROTECT block because UCEPROTECT do not provide them enough information to do so. The most recent set of blacklistings come after a recent media advertising campaign by IONOS selling their small business server services.

CONCLUSION

It seems from a lot of digging, in the UK that many mid-level email providers are still using UCEPROTECT as a valid blacklisting service without realising that this misses a lot of genuine emails.

UCEPROTECT have a vested interest to keep blocking neighbouring IP blocks to encourage small providers to pay them to be whitelisted. It's a scam. But without a stepchange in the way medium-sized providers operate (ie they should ignore UCEPROTECT-3) there's nothing a small server host can do to escape the blacklist mark.

Martin
  • 177
  • 1
  • 2
  • 13