Postfix 2.6.6 with TLS - unable to receive emails from GMail (and a couple of other MTAs) but others are OK, why?

Question

I've just had to look at a CentOS 6 server running Postfix 2.6.6 which was able to send emails to everyone, but could not receive them from GMail (and a few other MTAs) due to incoming TLS negotiation problems.

A connection from a .google.com SMTP server (i.e., GMail) resulted in this:

Aug 23 19:34:29 server1 postfix/smtpd[7659]: connect from mail-lf1-f44.google.com[209.85.167.44]
Aug 23 19:34:29 server1 postfix/smtpd[7659]: setting up TLS connection from mail-lf1-f44.google.com[209.85.167.44]
Aug 23 19:34:29 server1 postfix/smtpd[7659]: SSL_accept error from mail-lf1-f44.google.com[209.85.167.44]: -1
Aug 23 19:34:29 server1 postfix/smtpd[7659]: warning: TLS library problem: 7659:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher:s3_srvr.c:1387:
Aug 23 19:34:29 server1 postfix/smtpd[7659]: lost connection after STARTTLS from mail-lf1-f44.google.com[209.85.167.44]
Aug 23 19:34:29 server1 postfix/smtpd[7659]: disconnect from mail-lf1-f44.google.com[209.85.167.44]

Increasing the TLS logging verbosity:

Aug 23 21:56:15 server1 postfix/smtpd[18103]: initializing the server-side TLS engine
Aug 23 21:56:15 server1 postfix/smtpd[18103]: connect from mail-lf1-f47.google.com[209.85.167.47]
Aug 23 21:56:15 server1 postfix/smtpd[18103]: setting up TLS connection from mail-lf1-f47.google.com[209.85.167.47]
Aug 23 21:56:15 server1 postfix/smtpd[18103]: mail-lf1-f47.google.com[209.85.167.47]: TLS cipher list "ALL:+RC4:@STRENGTH:!aNULL:!LOW:!EXP:!MEDIUM:!ADH:!AECDH:!MD5:!DSS:!ECDSA:!CAMELLIA128:!3DES:!CAMELLIA256:!RSA+AES:!eNULL"
Aug 23 21:56:15 server1 postfix/smtpd[18103]: SSL_accept:before/accept initialization
Aug 23 21:56:15 server1 postfix/smtpd[18103]: SSL3 alert write:fatal:handshake failure
Aug 23 21:56:15 server1 postfix/smtpd[18103]: SSL_accept:error in SSLv3 read client hello C
Aug 23 21:56:15 server1 postfix/smtpd[18103]: SSL_accept error from mail-lf1-f47.google.com[209.85.167.47]: -1
Aug 23 21:56:15 server1 postfix/smtpd[18103]: warning: TLS library problem: 18103:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher:s3_srvr.c:1387:
Aug 23 21:56:15 server1 postfix/smtpd[18103]: lost connection after STARTTLS from mail-lf1-f47.google.com[209.85.167.47]
Aug 23 21:56:15 server1 postfix/smtpd[18103]: disconnect from mail-lf1-f47.google.com[209.85.167.47]

No shared cipher?!

TLS was already enabled with an (albeit self-signed) server certificate; clients were connecting successfully to send and receive mail through IMAP/POP with SASL.

I read up on common causes of the 1408A0C1 postfix error but none seemed to quite apply to this scenario. Some of the checks I ran produced results which seemed to contradict what I expected;

postconf -d | grep cipherlist had this fairly short list of exclusions:

tls_export_cipherlist = ALL:+RC4:@STRENGTH
tls_high_cipherlist = ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH
tls_low_cipherlist = ALL:!EXPORT:+RC4:@STRENGTH
tls_medium_cipherlist = ALL:!EXPORT:!LOW:+RC4:@STRENGTH
tls_null_cipherlist = eNULL:!aNULL

and TLS protocols were fairly lenient:

smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtp_tls_mandatory_protocols  = !SSLv2, !SSLv3
smtpd_tls_protocols           = !SSLv2, !SSLv3
smtp_tls_protocols            = !SSLv2, !SSLv3

So why wasn't it able to negotiate a cipher?

I first set about updating OpenSSL (which was OpenSSL 1.0.1e-fips 11 Feb 2013, the latest available via yum); I did as instructed in this article and resultantly the box runs OpenSSL 1.0.2p 14 Aug 2018.

But still, the GMail reception problem continued...

I left those as-is to give all possible TLS variants a chance to succeed, and studied the ciphers in more detail.

Disabling all of the cipher exclusions, I sent a test email from my gmail and unsurprisingly it came through:

Aug 23 23:39:52 server1 postfix/smtpd[6036]: connect from mail-lj1-f171.google.com[209.85.208.171]
Aug 23 23:39:52 server1 postfix/smtpd[6036]: setting up TLS connection from mail-lj1-f171.google.com[209.85.208.171]
Aug 23 23:39:52 server1 postfix/smtpd[6036]: mail-lj1-f171.google.com[209.85.208.171]: TLS cipher list "ALL:+RC4:@STRENGTH"
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:before/accept initialization
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 read client hello B
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 write server hello A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 write certificate A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 write server done A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 flush data
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 read client key exchange A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 read finished A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 write session ticket A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 write change cipher spec A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 write finished A
Aug 23 23:39:52 server1 postfix/smtpd[6036]: SSL_accept:SSLv3 flush data
Aug 23 23:39:52 server1 postfix/smtpd[6036]: Anonymous TLS connection established from mail-lj1-f171.google.com[209.85.208.171]: TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)
Aug 23 23:39:52 server1 postfix/smtpd[6036]: 66BB15DC6: client=mail-lj1-f171.google.com[209.85.208.171]
Aug 23 23:39:52 server1 postfix/cleanup[6424]: 66BB15DC6: message-id=<CAK9Gk9r+6gt7g_U987A0XaGdKJGY=80n0rK595mqrmfGaL5LKQ@mail.gmail.com>
Aug 23 23:39:52 server1 opendkim[6890]: 66BB15DC6: mail-lj1-f171.google.com [209.85.208.171] not internal
Aug 23 23:39:52 server1 opendkim[6890]: 66BB15DC6: not authenticated
Aug 23 23:39:52 server1 opendkim[6890]: 66BB15DC6: DKIM verification successful
Aug 23 23:39:52 server1 postfix/qmgr[6032]: 66BB15DC6: from=<gmailaddress>, size=3988, nrcpt=1 (queue active)
Aug 23 23:39:52 server1 postfix/smtpd[6036]: disconnect from mail-lj1-f171.google.com[209.85.208.171]
Aug 23 23:39:52 server1 postfix/pipe[6425]: 66BB15DC6: to=<myinbox>, relay=dovecot, delay=0.48, delays=0.29/0.01/0/0.18, dsn=2.0.0, status=sent (delivered via dovecot service)

So if GMail negotiated AES128-GCM-SHA256 via TLSv1.2 -- why wasn't it working before? Particularly as their cipherlist appears to be any possible matches, with RC4 at the end, ordered in order of strength?

To test, I took a specific cipherlist, prepended the AES128 hash depths (italicised) - ECDHE and DHE methods - and explicitly declared it in postfix, which worked:

tls_high_cipherlist=ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA

I didn't want to have to constantly update cipherlist recipes though, so commented out the tls_high_cipherlist declaration.

I then did a few things.

In postfix's master.cf I added -o smtp_tls_mandatory_protocols=TLSv1 to the smtpd section, as recommended.

I wanted to improve the TLS protocols by using declarations from https://blog.kruyt.org/postfix-and-tls-encryption/, which explicitly approve TLSv1.2 and TLSv1.1 before excluding the older ones. However with postfix 2.6.6, this doesn't work; I saw this in the log not long after a reload/restart:

warning: Invalid TLS protocol list "TLSv1.2, TLSv1.1, !TLSv1, !SSLv2, !SSLv3": disabling TLS support

This happened if I mixed exclusion !rules with inclusion rules, so I reverted to the explicit exclusion format:

smtpd_tls_protocols = !TLSv1, !SSLv2, !SSLv3
smtp_tls_protocols = !TLSv1, !SSLv2, !SSLv3
smtp_tls_ciphers = high
smtpd_tls_ciphers = high
smtpd_tls_mandatory_protocols = !TLSv1, !SSLv2, !SSLv3
smtp_tls_mandatory_protocols = !TLSv1, !SSLv2, !SSLv3
smtp_tls_mandatory_ciphers = high
smtpd_tls_mandatory_ciphers = high

and this was accepted by postfix.

I set opportunistic TLS with

smtp_tls_security_level = may
smtpd_tls_security_level = may

I noticed that even after resolving the GMail delivery problem, a handful of other MTAs were still failing:

server1 postfix/smtpd[28167]: connect from mta3.email.secretescapes.com[198.245.84.110]
server1 postfix/smtpd[28167]: setting up TLS connection from mta3.email.secretescapes.com[198.245.84.110]
server1 postfix/smtpd[28167]: mta3.email.secretescapes.com[198.245.84.110]: TLS cipher list "ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH:!MD5:!DES:!ADH:!RC4:!PSD:!SRP:!3DES:!eNULL:!aNULL"
server1 postfix/smtpd[28167]: SSL_accept:before/accept initialization
server1 postfix/smtpd[28167]: SSL_accept:error in SSLv2/v3 read client hello A
server1 postfix/smtpd[28167]: SSL_accept error from mta3.email.secretescapes.com[198.245.84.110]: -1
server1 postfix/smtpd[28167]: warning: TLS library problem: 28167:error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:647:
server1 postfix/smtpd[28167]: lost connection after STARTTLS from mta3.email.secretescapes.com[198.245.84.110]
server1 postfix/smtpd[28167]: disconnect from mta3.email.secretescapes.com[198.245.84.110]

-

server1 postfix/smtpd[1451]: initializing the server-side TLS engine
server1 postfix/smtpd[1451]: connect from mta.email.bbc.com[198.245.84.99]
server1 postfix/smtpd[1451]: setting up TLS connection from mta.email.bbc.com[198.245.84.99]
server1 postfix/smtpd[1451]: mta.email.bbc.com[198.245.84.99]: TLS cipher list "ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH:!MD5:!aDSS:!kECDH:!kDH:!SEED:!IDEA:!DES:!ADH:!RC2:!RC4:!RC5:!PSD:!SRP:!3DES:!eNULL:!aNULL"
server1 postfix/smtpd[1451]: SSL_accept:before/accept initialization
server1 postfix/smtpd[1451]: SSL_accept:error in SSLv2/v3 read client hello A
server1 postfix/smtpd[1451]: SSL_accept error from mta.email.bbc.com[198.245.84.99]: -1
server1 postfix/smtpd[1451]: warning: TLS library problem: 1451:error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:647:
server1 postfix/smtpd[1451]: lost connection after STARTTLS from mta.email.bbc.com[198.245.84.99]
server1 postfix/smtpd[1451]: disconnect from mta.email.bbc.com[198.245.84.99]
server1 postfix/smtpd[1451]: connect from unknown[107.174.30.57]
server1 postfix/smtpd[1451]: 4F6D0629C: client=unknown[107.174.30.57]

I then tried a few permutations of the tls_high_cipherlist, before ending up using the bettercrypto.org recommended list:

tls_high_cipherlist=EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA

The server's been tested with various SSL testers, including CheckTLS (100% result), Luxsci (A+ result), htbridge's SSL tester (good grade) and SSL-Tools.

I'm aware you can policy map outgoing connections, but I'm looking for an equivalent so I can selectively disable any exclusions for particular MTAs (as I find them) and work out what cipher they are negotiating, then adjust the cipherlist I present.

I can't see any massively obvious flaws in the current postfix TLS configuration. I'm at a loss as to why certain MTAs can't negotiate a TLS cipher when others manage fine. Even the picky one, GMail, is now OK.

Any ideas? :-)

I noted this SE question about why Google prefers the cipher it does, which made for interesting reading.

Courtesy of this Postfix users forum thread, I found a useful recipe to quickly see how many TLS connections were successfully made across all of the available ciphers:

egrep "TLS connection established from.*with cipher" /var/log/maillog* | awk '{printf("%s %s %s %s\n", $12, $13, $14, $15)}' | sort | uniq -c | sort -

score 2 · Answer 1 · answered Aug 31 '18 at 16:54

Ultimately, this problem seems to be due to some senders still only being able to negotiate SSLv3. This is A Bad Thing, but not receiving emails is Also Not Great.

It is as simple as that really. I'm not happy with this, but if sending MTAs refuse to renegotiate to TLS, and just eternally retry with SSLv3, there's not much we can do for now.

So, for the time being, I've resorted to permitting incoming emails to be encrypted with SSLv3, but requiring all clients on the server to still have to authenticate via TLS.

I accomplished this by easing off the protocol negators in main.cf:

smtpd_tls_mandatory_protocols = !TLSv1, !SSLv2
smtpd_tls_protocols = !TLSv1, !SSLv2

NB that the smtp_tls_ definitions are for mail being sent from this server, so relate to client connections, so you can make these as strict as you like (as long as you're willing to explain to everyone how to set up their email clients):

smtp_tls_mandatory_protocols = !TLSv1, !SSLv2, !SSLv3
smtp_tls_protocols = !TLSv1, !SSLv2, !SSLv3

If sending MTAs are unable to negotiate TLS, expecting SSLv3, this is what you'll typically see in the maillog:

warning: TLS library problem: 364:error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:647:

If you dial up the logging level, you'll see the TLS negotiation failing, with more messages like

postfix/smtpd[26234]: SSL_accept:before/accept initialization
postfix/smtpd[26234]: SSL_accept:error in SSLv2/v3 read client hello A
postfix/smtpd[26234]: SSL_accept error from 201-62-89-201.life.com.br[201.62.89.201]: -1

I've been testing periodically disallowing SSLv3 connections (inevitably the MTAs retry for many days), so you can allow a few failures to accrue in the logs then analyse them.

This bash script helps me analyse which servers are attempting to use SSLv3:

#!/bin/sh
egrep "SSL_accept error" /var/log/maillog | awk '{printf("%s %s %s\n", $9, $10, $11)}' | sort | uniq -c | sort -`

And this variation shows negotiated ciphers - so once you re-enable SSLv3, you can leave it for a while then confirm that MTAs are using SSLv3:

#!/bin/sh
egrep "TLS connection established from.*with cipher" /var/log/maillog | awk '{printf("%s %s %s %s\n", $12, $13, $14, $15)}' | sort | uniq -c | sort -

I'm yet to find a better answer; I'd like to take a more nuanced approach and be able to selectively permit SSLv3 for certain MTAs depending on whether their connection attempts fail with certain errors in maillog. All comments / suggestions welcome.

I would recommend forgetting about ssl3. Are you really seeing that many legitimate email connections from it? I use tls1.2 (or higher) for all (smtp and smtpd). I don't see issues with Gmail or anyone else (remote or local). Don't mess with cipher lists. Just use exclude. IMO if a remote server cannot neg. v1.2 by now, they have deeper issues and deliver-ability problems going on. — B. Shea, Oct 26 '19 at 22:26

Postfix 2.6.6 with TLS - unable to receive emails from GMail (and a couple of other MTAs) but others are OK, why?

1 Answers1