16

A few days ago Gmail suddenly decided to stop sending mails to my mailserver. I am using Postfix and Dovecot with an paid SSL Certificate running on Debian 7 with everything updated.

My mail.log shows the following error:

Dec 19 11:09:11 server postfix/smtpd[19878]: initializing the server-side TLS engine
Dec 19 11:09:11 server postfix/tlsmgr[19880]: open smtpd TLS cache btree:/var/lib/postfix/smtpd_scache
Dec 19 11:09:11 server postfix/tlsmgr[19880]: tlsmgr_cache_run_event: start TLS smtpd session cache cleanup
Dec 19 11:09:11 server postfix/smtpd[19878]: connect from mail-wi0-x230.google.com[2a00:1450:400c:c05::230]
Dec 19 11:09:11 server postfix/smtpd[19878]: setting up TLS connection from mail-wi0-x230.google.com[2a00:1450:400c:c05::230]
Dec 19 11:09:11 server postfix/smtpd[19878]: mail-wi0-x230.google.com[2a00:1450:400c:c05::230]: TLS cipher list "aNULL:-aNULL:ALL:+RC4:@STR                              ENGTH:!aNULL:!DES:!3DES:!MD5:!DES+MD5:!RC4:!RC4-MD5"
Dec 19 11:09:11 server postfix/smtpd[19878]: SSL_accept:before/accept initialization
Dec 19 11:09:11 server postfix/smtpd[19878]: SSL_accept:error in unknown state
Dec 19 11:09:11 server postfix/smtpd[19878]: SSL_accept error from mail-wi0-x230.google.com[2a00:1450:400c:c05::230]: -1
Dec 19 11:09:11 server postfix/smtpd[19878]: warning: TLS library problem: 19878:error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown                               protocol:s23_srvr.c:647:
Dec 19 11:09:11 server postfix/smtpd[19878]: lost connection after STARTTLS from mail-wi0-x230.google.com[2a00:1450:400c:c05::230]
Dec 19 11:09:11 server postfix/smtpd[19878]: disconnect from mail-wi0-x230.google.com[2a00:1450:400c:c05::230]

excerpts from my postfix main.cf:

smtpd_use_tls=yes
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_CAfile = path to CA Bundle
smtpd_tls_cert_file= path to cert (pem)
smtpd_tls_key_file=path to key (pem)
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_exclude_ciphers = aNULL, DES, 3DES, MD5, DES+MD5, RC4, RC4-MD5
smtpd_tls_protocols=!SSLv2,!TLSv1,!TLSv1.1,!SSLv3
smtpd_tls_mandatory_ciphers = medium
smtpd_tls_received_header = yes
tls_preempt_cipherlist = yes
tls_medium_cipherlist = AES256+EECDH:AES256+EDH

I don't know where the problem is, because I regularly receive mails from others. There are no errors connecting to port 25 via telnet or port 465 via openssl

Addition: I got this mail in return from Google:

Delivery to the following recipient failed permanently:

     <removed>

Technical details of permanent failure:
TLS Negotiation failed

----- Original message -----
[...]

Maybe it's an issue with my cipherlist?

Answer to masegaloeh's question:

openssl s_client -connect localhost:25 -starttls smtp
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
[...]
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]
---
No client certificate CA names sent
---
SSL handshake has read 6267 bytes and written 477 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
Protocol  : TLSv1.2
Cipher    : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: [...]
Session-ID-ctx:
Master-Key: [...]
Key-Arg   : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 3600 (seconds)
TLS session ticket: [...]

Compression: 1 (zlib compression)
Start Time: 1418986680
Timeout   : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)

---
250 DSN

Update 1: Reissued my SSL certificate. Generated everything as following:
openssl req -nodes -newkey rsa:2048 -keyout myserver.key -out server.csr -sha256

I then created a new file consisting of the crt and the key, after this I created the CA bundle:
cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt > bundle.crt

Added everything to my dovecot and postfix config and restarted both services.
Google still fails to send mails zo my server resulting in TLS Negotiation failed

I tried another mail provider (web.de) and the mail gets send.
web.de log:

Dec 19 17:33:15 server postfix/smtpd[14105]: connect from mout.web.de[212.227.15.3]
Dec 19 17:33:15 server postfix/smtpd[14105]: setting up TLS connection from mout.web.de[212.227.15.3]
Dec 19 17:33:15 server postfix/smtpd[14105]: mout.web.de[212.227.15.3]: TLS cipher list "aNULL:-aNULL:ALL:+RC4:@STRENGTH"
Dec 19 17:33:15 server postfix/smtpd[14105]: mout.web.de[212.227.15.3]: save session EA1635ED786AFC2D9C7AB43EF43620A1D9092DC640FDE21C01E7BA25981D2445&s=smtp&l=268439647 to smtpd cache
Dec 19 17:33:15 server postfix/tlsmgr[14107]: put smtpd session id=EA1635ED786AFC2D9C7AB43EF43620A1D9092DC640FDE21C01E7BA25981D2445&s=smtp&l=268439647 [data 127 bytes]
Dec 19 17:33:15 server postfix/tlsmgr[14107]: write smtpd TLS cache entry EA1635ED786AFC2D9C7AB43EF43620A1D9092DC640FDE21C01E7BA25981D2445&s=smtp&l=268439647: time=1419006795 [data 127 bytes]
Dec 19 17:33:15 server postfix/smtpd[14105]: Anonymous TLS connection established from mout.web.de[212.227.15.3]: TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)

Soultion:
After enabling TLSv1 and TLSv1.1 in the smtpd_(mandatory)_protocols section everything works fine. Thanks masegaloeh ! 

Dec 20 11:44:46 server postfix/smtpd[31966]: initializing the server-side TLS engine
Dec 20 11:44:46 server postfix/tlsmgr[31968]: open smtpd TLS cache btree:/var/lib/postfix/smtpd_scache
Dec 20 11:44:46 server postfix/tlsmgr[31968]: tlsmgr_cache_run_event: start TLS smtpd session cache cleanup
Dec 20 11:44:46 server postfix/smtpd[31966]: connect from mail-wi0-x235.google.com[2a00:1450:400c:c05::235]
Dec 20 11:44:46 server postfix/smtpd[31966]: setting up TLS connection from mail-wi0-x235.google.com[2a00:1450:400c:c05::235]
Dec 20 11:44:46 server postfix/smtpd[31966]: mail-wi0-x235.google.com[2a00:1450:400c:c05::235]: TLS cipher list "aNULL:-aNULL:ALL:+RC4:@STRENGTH"
Dec 20 11:44:46 server postfix/smtpd[31966]: Anonymous TLS connection established from mail-wi0-x235.google.com[2a00:1450:400c:c05::235]: TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)
Octfx
  • 273
  • 1
  • 2
  • 7

3 Answers3

21

TLDR: Your TLS protocols is too strict because you only allow TLSv1.2 connection.

smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_protocols=!SSLv2,!TLSv1,!TLSv1.1,!SSLv3

And GMAIL send email to your server with TLSv1 protocol. That's why TLS negotiation fails.

The obvious solution is allowing TLSv1 and TLSv1.1 protocols and still disabling (insecure) SSLv2 and SSLv3 protocols.

Explanation

I can confirm your case when failing to receive email from GMAIL and FACEBOOK over STARTTLS.

Why do only GMAIL who fails to send email to my server

This is the maillog snippet when GMAIL send email

Dec 19 23:37:47 tls postfix/smtpd[3876]: initializing the server-side TLS engine
Dec 19 23:37:47 tls postfix/smtpd[3876]: connect from mail-wg0-f47.google.com[74.125.82.47]
Dec 19 23:37:48 tls postfix/smtpd[3876]: setting up TLS connection from mail-wg0-f47.google.com[74.125.82.47]
Dec 19 23:37:48 tls postfix/smtpd[3876]: mail-wg0-f47.google.com[74.125.82.47]: TLS cipher list "aNULL:-aNULL:ALL:+RC4:@STRENGTH:!aNULL:!DES:!3DES:!MD5:!DES+MD5:!RC4:!RC4-MD5"
Dec 19 23:37:48 tls postfix/smtpd[3876]: SSL_accept:before/accept initialization
Dec 19 23:37:48 tls postfix/smtpd[3876]: SSL_accept:error in unknown state
Dec 19 23:37:48 tls postfix/smtpd[3876]: SSL_accept error from mail-wg0-f47.google.com[74.125.82.47]: -1
Dec 19 23:37:48 tls postfix/smtpd[3876]: warning: TLS library problem: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:647:
Dec 19 23:37:48 tls postfix/smtpd[3876]: lost connection after STARTTLS from mail-wg0-f47.google.com[74.125.82.47]
Dec 19 23:37:48 tls postfix/smtpd[3876]: disconnect from mail-wg0-f47.google.com[74.125.82.47]

And this is the maillog snippet when FACEBOOK send email

Dec 19 23:11:14 tls postfix/smtpd[3844]: initializing the server-side TLS engine
Dec 19 23:11:14 tls postfix/tlsmgr[3846]: open smtpd TLS cache btree:/var/lib/postfix/smtpd_scache
Dec 19 23:11:14 tls postfix/tlsmgr[3846]: tlsmgr_cache_run_event: start TLS smtpd session cache cleanup
Dec 19 23:11:14 tls postfix/smtpd[3844]: connect from outcampmail003.ash2.facebook.com[66.220.155.162]
Dec 19 23:11:14 tls postfix/smtpd[3844]: setting up TLS connection from outcampmail003.ash2.facebook.com[66.220.155.162]
Dec 19 23:11:14 tls postfix/smtpd[3844]: outcampmail003.ash2.facebook.com[66.220.155.162]: TLS cipher list "aNULL:-aNULL:ALL:+RC4:@STRENGTH:!aNULL:!DES:!3DES:!MD5:!DES+MD5:!RC4:!RC4-MD5"
Dec 19 23:11:14 tls postfix/smtpd[3844]: SSL_accept:before/accept initialization
Dec 19 23:11:15 tls postfix/smtpd[3844]: SSL_accept:error in unknown state
Dec 19 23:11:15 tls postfix/smtpd[3844]: SSL_accept error from outcampmail003.ash2.facebook.com[66.220.155.162]: -1
Dec 19 23:11:15 tls postfix/smtpd[3844]: warning: TLS library problem: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:647:
Dec 19 23:11:15 tls postfix/smtpd[3844]: lost connection after STARTTLS from outcampmail003.ash2.facebook.com[66.220.155.162]
Dec 19 23:11:15 tls postfix/smtpd[3844]: disconnect from outcampmail003.ash2.facebook.com[66.220.155.162]
Dec 19 23:11:16 tls postfix/smtpd[3844]: connect from outcampmail004.ash2.facebook.com[66.220.155.163]
Dec 19 23:11:17 tls postfix/smtpd[3844]: 962C281443: client=outcampmail004.ash2.facebook.com[66.220.155.163]
Dec 19 23:11:18 tls postfix/cleanup[3849]: 962C281443: message-id=<722b2b198d163c43d3bf013bdd396817@www.facebook.com>
Dec 19 23:11:18 tls postfix/qmgr[3843]: 962C281443: from=<notification+zj4zc0zzjfac@facebookmail.com>, size=18002, nrcpt=1 (queue active)
Dec 19 23:11:18 tls postfix/local[3850]: 962C281443: to=<root@tls.example.net>, orig_to=<zera@tls.example.net>, relay=local, delay=1.6, delays=1.5/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Dec 19 23:11:18 tls postfix/qmgr[3843]: 962C281443: removed
Dec 19 23:11:24 tls postfix/smtpd[3844]: disconnect from outcampmail004.ash2.facebook.com[66.220.155.163]

Some analysis

  • In first snippet, GMAIL will try to send email over STARTTLS. When TLS negotiation, some error occurs, so GMAIL server disconnect it. We will discuss why the error occurring below.
  • In second snippet, FACEBOOK also failing to send email over STARTTLS. In fallback process, FACEBOOK resend email with plain text mode. In this case our server happily accept it.

So, that's explain why only GMAIL fails to send email to your server. GMAIL doesn't have mechanism to fallback if TLS negotiation fails. Other mail server may use fallback mechanism to ensure the email delivery succeed.

Why TLS Negotiation error occurs

I spot interesting line from web.de maillog

Dec 19 17:33:15 foxdev postfix/smtpd[14105]: Anonymous TLS connection established from mout.web.de[212.227.15.3]: TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)

And find out that you specify this configuration in main.cf

smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_protocols=!SSLv2,!TLSv1,!TLSv1.1,!SSLv3

That means your server only accept TLS connection when TLSv1.2 used. Other than TLSv1.2, your server will complain TLS negotiation error.

If I change smtpd_tls_(mandatory_)protocols to !SSLv2,!SSLv3,!TLSv1, the error still occurs. That means GMAIL and FACEBOOK will attempt contact your mail server with protocols other than TLSv1.1 and TLSv1.2.

If I change smtpd_tls_(mandatory_)protocols to !SSLv2,!SSLv3, TLS negotiation will success. It confirm that GMAIL and FACEBOOK will contact your server with TLSv1 protocol

Dec 20 00:21:46 tls postfix/smtpd[4261]: Anonymous TLS connection established from outmail038.prn2.facebook.com[66.220.144.165]: TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)
Dec 20 00:23:00 tls postfix/smtpd[4261]: Anonymous TLS connection established from mail-wi0-f174.google.com[209.85.212.174]: TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)

Other folks in FreeBSD forum also confirm this behavior.

Solution

The obvious solution is enable TLSv1 and TLSv1.1 in your postfix. This will ensure some mail server who doesn't have fallback mechanism - like GMAIL - can still communicate with your server.

I don't know your reason to disable TLSv1 and TLSv1.1 support, leaving only TLSv1.2 protocol. If it is an webserver and your user will use modern browser only, then you can disable TLSv1 in your server. This is acceptable because only older browser who doesn't support protocol TLSv1.

Community
  • 1
masegaloeh
  • 17,978
  • 9
  • 56
  • 104
1

For me problem was newer default value in /etc/postfix/main.cf

compatibility_level = 2

When I changed it to

compatibility_level = 1

It started working.

valdisvi
  • 41
  • 3
0

One potential issue I can see is the apparent use of a self-signed certificate as reported by OpenSSL:

Verify return code: 19 (self signed certificate in certificate chain)

If you are using a paid SSL certificate, you should not be using a self-signed cert.

I would check that your PEM file contains your paid certificate, and also verify that it contains the full certificate chain.

Craig Watson
  • 9,370
  • 3
  • 30
  • 46
  • Well the self signed certificate is the root cert from the CA which is self signed by the CA. – Octfx Dec 19 '14 at 13:30
  • Who is your CA? If they use self-signed certificates in their chain, you'll need to provide the entire chain in your .pem file. – Craig Watson Dec 19 '14 at 13:30
  • It is a certificate from Comodo. I do not use certificates signed by me. As said Comodo signs their root cert by them self which in this case results in `code: 19 (self signed certificate)` – Octfx Dec 19 '14 at 13:33
  • 1
    You shouldn't receive a `code 19` message if your full chain is being served. I use a cert from StartSSL, which gives the same error at the top of the command, but as I provide the full chain (including the root CA) within the `smtpd_tls_cert_file` PEM file, the client has all of the certificates necessary to validate the full chain. – Craig Watson Dec 19 '14 at 13:37
  • I will reissue my cert just to test it. The Problem is, that I did not change anything, google just can't deliver mails to me anymore – Octfx Dec 19 '14 at 16:13
  • So, I created my CA bundle without the root cert from Comodo and added `-CApath /etc/ssl/certs` to the `openssl`command. This results in `Verify return code: 0 (ok)` – Octfx Dec 19 '14 at 18:33