Why SSL does not work between my 2 servers?

Question

I have 2 servers with Ubuntu 18.04:

monitoring.example.com (with ELK on a single server)
www.example.com (with Filebeat)

on the server ELK

Create directories to store SSL certificates

$ sudo mkdir -p /etc/elk-certs

Generate SSL Certificates

$ sudo openssl req -subj '/CN=monitoring.example.com/' -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout /etc/elk-certs/monitoring-example-com.key -out /etc/elk-certs/monitoring-example-com.crt

Change the owner

$ sudo chown logstash /etc/elk-certs/monitoring-example-com.crt
$ sudo chown logstash /etc/elk-certs/monitoring-example-com.key

Send the SSL certificate to the client server

$ sudo scp /etc/elk-certs/monitoring-example-com.crt root@22.22.22.222:/tmp

on the server client

Create the directories to store the SSL certificate

$ sudo mkdir -p /etc/elk-certs

Copy the certificate into the directory

$ sudo mv /tmp/monitoring-example-com.crt /etc/elk-certs/

on the server ELK

Here is the configuration file /etc/logstash/conf.d/logstash.conf on the server monitoring.example.com :

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/elk-certs/monitoring-example-com.crt"
    ssl_key => "/etc/elk-certs/monitoring-example-com.key"      
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Restart Logstash

$ sudo systemctl restart logstash

on the server client

Here is the configuration file /etc/filebeat/filebeat.yml on the server www.example.com :

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["monitoring.example.com:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/elk-certs/monitoring-example-com.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/elk-certs/monitoring-example-com.crt"

  # Client Certificate Key
  #ssl.key: "/etc/elk-certs/monitoring-example-com.key"

Restart Filebeat

$ sudo systemctl restart filebeat

PROBLEM

$ curl -v --cacert /etc/elk-certs/monitoring-example-com.crt https://monitoring.example.com:5044

* Rebuilt URL to: https://monitoring.example.com:5044/
*   Trying 2001:43d9:363:1000::2b16...
* TCP_NODELAY set
*   Trying 51.95.207.228...
* TCP_NODELAY set
* Connected to monitoring.example.com (51.95.207.228) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/elk-certs/monitoring-example-com.crt
  CApath: /etc/ssl/certs
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=monitoring.example.com
*  start date: May 11 22:26:42 2019 GMT
*  expire date: May  8 22:26:42 2029 GMT
*  subjectAltName does not match monitoring.example.com
* SSL: no alternative certificate subject name matches target host name 'monitoring.example.com'
* stopped the pause stream!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'monitoring.example.com'

Currently Logstash does not receive any data from Filebeat.

I've never dealt with ELK, but does the cert need to have the proper extensions, such as `keyUsage = critical, digitalSignature, keyEncipherment` or `extendedKeyUsage = serverAuth`? — user3629081, May 13 '19 at 21:51
Can you share the output of a `openssl s_client -connect monitoring.example.com:5044`? — user3629081, May 14 '19 at 02:18

Sergey Nudnov · Accepted Answer · 2019-05-14T03:53:12.200

1

You are missing the SAN record in the certificate.

Generate certificate again with the following commands:

basename=/etc/elk-certs/monitoring-example-com

openssl req -newkey rsa:4096 -nodes -keyout $basename.key -subj "/CN=monitoring.example.com" -out $basename.csr

openssl x509 -req -extfile <(printf "subjectAltName=DNS:monitoring.example.com") -sha256 -days 3650 -in $basename.csr -signkey $basename.key -out $basename.crt

I have added -sha256 above, but you could remove that if you wish

Test the generated certificate:

openssl x509 -in $basename.crt -text -noout

There should be the following data:

Subject: CN=monitoring.example.com

X509v3 extensions:
    X509v3 Subject Alternative Name:
        DNS:monitoring.example.com

Also ensure to remove comments there:

# Certificate for SSL client authentication
ssl.certificate: "/etc/elk-certs/monitoring-example-com.crt"

# Client Certificate Key
ssl.key: "/etc/elk-certs/monitoring-example-com.key"

edited May 14 '19 at 03:53

answered May 14 '19 at 00:37

Sergey Nudnov

833
6
12

@tropcool my bad about first suggestion. Still problem is in certificate. It is looking for SAN (Subject Alternative Name). I have updated my answer. The correct certificate should be on the server. But because it is self-signed certificate, client should have it too - as a certificate authority. So it is possible you don't need to uncomment client authentication and client certificate lines, as I advised in answer. – Sergey Nudnov May 14 '19 at 03:52
@tropcool Your certificate has wrong SAN. As of the commands, what did you count second command? `openssl x509 -req -extfile`? – Sergey Nudnov May 14 '19 at 11:44
@tropcool You could make it with your original command too. Locate the openssl.cnf file and in the `[SAN]` section in it replace parameter `subjectAltName` with this `subjectAltName=DNS:monitoring.example.com` – Sergey Nudnov May 14 '19 at 11:46
@tropcool Do you see the subjectAltName in your openssl.cnf? Was you able to replace it and make the new certificate? – Sergey Nudnov May 14 '19 at 12:53
@tropcool looks good – Sergey Nudnov May 14 '19 at 13:14
@tropcool You have totally different problem now. Your server is not up. If you made any changes in its config, like uncommenting the certificate lines, revert them. The only change you should have is a new key/certificate pair – Sergey Nudnov May 14 '19 at 13:38
@tropcool SSL3 handshake would fail if SSL3 is disabled on server. I see, it establishes TLS1.2 connection. Try to run test connection command with the `-tls1` option - it should provide no errors. Also if I was able to help you, could you please mark my answer as accepted by clicking on `v` under the answer's score counter – Sergey Nudnov May 16 '19 at 03:07
@tropcool It is quite a task to assess. Why I do know stuff about SSL certificates and communications overall, I don't know nothing about your applications. And, unfortunately, I don't understand French well. So there I could not help – Sergey Nudnov May 16 '19 at 19:08

Why SSL does not work between my 2 servers?

1 Answers1