2

I am trying to secure a rethinkdb cluster behind stunnel. The service needs to support multiple Certificate Authorities (CA). Currently I concat the accepted CAs into one file (/certs/ca.pem) but it seems stunnel will only accept connections that match the first certificate in the file.

My stunnel configuration:

foreground = yes
sslVersion = TLSv1.2
options = NO_SSLv2
options = NO_SSLv3

[driver]
client = no
accept = 28415
connect = 127.0.0.1:28015
cert = /certs/server.pem
key = /certs/server-key.pem
CAfile = /certs/ca.pem
verify = 2

Stunnel version 5.06

Stunnel's log:

2016.02.18 22:18:51 LOG5[18]: Service [driver] accepted connection from 209.136.228.130:58728
2016.02.18 22:18:51 LOG4[18]: CERT: Verification error: self signed certificate
2016.02.18 22:18:51 LOG4[18]: Rejected by CERT at depth=0: C=US, OU=Edit LLC, L=Fresno, O=Edit LLC, ST=CA, CN=jason-Lemur-Ultra
2016.02.18 22:18:51 LOG3[18]: SSL_accept: 140890B2: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
2016.02.18 22:18:51 LOG5[18]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket

And on the client side I get the following error:

SSL handshake failed: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:590)

I am not sure why stunnel says no certificate returned.

Edit: The error is coming from openssl. Here is how I can reproduce:

$ cat ca.cert.pem incomming-ca.pem > bigca.pem
$ openssl verify -CAfile bigca.pem incomming-ca.pem 
incomming-ca.pem: C = US, OU = Edit LLC, L = Fresno, O = Edit LLC, ST = CA, CN = jason-Lemur-Ultra
error 18 at 0 depth lookup:self signed certificate
OK
$ openssl verify -CAfile bigca.pem ca.cert.pem 
ca.cert.pem: OK
$ cat incomming-ca.pem ca.cert.pem > bigca.pem
$ openssl verify -CAfile bigca.pem incomming-ca.pem 
incomming-ca.pem: OK

Edit(2): Here I try to verify a signed certificate instead of sending a root CA

$ openssl genrsa -des3 -out server.key 1024
$ openssl req -new -key server.key -out server.csr
$ openssl x509 -req -days 360 -in server.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.crt
$ cat ca.cert.pem incomming-ca.pem > bigca.pem
$ openssl verify -CAfile bigca.pem server.crt 
server.crt: OK

Cool, but lets switch the order of bigca.pem

$ cat incomming-ca.pem ca.cert.pem > bigca.pem
$ openssl verify -CAfile bigca.pem server.crt 
server.crt: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
error 7 at 0 depth lookup:certificate signature failure
140351186847392:error:0407006A:rsa  routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
140351186847392:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:
140351186847392:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:233:
zbyte
  • 121
  • 1
  • 5
  • Server is using: `OpenSSL 1.0.1k` Client is using: `python 2.7.11` & `OpenSSL 1.0.1f` – zbyte Feb 19 '16 at 00:37
  • Make sure the cat'ed file (bigca.pem) is **correctly formatted**: first cert `-----BEGIN` line, data, `-----END` line *with LF or CRLF*, and `-----BEGIN` line for second cert starts at the *beginning of* the next line or a subsequent line (after comments) but *nothing else*: no space, Unicode BOM or suchlike. Use `cat -A` or `sed -n l` or similar to be certain. – dave_thompson_085 Feb 19 '16 at 02:39
  • Verified bigca.pem was cat'ed properly. using `cat -A` to make the file actually causes openssl to have an error reading – zbyte Feb 19 '16 at 03:00
  • 1
    No, I meant create 'bigca' then use `cat -A` or similar to *display* it and look to see if there any normally-invisible characters where they shouldn't be. But your edit worries me; it suggests the certs in `ca.cert.pem` and `incomming-ca.pem` have the **same (CA) name**. Try `openssl x509 -noout -subject -issuer` on each file separately. Every CA root cert should have the same value for subject and issuer and that value should be *different from any other CA cert*. If there are multiple CA certs *with the same name* then OpenSSL may be unable to use more than one of them. – dave_thompson_085 Feb 19 '16 at 06:11
  • Bingo. Thank you! – zbyte Feb 19 '16 at 07:20

1 Answers1

2

By configuring stunnel to require client certificates, using:

verify = 2

You are telling stunnel to drop/refuse any clients who do not provide a valid client certificate. And this log message indicates that the client didn't provide a client certificate, and is thus rejected:

SSL3_GET_CLIENT_CERTIFICATE:no certificate returned

This we know. Now, for why this happens. That client-side message is our hint:

[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca

A TLS server requests that the client send its client certificate by sending a list of trusted CAs; these are the CA certificates that are in your /certs/ca.pem file. The client, then, looks for a certificate that comes from one of those CAs; if the client does not have a certificate at all, or does not have a certificate which comes from one of those CAs, then the client will not provide a certificate at all.

The fact that your client is saying that it does not recognize any of the CAs sent by the server says that your client either a) doesn't have a client certificate, or b) its client certificate is from a CA that is not in the /certs/ca.pem file.

I am not sure which TLS client you are using, so I cannot help with that, but the above suggests that you check the configuration of the client certificate/key for that client, and check that the certificate configured for use by the client is from one of the CAs in your /certs/ca.pem file.

Hope this helps!

Castaglia
  • 3,239
  • 3
  • 19
  • 40
  • I am using python 2.7.11 on the client side. What is strange is I can get the client to connect without changing the CA certs param, rather I just use the first cert mentioned in `/certs/ca.pem` with the matching key. Why would stunnel say no certificate return when right above it says it rejected a self signed cert? I know those messages were caused by a single connection attempt. – zbyte Feb 19 '16 at 00:12
  • 1
    Those log messages might from the OpenSSL error queue. _I.e._ perhaps OpenSSL rejected the self-signed cert as part of the _verification_ process. Then, later, `stunnel` asks for the _verified_ peer certificate, and gets none -- because it was rejected earlier. – Castaglia Feb 19 '16 at 00:13
  • That makes sense. I have been switching between two self signed certs, both of which are in `/certs/ca.pem`, if I switch the ordering in `ca.pem` I can get a previously rejected cert to be accepted but then it rejects the other. – zbyte Feb 19 '16 at 00:35
  • I suspect that `stunnel`'s verification code is not really expecting to verify a self-signed cert provided by a client; such things are not very common (or recommended) in a production deployment. If you use one of those CAs to issue a cert for your client, and then use that cert in your client, it _should_ work as you expect. – Castaglia Feb 19 '16 at 00:38
  • You are right. I can reproduce the failure locally :) Editing post to show. – zbyte Feb 19 '16 at 00:42
  • Does this mean you will accept my answer for your question? :) – Castaglia Feb 19 '16 at 01:08
  • Client is definitely sending a cert because server is trying to validate it and failing. Server sends back an alert, and the client log message means it *received* the alert from the server saying 'unknown ca'. – dave_thompson_085 Feb 19 '16 at 02:46
  • Unfortunately sending a cert signed by one of the CAs produces the same results. – zbyte Feb 19 '16 at 03:06