Puppet's automatically generated certificates failing

Question

I am running a default configuration of Puppet on Debian Squeeze 6.0.4.

The server's FQDN is master.example.com. The client's FQDN is client.example.com.

I am able to contact the puppet master and send a CSR. I sign it using puppetca -sa but the client will still not connect.

The two machines have accurate date and time.

This is what appears in /var/log/syslog:

Apr  3 17:03:52 localhost puppet-agent[18653]: Reopening log files
Apr  3 17:03:52 localhost puppet-agent[18653]: Starting Puppet client version 2.6.2
Apr  3 17:03:53 localhost puppet-agent[18653]: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Apr  3 17:03:53 localhost puppet-agent[18653]: Using cached catalog
Apr  3 17:03:53 localhost puppet-agent[18653]: Could not retrieve catalog; skipping run

Here is some interesting output:

OpenSSL client test:

client:~# openssl s_client -host master.example.com -port 8140 -cert /var/lib/puppet/ssl/certs/client.example.com.pem -key /var/lib/puppet/ssl/private_keys/client.example.com.pem -CAfile /var/lib/puppet/ssl/certs/ca.pem
CONNECTED(00000003)
depth=1 /CN=Puppet CA: master.example.com
verify return:1
depth=0 /CN=master.example.com
verify error:num=7:certificate signature failure
verify return:1
depth=0 /CN=master.example.com
verify return:1
18509:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error:s3_pkt.c:1102:SSL alert number 51
18509:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:
client:~#

master's certificate:

root@master:/etc/puppet# openssl x509 -text -noout -in /etc/puppet/ssl/certs/master.example.com.pem
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 2 (0x2)
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN=Puppet CA: master.example.com
    Validity
        Not Before: Apr  2 20:01:28 2012 GMT
        Not After : Apr  2 20:01:28 2017 GMT
    Subject: CN=master.example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSA Public Key: (1024 bit)
            Modulus (1024 bit):
                00:a9:c1:f9:4c:cd:0f:68:84:7b:f4:93:16:20:44:
                7a:2b:05:8e:57:31:05:8e:9c:c8:08:68:73:71:39:
                c1:86:6a:59:93:6e:53:aa:43:11:83:5b:2d:8c:7d:
                54:05:65:c1:e1:0e:94:4a:f0:86:58:c3:3d:4f:f3:
                7d:bd:8e:29:58:a6:36:f4:3e:b2:61:ec:53:b5:38:
                8e:84:ac:5f:a3:e3:8c:39:bd:cf:4f:3c:ff:a9:65:
                09:66:3c:ba:10:14:69:d5:07:57:06:28:02:37:be:
                03:82:fb:90:8b:7d:b3:a5:33:7b:9b:3a:42:51:12:
                b3:ac:dd:d5:58:69:a9:8a:ed
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:FALSE
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Subject Key Identifier:
            8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha1WithRSAEncryption
    7b:2c:4f:c2:76:38:ab:03:7f:c6:54:d9:78:1d:ab:6c:45:ab:
    47:02:c7:fd:45:4e:ab:b5:b6:d9:a7:df:44:72:55:0c:a5:d0:
    86:58:14:ae:5f:6f:ea:87:4d:78:e4:39:4d:20:7e:3d:6d:e9:
    e2:5e:d7:c9:3c:27:43:a4:29:44:85:a1:63:df:2f:55:a9:6a:
    72:46:d8:fb:c7:cc:ca:43:e7:e1:2c:fe:55:2a:0d:17:76:d4:
    e5:49:8b:85:9f:fa:0e:f6:cc:e8:28:3e:8b:47:b0:e1:02:f0:
    3d:73:3e:99:65:3b:91:32:c5:ce:e4:86:21:b2:e0:b4:15:b5:
    22:63
root@master:/etc/puppet#

CA's certificate:

root@master:/etc/puppet# openssl x509 -text -noout -in /etc/puppet/ssl/certs/ca.pem
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN=Puppet CA: master.example.com
    Validity
        Not Before: Apr  2 20:01:05 2012 GMT
        Not After : Apr  2 20:01:05 2017 GMT
    Subject: CN=Puppet CA: master.example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSA Public Key: (1024 bit)
            Modulus (1024 bit):
                00:b5:2c:3e:26:a3:ae:43:b8:ed:1e:ef:4d:a1:1e:
                82:77:78:c2:98:3f:e2:e0:05:57:f0:8d:80:09:36:
                62:be:6c:1a:21:43:59:1d:e9:b9:4d:e0:9c:fa:09:
                aa:12:a1:82:58:fc:47:31:ed:ad:ad:73:01:26:97:
                ef:d2:d6:41:6b:85:3b:af:70:00:b9:63:e9:1b:c3:
                ce:57:6d:95:0e:a6:d2:64:bd:1f:2c:1f:5c:26:8e:
                02:fd:d3:28:9e:e9:8f:bc:46:bb:dd:25:db:39:57:
                81:ed:e5:c8:1f:3d:ca:39:cf:e7:f3:63:75:f6:15:
                1f:d4:71:56:ed:84:50:fb:5d
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:TRUE
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Certificate Sign, CRL Sign
        X509v3 Subject Key Identifier:
            8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
Signature Algorithm: sha1WithRSAEncryption
    1d:cd:c6:65:32:42:a5:01:62:46:87:10:da:74:7e:8b:c8:c9:
    86:32:9e:c2:2e:c1:fd:00:79:f0:ef:d8:73:dd:7e:1b:1a:3f:
    cc:64:da:a3:38:ad:49:4e:c8:4d:e3:09:ba:bc:66:f2:6f:63:
    9a:48:19:2d:27:5b:1d:2a:69:bf:4f:f4:e0:67:5e:66:84:30:
    e5:85:f4:49:6e:d0:92:ae:66:77:50:cf:45:c0:29:b2:64:87:
    12:09:d3:10:4d:91:b6:f3:63:c4:26:b3:fa:94:2b:96:18:1f:
    9b:a9:53:74:de:9c:73:a4:3a:8d:bf:fa:9c:c0:42:9d:78:49:
    4d:70
root@master:/etc/puppet#

Client's certificate:

client:~# openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/client.example.com.pem
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 3 (0x3)
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN=Puppet CA: master.example.com
    Validity
        Not Before: Apr  2 20:01:36 2012 GMT
        Not After : Apr  2 20:01:36 2017 GMT
    Subject: CN=client.example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSA Public Key: (1024 bit)
            Modulus (1024 bit):
                00:ae:88:6d:9b:e3:b1:fc:47:07:d6:bf:ea:53:d1:
                14:14:9b:35:e6:70:43:e0:58:35:76:ac:c5:9d:86:
                02:fd:77:28:fc:93:34:65:9d:dd:0b:ea:21:14:4d:
                8a:95:2e:28:c9:a5:8d:a2:2c:0e:1c:a0:4c:fa:03:
                e5:aa:d3:97:98:05:59:3c:82:a9:7c:0e:e9:df:fd:
                48:81:dc:33:dc:88:e9:09:e4:19:d6:e4:7b:92:33:
                31:73:e4:f2:9c:42:75:b2:e1:9f:d9:49:8c:a7:eb:
                fa:7d:cb:62:22:90:1c:37:3a:40:95:a7:a0:3b:ad:
                8e:12:7c:6e:ad:04:94:ed:47
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:FALSE
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Subject Key Identifier:
            8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha1WithRSAEncryption
    33:1f:ec:3c:91:5a:eb:c6:03:5f:a1:58:60:c3:41:ed:1f:fe:
    cb:b2:40:11:63:4d:ba:18:8a:8b:62:ba:ab:61:f5:a0:6c:0e:
    8a:20:56:7b:10:a1:f9:1d:51:49:af:70:3a:05:f9:27:4a:25:
    d4:e6:88:26:f7:26:e0:20:30:2a:20:1d:c4:d3:26:f1:99:cf:
    47:2e:73:90:bd:9c:88:bf:67:9e:dd:7c:0e:3a:86:6b:0b:8d:
    39:0f:db:66:c0:b6:20:c3:34:84:0e:d8:3b:fc:1c:a8:6c:6c:
    b1:19:76:65:e6:22:3c:bf:ff:1c:74:bb:62:a0:46:02:95:fa:
    83:41
client:~#

EDIT:

Someone suggested redoing the certificates. I have done this several times and it does not fix the issue.

score 6 · Answer 1 · answered Nov 28 '12 at 02:11

6

I just hit a similar issue on CentOS 6.3. The problem was dat/etime out of sync. I fixed that and all worked just fine.

answered Nov 28 '12 at 02:11

Robin Bowes

69
1
1

Thanks @RobinBowes, this was exactly my issue. It happened to me because I was rebuilding a server – Aaron R. Aug 12 '14 at 17:27

score 5 · Accepted Answer · edited Oct 08 '15 at 16:43

5

I've seen this before when something has gone weird with the cert signing process. The easiest way to fix this is to regen the client certs.

revoke the cert: puppetca revoke client.example.com
run puppetca clean on the client
stop puppet on the client
remove everything in the /var/lib/puppet/ssl directory
start puppet on the client
do an initial run I tend to use puppet agent --test
sign the new certificate on the server
test the client

edited Oct 08 '15 at 16:43

AndiDog

321
2
5
19

answered Apr 03 '12 at 22:12

Zypher

36,995
5
52
95

I have done this several times. I am looking for a way to diagnose more specifically what is going on. – gparent Apr 04 '12 at 02:48
I'll accept your answer even if it didn't solve the issue because it's probably the most logical one for anyone else who gets problems. – gparent Feb 24 '13 at 20:57
Fun fact... steps 3 and 5 are REALLY important :-D If you happen to forget to restart the puppet agent service then manual `puppet agent --test` invocations will work fine and you will see the previous agent process print the correct `Certificate fingerprint: ...` value in syslog on interval runs (or when forced using `pkill -SIGUSR1 puppet`). But those runs will fail with `Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey: key values mismatch` errors, likely because the private key resident in memory doesn't match the new public cert. Thanks Zypher! – Greg Bray May 01 '20 at 00:13

score 2 · Answer 3 · answered Oct 14 '15 at 00:03

2

Make sure your not out of disk space. Puppet will not warn of this it will just create certificates that are of 0 length.

answered Oct 14 '15 at 00:03

jond

21
1

score 1 · Answer 4 · answered Apr 03 '12 at 22:42

1

Check the date/time on the client system. If it is significantly out of sync, you will get SSL authentication errors.

answered Apr 03 '12 at 22:42

Kyle Smith

9,563
1
30
32

See question for this. – gparent Apr 04 '12 at 02:47
Ah, thanks for updating the question to include this information. – Kyle Smith Apr 26 '12 at 10:00
It was actually in the very first revision, but no problem. – gparent May 18 '12 at 14:54

score 0 · Answer 5 · answered Nov 28 '12 at 02:35

0

I've had this before over slow links (wan links) - the ssl handshake times out.

I fixed it by using passenger & apache instead of the webrick server.

answered Nov 28 '12 at 02:35

Sirex

5,447
2
32
54

Puppet's automatically generated certificates failing

5 Answers5

Linked