11

I have a Puppet Master/Agent set up, and have successfully signed the certificate for the agent on the master. However, when I run puppet agent --test I get a failure that looks like this:

Warning: Unable to fetch my node definition, but the agent run will continue:  
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Info: Retrieving plugin  
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Error: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com] Could not retrieve file metadata for puppet://hostname.domain.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Warning: Not using cache on failed catalog   
Error: Could not retrieve catalog; skipping run  
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]

the hostname.domain.com is the master

How do I fix this? I've made sure that both clocks are at the correct time in the same time zone, I've deleted everything in the agent /var/lib/puppet/ssl directory and resigned, I don't know what else to do.

030
  • 5,731
  • 12
  • 61
  • 107
John Smith
  • 111
  • 1
  • 1
  • 3
  • Your master seems to be using a different certificate than your client trusts? Has the master's certificate changed? – Shane Madden Jun 12 '13 at 16:29
  • @ShaneMadden I do not think so...should I clean and revoke the masters and the clients certificates? I haven't messed with the masters certificates at all, but here's what the output of "puppet cert list --all" looks like: + "masterhost.domain.com" (SHA1) E1:F7:6A:21:CB:CD:xx:xx:xx:xx... + "agenthost.domain.com" (SHA256) 5A:D9:7B:96:0B:FF:E4:87:58:AF:00:xx:xx:xx:xx.. – John Smith Jun 12 '13 at 16:35
  • And that `masterhost.domain.com` is the same one as `hostname.domain.com` in your question, right? Let's try this, we'll see if the certificates verify manually; run `openssl s_client -connect masterhost.domain.com:8140 -showcerts`, and copy the certificate data (starts with `-----BEGIN CERTIFICATE-----`, include that line and the end certificate line) into a new file, then run `openssl verify -CAfile /var/lib/puppet/ssl/certs/ca.pem /path/to/file/from/last/command`, and see if it verifies. – Shane Madden Jun 12 '13 at 17:32
  • @ShaneMadden It seems like something is amiss.... When I ran the "-showcerts" command, it gave me two "begin" and "end" certificates, so I tried first adding one of those to a new file, and got this: /var/lib/puppet/ssl/ca/test: /CN=masterhost.domain.com error 7 at 0 depth lookup:certificate signature failure 22297:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100: 22297:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:697: 22297:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:173: – John Smith Jun 12 '13 at 17:52
  • That's.. strange. It sounds like it's sending the root certificate in addition to the server cert in the connection, so maybe just compare the contents of the second cert from `-showcerts` with the contents of `/var/lib/puppet/ssl/certs/ca.pem` - they should be identical? – Shane Madden Jun 12 '13 at 18:24
  • @ShaneMadden They are not! Ca.pem is different from the second output of the -showcerts command – John Smith Jun 12 '13 at 18:32
  • Interesting! Can you get that second cert from `-showcerts` into a file, the compare `openssl x509 -in /path/to/cert -noout -text` between the two? – Shane Madden Jun 12 '13 at 18:44

2 Answers2

14

Re-create the entire client certificate setup. This has always fixed any cert issues we have experienced in the past. The following instruction assumes your agent's hostname is agenthost.hostname.com

On the client, delete all stored certs, including the CA:

find /var/lib/puppet/ssl -name '*.pem' -delete

On the master, delete any pending CSRs or old client certificates for this client:

find /var/lib/puppet/ssl -name agenthost.domain.com.pem -delete

Then, on the client, reconnect to the master and send a CSR:

puppet agent -t --waitforcert=60

and when it is waiting (if you have not set autosigning enabled) then on the master approve the CSR so a new client cert is sent back:

puppet cert sign agenthost.domain.com

This should make the agent re-download the puppet CA certificates, and re-apply for its own certificate.

We had to use this procedure in the past when we changed puppet servers and the CA certs changed, or when we rebuilt a host with the same hostname.

Make sure your agent knows its real fully-qualified hostname; use the 'hostname' command to ensure that it is what you expect it to be.

M. Glatki
  • 1,868
  • 1
  • 16
  • 33
Steve Shipway
  • 742
  • 5
  • 17
  • 3
    On Master, `puppet cert clean ‘agentName’` works for me. RHEL Puppet Enterprise ssl location: `/etc/puppetlabs/puppet/ssl`. And I've synced the time between Master and Agent. – Ivan Chau May 02 '14 at 03:31
  • Puppet Enterprise, and the newer Puppet 4, store their certs under /etc/puppetlabs/puppet/ssl, as mentioned above. The /var/lib/puppet/ssl location is for the Puppet 2.x and 3.x community editions. – Steve Shipway Dec 21 '15 at 00:28
  • With puppet 5 or later, try `puppetserver ca clean --certname $HOST` on the Puppetserver, and `puppet ssl clean` on the client with the problem (use the client fqdn for $HOST) – Steve Shipway Oct 28 '21 at 00:22
7

I have a similar problem. I have set up a vagrant environment with one puppetmaster and several clients. The problem is when I destroy and create the puppetmaster, clients detect the new puppetmaster as an impostor.

Deleting /etc/puppet/ssl on the client solves the problem.

Remember that your ssl configuration will be cached, so a restart of the puppet master is required, if you decide to also delete your /etc/puppet/ssl on that host:

sudo /etc/init.d/puppetmaster restart
Andrew Schulman
  • 8,561
  • 21
  • 31
  • 47
adrianlzt
  • 189
  • 1
  • 2