I recently revoked/cleaned a Puppet agent cert, and this seems to have negative effects in PuppetDB. I see a bug has been filed here with some instructions on fixing the issue. A user had a similar issue here, but none of this is working for me.
The server is running CentOS 6.2, Puppet 2.7.13, and Puppet DB 0.9. The error is:
root@harp:/etc/puppetdb/ssl> puppet agent --test
err: Cached facts for harp failed: Failed to find facts from PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
info: Loading facts in /etc/puppet/modules/dns/lib/facter/datacenter.rb
info: Caching facts for harp
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
err: Could not run Puppet configuration client: Could not retrieve local facts: Failed to submit 'replace facts' command for harp to PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
NTP is working properly from what I see and the datetime looks good. "harp" is actually the puppet master server, so there shouldn't be an issue with time between the agent and server here since they're the same.
Old certificate:
root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp (DF:8F:65:36:58:4C:DE:66:2B:65:D1:E6:18:B7:F2:33)
Clean and generate new cert for agent:
root@harp:/etc/puppetdb/ssl> puppet cert clean harp
notice: Revoked certificate with serial 18
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/ca/signed/harp.pem'
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/certs/harp.pem'
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/certificate_requests/harp.pem'
notice: Removing file Puppet::SSL::Key harp at '/var/lib/puppet/ssl/private_keys/harp.pem'
root@harp:/etc/puppetdb/ssl> puppet agent --test
info: Creating a new SSL key for harp
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for harp
info: Certificate Request fingerprint (md5): 72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled
root@harp:/etc/puppetdb/ssl> puppet cert list
harp (72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD)
root@harp:/etc/puppetdb/ssl> puppet cert sign harp
notice: Signed certificate request for harp
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/ca/requests/harp.pem'
root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp (4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79)
root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb: /etc/init.d/puppetdb: line 77: kill: (8623) - No such process
[FAILED]
Starting puppetdb: [ OK ]
OK then, restart again for good measure:
root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb: [ OK ]
Starting puppetdb: [ OK ]
Run the SSL configuration script
root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
cp: cannot stat `/var/lib/puppet/ssl/certs/harp.pem': No such file or directory
root@harp:/etc/puppetdb/ssl> ls -la /var/lib/puppet/ssl/certs
total 12
drwxr-xr-x 2 puppet root 4096 Jun 19 07:19 ./
drwxrwx--x 8 puppet root 4096 Apr 24 10:04 ../
-rw-r--r-- 1 puppet root 1854 Apr 24 10:04 ca.pem
OK then, try again for good measure:
root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
Certificate was added to keystore
Usage: pkcs12 [options]
where options are
-export output PKCS12 file
-chain add certificate chain
-inkey file private key if not infile
-certfile f add all certs in f
-CApath arg - PEM format directory of CA's
-CAfile arg - PEM format file of CA's
-name "name" use name as friendly name
-caname "nm" use nm as CA friendly name (can be used more than once).
-in infile input filename
...snip...
-CSP name Microsoft CSP name
-LMK Add local machine keyset attribute to private key
It does not appear that the keystores in /etc/puppetdb/ssl have changed/regenerated. At this point, running puppet agent --test results in the same errors, and restarting puppet and puppetdb do not help.
Keystore info:
root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/keystore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
harp.mydomain.com, May 25, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): 06:A8:D3:2A:70:F3:6D:34:62:91:45:22:8A:C4:A8:86
root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
puppetdb ca, May 25, 2012, trustedCertEntry,
Certificate fingerprint (MD5): 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp.mydomain.com
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
err: Could not call fingerprint: Could not find a certificate or csr for harp.mydomain.com
root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
harp 4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79
How can I get the puppetdb keystore to actually regenerate? I tried deleting the files in /etc/puppetdb/ssl/, but no luck.
