Puppet agent to puppet master (running on nginx/ruby1.9) SSL certificate issues

Question

I have two machines, puppet master--host name puppet--and a lone client, host name git. The puppet agent on the master machine works without issue. The agent on git fails with a '400 No required SSL certificate was sent'. First, the configuration of the puppet master, which is a thin/nginx affair:

puppet:~# ruby -v
ruby 1.9.2p0 (2010-08-18 revision 29036) [i486-linux]

puppet:~# puppet --version
2.7.9

puppet:~# cat /etc/nginx/sites-enabled/default 
server {
  listen puppet:8140;

  ssl on;
  ssl_certificate /var/lib/puppet/ssl/certs/puppet.pem;
  ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppet.pem;
  ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
  ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
  ssl_verify_client on;

  proxy_redirect   off;
  proxy_set_header Host             $host;
  proxy_set_header X-Real-IP        $remote_addr;
  proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
  proxy_set_header X-Client-Verify  $ssl_client_verify;
  proxy_set_header X-Client-Verify  SUCCESS;
  proxy_set_header X-Client-DN      $ssl_client_s_dn;
  proxy_set_header X-SSL-Subject    $ssl_client_s_dn;
  proxy_set_header X-SSL-Issuer     $ssl_client_i_dn;

  default_type application/x-raw;

  location /production/file_content/ {
    rewrite ^/production/file_content/modules/([^/]+)/(.*) /$1/files/$2;
    break;
    root /etc/puppet/modules/;
  }
  location / {
    proxy_pass http://puppet-production;
  }
}


# cat /etc/nginx/conf.d/puppet-production-upstream.conf 
upstream puppet-production {  
  server unix:/var/run/puppet/master.00.sock;
  server unix:/var/run/puppet/master.01.sock;
  server unix:/var/run/puppet/master.02.sock;
}


puppet:~# cat /etc/supervisor/conf.d/puppetmaster.conf 
# This file is autogenerated by Puppet. Manual changes will be overwritten!
[program:puppetmaster]
command=/usr/bin/thin start -e development --socket /var/run/puppet/master.%(process_num)02d.sock --user puppet --group puppet --chdir /etc/puppet -R /etc/puppet/config.ru
process_name=%(program_name)s_%(process_num)02d
numprocs=3
priority=999
autostart=true
autorestart=unexpected
startsecs=3
startretries=3
exitcodes=0,2
stopsignal=TERM
stopwaitsecs=10
redirect_stderr=false
stdout_logfile=/var/log/supervisor/puppetmaster/puppetmaster.out
stdout_logfile_maxbytes=250MB
stdout_logfile_backups=10
stderr_logfile=/var/log/supervisor/puppetmaster/puppetmaster.err
stderr_logfile_maxbytes=250MB
stderr_logfile_backups=10

puppet:~# cat /etc/puppet/puppet.conf 
[main]
ssldir=$vardir/ssl

[master]
certname=puppet

Applying the work-around here I'm able to get the git agent only to this point when attempting to introduce git to puppet master:

git:~# puppet agent --waitforcert 30 --test
err: Could not request certificate: Error 400 on SERVER: <html>
<head><title>400 No required SSL certificate was sent</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.1.8</center>
</body>
</html>

This resource suggests in its Simulate SSL Connection section to run, from my git box:

openssl s_client -host puppet -port 8140 -cert /var/lib/puppet/ssl/certs/git.troutwine.us.pem -key /var/lib/puppet/ssl/private_keys/git.troutwine.us.pem -CAfile /var/lib/puppet/ssl/certs/ca.pem

The problem with this being that I'm missing /var/lib/puppet/ssl/certs/git.troutwine.us.pem:

git:~# tree /var/lib/puppet/ssl/
/var/lib/puppet/ssl/
├── certificate_requests
├── certs
│   └── ca.pem
├── private
├── private_keys
│   └── git.troutwine.us.pem
└── public_keys
    └── git.troutwine.us.pem

Plain old webrick puppetmasterd works just fine--it's only the nginx/puppet combination that's failing me. Both machines are running ntpd and have an acceptable time spread. What am I doing wrong?

Use the one in `public_keys` instead - it's already been signed by the puppet CA, correct? — Shane Madden, Feb 02 '12 at 20:42
No, I'll make this more clear in the body of the question, but the first puppet agent command you see for `git` is the attempt to introduce the `git` box to puppet master. — troutwine, Feb 02 '12 at 20:58
They are. I've ntp running on both machines. The time difference is ~200ms. — troutwine, Feb 02 '12 at 22:10

score 2 · Accepted Answer · answered Feb 02 '12 at 23:14

newl in the #puppet channel suggested modifying ssl_verify_client to be 'optional', rather than 'on'. I've done this and everything's peachy now.

I had myself convinced that this was a Bad Thing to do, but after newl's suggestion I couldn't recall why. If anyone does believe this to be a less than ideal configuration setting, do let me know.

score 1 · Answer 2 · answered Feb 03 '12 at 07:47

If you run a puppet agent on the master, you should make sure that they do NOT share SSL directories. I have seen Weird Stuff result from that configuration.

Snippets from my /etc/puppet/puppet.conf:

[main]
# Where SSL certificates are kept for the puppet master and other
# subcommands.
# Note that this is a global setting because most of the subcommands
# other than 'agent' are only valid in puppetmaster context.
# The default value is '$confdir/ssl'.
vardir = /var/lib/puppetmaster
ssldir = $vardir/ssl

[agent]
# The var & SSL dir for the agent; listed explicitly because the master
# and other subcommands intended for the master should use
# the different SSL state.
# The default value is '$confdir/ssl'.
vardir = /var/lib/puppet
ssldir = $vardir/ssl

That didn't end up being the problem, but thank you. I'll modify my environment. — troutwine, Feb 03 '12 at 18:25

Puppet agent to puppet master (running on nginx/ruby1.9) SSL certificate issues

2 Answers2