5

In an extremely interesting presentation at Puppet Camp London, Tomas Doran suggested a pretty radical approach for keeping everything automated by managing tons of Docker containers with Puppet.

As a security-conscious person, I like the idea of Docker, as everything runs in its own LXC and I configure all services to run as non-root users.

As a systems-administration-conscious person, I like the idea of Puppet management, as I can keep all configuration in a Git repository and even maintain different environments, all in version control. The advantage is also that I have tear-down-able (is that a word?) environments that I can theoretically rebuild from scratch without too much manual intervention.

However, there are things I'd like not to keep in a Git repository, namely SSL certificates, database passwords, etc.

How do organizations managing massive amounts of machines (like CERN) use provisioning services like Puppet and Chef while still maintaining security? Certain things seem easy, like enforcing permissions on files, but other things seem difficult, like installing SSL keys or SSH host keys, which requires manual intervention.

AviD
  • 72,138
  • 22
  • 136
  • 218
Naftuli Kay
  • 6,715
  • 9
  • 47
  • 75

1 Answers1

6

I used to run a puppet installation, so I'll speak to what I did then.

  1. SSH Host Keys are ephemeral. So, when building a new system or rebuilding an existing one, we just let new keys be generated by SSHD when it first started. There were relatively few people SSHing into these machines, and when rebuilding the machines, we published the host key fingerprints on an internal wiki. So an attacker would have to compromise the wiki host and MITM SSH connections, and be in place to do so immediately after the machine was built.

  2. We always generated new SSL keys when reinstalling the machine. One of the human steps in rebuilding services that involved SSL was generating the key, the CSR, and getting it signed by the CA and then installing the cert. We did use a script that was prepopulated by a puppet template to have the CN & other fields properly set.

I doubt CERN has a lot of hosts with SSL certificates involved. Or maybe they do, but have an automated CA to sign certs at puppet's request.

I also don't consider storing certs & keys in a well-controlled git repository necessarily bad. We just took the point at which we were rebuilding a system as a great opportunity to rotate our keys.

David
  • 15,814
  • 3
  • 48
  • 73
  • It looks like you missed a great opportunity to use SSHFP DNS records to publish your public keys. – Bruno Rohée Jun 19 '14 at 12:57
  • SSHFP is only useful if you have DNSSEC. Without DNSSEC, you can't know if the SSHFP record is valid. (Though, admittedly, MITMing DNS is probably comparable to breaking our wiki...) – David Jun 19 '14 at 16:18
  • IIRC if you don't have DNSSEC (or to be pedantic, if the reply isn't authenticated) it still informs you with the message "Matching fingerprint found in DNS" in addition to presenting you the fingerprint and the yes/no question. So even without DNSSEC it brings some level of convenience. – Bruno Rohée Jun 19 '14 at 17:41
  • http://jpmens.net/2012/07/27/verifyhostkeydns-yessssss/ explains pretty well how it works in both cases, with example outputs – Bruno Rohée Jun 19 '14 at 17:43