How can the little guys effectively learn and use Puppet?

Question

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often. Their biggest objections are:

• "We're not programmers, we're sysadmins";
• Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;
• Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;
• One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;
• "Let's just run it and see how it works"
• Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

Answer 1

I started using Puppet ahead of deploying a new infrastructure and simply bought a (well-regarded) book on the topic. I don't think most people actually obtain professional Puppet training. I worked on examples until I could mold the process to my environment. This was December 2011, so within a few weeks, I was able to understand the basics and get a production framework in place. I wasn't new to configuration management, having a CFEngine background, but many of your sysadmins' concerns resonate. I made mistakes and had to refactor several times, but I did get things working satisfactorily.

A few notes on your points...

• The traditional systems administration role is changing. Adapt or be left behind. I've been a successful systems engineer, but am having to retool as well (learning Python, for example). The focus on individual servers is diminished as hardware abstraction through virtualization and public and private cloud services gain traction. This means automation of systems tasks and the use of configuration management to wrest control of a larger number of servers. Add DevOps concepts to the mix, and you'll see that the customer/end-user expectations and requirements are changing.

• Puppet modules available online differ in style and structure and yes, I saw lots of overlap, redundancy and duplicated efforts. One developer I worked with said, "you could have developed your own tools in the time you spent looking online for something that works!" That gave me pause as I realized that Puppet seems to appeal more to developer types than admins looking for a best-practices or the right way approach.

• Document heavily in order to get a feel for how things are connected. Given the shaky definitions and lack of a standard way of doing things, your configuration management structure really is unique to your environment. That transparency is going to have to be developed within.

• I'd argue that it's reasonably easy to duplicate a module to accommodate a new daemon or add a service to an existing manifest, depending on how you've organized your servers and roles.

• I spent a lot of time testing on a single target before pushing changes to larger groups of servers. Running puppetd by hand on a representative server allowed me to debug changes and assess their impact. Maybe that's a bit conservative, but it was necessary.

• I'm not sure how much I'd depend on community modules. I did have to start using Augeas for some work, and lamented the fact that it was a functionality I took for granted in CFEngine.

In all, I feel like there isn't a well-defined standard when it comes to Puppet. I had trouble figuring out how to organize directory structure on my Puppetmaster, understanding how to manage certificate signing, getting proper reverse DNS in place everywhere, getting Puppet to scale appropriately for the environment and understanding when to leverage community modules versus building my own. It's a shift in thinking and I see how that would make a sysadmin panic. However, this was also solution built from scratch, so I had the luxury of evaluating tools. The decision to go this way was based on mindshare and the momentum behind Puppet. It was worth the effort to learn something new.

Remember, this site is a good resource, too.

Answer 2

On a previous job, I was assigned the task of doing a pilot implementation of Puppet. Now, I have programming background, though not Ruby, so I don't have as much problem as others do.

However, it's interesting to note that programmers without experience with non-traditional paradigms have problem with Puppet too, because Puppet is declarative, not imperative. In this sense, Puppet works pretty much like any configuration file: you say how things should be, and Puppet takes care of the rest.

After the pilot I had the opportunity to train a a dozen other admins with Puppet, plus giving presentations about it in two events. My take from that experience is that some admins took to it, and some didn't. These were all traditional admins, without programming skills, and varying levels of expertise.

One particular thing I noticed is that Puppet requires constant practice. People who were trained, wrote modules, and then spent a whole month or two doing something else came back to Puppet with little useful skill. People who kept doing small things in it every week never lost the ability.

Based on these two observations, I recommend you make sure everyone keeps adding some Puppet class, definition or module every week (preferably at least two or three times). Those who still can't get used to it might really lack the skill to do it.

Then again, if Puppet was imposed on them from higher up, they might simply be reacting to what they perceive as a management intruding in how they do their work -- which would be true enough, in fact. It might be the case that letting them choose which configuration management system to use would improve things. Here are some alternatives:

• ANSIBLE: This is new, but it's based on shell commands and ssh, which might entice it to traditional sysadmins.
• Chef: Maybe their problem is declarative style, in which case Chef would be better if they have Ruby experience.
• SaltStack: Python-based, and open-source
• CFEngine: old, fast, traditional -- it might win them over on the those grounds.

Answer 3

I've been using Puppet a bit over two years in small shops where I was the only sysadmin. The biggest hurdle I've had is learning how to develop software properly. There wasn't a week that had gone by where I didn't screw up something that I've told developers not to do a dozen times. I checked too much code in, I didn't break up checkins, I didn't tag, I didn't branch, didn't run the syntax checker, didn't use a standard, etc. If you're just starting out I'd recommend some of the following.

1. Realize you're developing software which you either don't know how to do or are doing poorly. This is expected because it's new.
2. Infrastructure as code is the reality and once you get over the hump it's quite powerful. I'd invite some developers over, show them your current development process (or lack thereof), don't take offense when they raise eyebrows, and take their suggests seriously. I'd recommend using whatever system and process your developers use unless it's completely inappropriate.
3. Puppet third party modules suck 90% of the time. I'd read them. I'd steal ideas from them. I would not pull them into my system without major edits. However I would pull in the puppet stdlib which adds some nice functionality.
4. augeas and hiera. Learn those two. The first allows complex editing of existing files in place. The second is an external data store.
5. Separate code from data. This is one of the harder concepts to learn. Hardcoding values like Monitoring Hosts into your module code is bad. Putting them in a data store (db, yaml (Hiera uses this be default), csv, whatever) that your modules can consume is good. An example is a webapp that uses Mysql. What this allows is the ability to push code and data separately. This makes your development process simpler.
6. puppet parser validate and puppet-lint as part of you pre or post code checkin process. Also rspec tests might be a good idea once you're up to speed.
7. write a style guide/code standard and use it. "where is the code that installs Apache" is a common problem. If your modules are mostly the same it should be easy.

In summary I've hit all of these problems and so have most of my sysadmin friends. It'll take some time to get good at using a config management system. Once you do you'll wonder how you ever lived without one. "Log into a server and make changes manually? Ick."

Answer 4

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Sounds like a darn good idea to start early - Puppet is more than just config management, it's a form of documentation.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often.

They need an attitude adjustment.

"We're not programmers, we're sysadmins";

Again, attitude. You -can- make a conf file for a server right? You can ease into the templating/'programmer' stuff as your needs and complexity evolves.

Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;

Tough one to answer - I always prefer the puppetlabs modules over most - and even at that, I don't use that many. Judgement call for sure. In my opinion, some modules are 'too frilly'.

Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;

This doesn't sound like a puppet problem, but more so an organizational or documentation issue?

One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;

That daemon could be a class if it's simple enough to manage. I'm not sure what you mean by conventions, puppet enforces conventions on you pretty well doesn't it? Or are we talking along the lines of code formatting?

"Let's just run it and see how it works"

Not that bad of an idea if you take it slow and safe. I'd still start with a VM to get the gist of things.

Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

postfix, exim, sendmail, mysql, postgresql, iftop, iptraf, perl, perl modules.. Pick what you want and use it? I guess this sound more like an attitude thing again...

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

I haven't attended any courses - while I am a programmer more than a sysadmin, I found it didn't need much programming skill to get anything accomplished.

The Puppet documentation, when followed is quite thorough. Just pay attention to the built-in types and spend some time looking at how other modules are put together. I wouldn't say it's -super- easy, but it's not -hard- either. It is a bit time consuming to get your infrastructure ready for puppet, but the time invested is assured to be well spent when you expand.

Answer 5

KISS (Keep it simple stupid) - Don't use new technologies just because they are there rather because you have a requirement for them, use the bare minimum that your deployment requires, update as required don't try to keep up with the bleeding edge. If you start with a basic setup and build on that it is easier to pickup as you go, and they shouldn't need a course (are these even available?).

The other area you can look at are your sysadmins. If they can't program as well, then are they advanced enough for a large deployment, where most of the work needs to be scripted whatever tools you use?

Answer 6

I work for a non profit as well and was responsible for initially bringing Linux boxes in house and shortly thereafter Puppet for managing them. There are a couple specific things we've done which have really helped get things rolling.

First and foremost I've tried to stay away from the third party modules. The inbuilt tools handle 90% of our management. The biggest third party utility I use is the firewall module. Any custom facts, etc are developed with the whole team involved. We developed a template module and keep the file management, package, services, etc all standardized off this template.

Second, after standardizing on using the inbuilt modules we started using Git and Atlassian's Crucible - free for non profits, by the way - for performing reviews of all configuration changes. This provides the transparency desired.

Third, I automated the setup for Puppet so that new hosts can be added automatically with a default set of options. There are several ways of addressing this. Since I already had a complete Kickstart environment I opted to add a script there.

Answer 7

"We're not programmers, we're sysadmins"

My how times have changed, for the worse: a greybeard like me was expected to be a better programmer than professional programmers, or else would have never been able to pass for a system administrator.

Now, we have "system administrators", who are basically Windows desktop users who have at some point converted to Linux and cannot program, and do not find anything whatsoever wrong with that.

The elephant in the room is why management tolerates such a destructive attitude. Destructive to whom or what? To the business and to the infrastructure.

Back to Puppet[, CFEngine, Chef] subject: as soon as one sets a solution like that in, one loses. Everyone loses. Why? Because whoever comes up with the idea is not capable of designing encapsulated configuration management in the form of nice, clean, Kickstart[, JumpStart, Automated Installer, AutoYaST, Ignite-UX, NIM] operating system packages. When you have to use an automated hacking tool like Puppet (or Chef, or CFEngine), it means you lack the wherewithal to design and to implement a process which would, by that same design, enforce completely pristine and lights out managed systems, fully automated and completely non-interactive.

Another important point is, if you have to have Puppet or some such solution to correct someone hacking system or application configuration by hand, that also goes back to not having the experience to design a process, and in that process a framework where configuration is packaged into discrete components. In effect, whoever implements Puppet and the like, has no concept of component owners, releases, configuration management, Capability Maturity Model. This is rapidly developing into a very serious problem in the industry.

Working with Puppet also helped me learn Ruby, which has come to replace Bash as my default system tools language."

Why is Ruby needed, when a comprehensive, end-to-end configuration management can be encapsulated in preinstall, postinstall, preremove and postremove sections of operating system packages, just by using Bourne shell programs, AWK, and sed? That someone would go to the length of learning an esoteric language of Ruby, and a dialect thereof in the context of Puppet, is completely unnecessary. The problem of configuration management is easily solvable (and to wit, has been solved) with shell programs and AWK, and a little sed(1) here and there as a glue.

It's a cool feeling to see your Puppet manifest configure an entire machine or a new service from scratch.

It is an even cooler thing seeing it done by Kickstart, AutoYaST, or JumpStart, without a single line of code, and being able to query the operating system by using built in tools, without needing any esoteric or extra software, no client-server architecture required (SSH is more than fine, way more than fine), and seeing your operating system being aware of each and every change made to it.

5.Separate code from data. This is one of the harder concepts to learn. Hardcoding values like Monitoring Hosts into your module code is bad. Putting them in a data store (db, yaml (Hiera uses this be default), csv, whatever) that your modules can consume is good. An example is a webapp that uses Mysql. What this allows is the ability to push code and data separately. This makes your development process simpler.

...Or you could just template your configuration files with shell variables, even backquotes (for example ls -1 ...) and write a shell script which uses AWK to call eval(1) and expand all the variables in the template, thereby leveraging the exact same powerful parser which shells have built-in. Why make it complex, when it can be really, really simple? Where will you store the configuration values? Why, anywhere you please, such as for example pkginfo(4) files, or a database like Oracle, or pretty much anywhere. No need for ultracomplex solutions. The library I mention above could simply be sourced from the preinstall or postinstall sections in the operating system packages, thereby removing duplication and leveraging a central piece of code...

But above all, I find that the above quote is an example of the next generation of system administrators needing tutoring not by system administrators, but by system engineers. Find yourself a greybeard and sign on as an apprentice.