1

This is a somewhat broader question about a topic I think every organization/business is running into once there's more than one DevOps person: How to keep and curate organizational operations knowledge so that it remains up-to-date, easily accessible, and is useful to newcomers?

The goal of this question is to collect a sample of solutions for people to choose from. My business/organization hasn't (yet) found a good way of dealing with this issue.

Definition: Operational Knowledge -- That what is needed to know to get things done, keep the infrastructure running, and up-to-date.

Some Examples: What AWS regions is our main RDS cluster in? Where and how do we store our secrets? What's the loadbalancing setup for our web page? What is our role/user structure in Postgres? How is the VPN setup and what's needed to add a new user (generate certs etc.)? What are the steps to deploy a new version of code XYZ to production?

How does your organization handle this? Do you run your own wiki (which one?), Github's wiki, Google docs, Evernote, markdown files in a git repo, Word docs (?!?) on a central server, or what else?

CDO DevOps
  • 151
  • 5
  • If you require "*deployment should be one-click*", then the answer can really only be "*all our ops knowledge can be derived from the salt/puppet/ansible git*". – anx Feb 05 '19 at 03:28
  • 2
    I've seen that done many times, and what it often amounts to is: "all of our documentation for our incredibly complex processes and systems is described via this complex processes and system, so we didn't write it down". Self-documenting systems rarely resemble documentation to those who don't already know about the thing which is documented. – Spooler Feb 05 '19 at 15:57
  • @anx - that only tells a future person "what", not "why." It doesn't explain what decisions were made, what trade-offs were considered, what bottlenecks or other problems were worked through. – mfinni Feb 05 '19 at 18:37
  • 1
    Onenote works miracles here. – Jim B Feb 05 '19 at 19:57

2 Answers2

3

Some organizations start with a wiki for a knowledge base. It's easy enough to keep track of for a small team, and requires little effort to build. Knowing some kind of markdown language is helpful, though there are web panels that reduce the publishing process to an authoring page.

There is a tendency to use Google docs, because it's shared, collaborative, and can be organized. This works well at first, but is prone to disorganization and the search function doesn't work well with a large and aging dataset (misses quite a bit). But if you can keep it organized, it's cheap and simple. This is effectively the same thing as the NAS "file cabinet" approach, and will store usually whatever you need it to.

A document management platform like Confluence works well for larger organizations, as it's designed to segregate responsibilities of maintaining documentation among teams, as well as provide clear versioning and sharing functionality. It's easy to keep organized, and integrates with various diagramming or other rich content systems. There is more of a learning curve than with a wiki or the classic "file cabinet" approach.

Spooler
  • 7,016
  • 16
  • 29
3

Once your team goes beyond 3 or 4 people, you need a librarian -- someone responsible for ensuring that documentation gets written, is consistent, and maintained. It's obviously not a full-time job, but it's vital. When no one is responsible for maintaining the document/wiki/repository/Sharepoint, documentation gets stale. And once it's stale, people stop using the document/repository, and stuff stays in people's heads.

Ron Trunk
  • 2,149
  • 1
  • 10
  • 19