18

Nowadays, agile systems adminitration and devops are some of the most trending topics regarding systems administration and operations. Both these concepts are mainly focused on bridging the gap between operations/sysadmins and the projects (developers, business, etc). Even if you never heard of the devops concept, I'm sure that this topic is your concern too.

So, what tools and techniques do you use to accomplish devops in you companies? I'm particularly interested in topics like change management, continous integration and automation, but not only in these topics. Please share your thoughts. I'm looking forward to read your answers/opinions :)

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Marco Ramos
  • 3,100
  • 22
  • 25
  • Part of the problem with "bridging" Development and Operations (System Administration) is the different top priority. System Administrator's #1 Priority is *Keep* things working, though a variety of common repetitive tasks. Development's #1 Priority is to *create* new functionality. These tasks can overlap greatly, but there *will comes times* when they're in contention. It is at those times of contention that your DevOp will have to choose to either be an Operator or a Developer. Certain settings can tolerate the lapse, but most will not enjoy the financial reprocussions. – Chris S Aug 14 '12 at 04:24
  • 2
    Also, I've heard recently someone discussing Administrators who also know how to program. Abilities do not determine priorities or primary responsibilities. Modern Administrators must be lazy; to that end they must be efficient in everything they do. Scripting, creating maintenance utilities, and understanding code is simply a basic skill set now. SAs who do not proses these skills are becoming relegated to small and lethargic business models (eg, manufacturing) where such inefficiency is tolerated. The changing knowledge base does not warrant the coopting of odious terminology. – Chris S Aug 14 '12 at 04:31

4 Answers4

30

  • svn/git - revision control, obviously.

  • trac/redmine/jira - ticketing.

  • cobbler - for base operating system server provisioning. Cobbler's a redhat-family focused product but I'm sure there's something similar for debian/ubuntu. Similarly most of the "cloud control panel" companies like RightScale will provide this for you. The watchword here is "JEOS" or "just enough operating system". My route is to use the "%packages --nobase" line in my kickstarts and to then build up my specific stack via...

  • puppet/chef - for configuration management and consistency enforcement. There are other options here too, it matters more that you use one than which. One trick I've found particularly important is to store the configs in the same version control system as the developers use. This helps pull together the two teams workflow and make it visible to each other.

  • func (or capistrano or cluster-ssh) - for running the deploy script across the cluster. The trick here is to make it something that the senior developers can run themselves to both push new things live and push the inevitable fixes.
    This is really the core of devops, empowering the developers to both break and fix the environment. A lot of sysadmins are too power hungry to let go like this, or their management still works on the mistaken notion that sysadmins should be policing developers (as if we can even read half of what they're doing).

  • cacti/ganglia/collectd/munin - graphs are soooooo key. Its the business value of metrics with the human value of simple visuals. Correlating the timestamp of code pushes with the timestamp of changes in the graphs is immensely valuable in troubleshooting performance regression and seeing real facts about performance decisions. There is a key point here in that the graphs need to be easy to see and use by the developers and their management needs to expect it of them.

  • nagios/zabbix/smokeping/etc - monitoring of server stuff and "base page" type performance metrics. Again the graphs are key. These are more for the ops side of the team.

  • gomez/keynote/browsermob - external monitoring of full browser performance, taking into account third party services, CDNs, and render time issues. These are more for the dev side of the team.

Thats a mix of tools and techniques, focus on the techniques. Specifically the change in mindset of the "sysadmin" side of devops from "admin" to "operations". Its about enabling the developers. Enabling them to do things, enabling them to fix things, enabling them to see real facts/metrics/graphs about what they did. Conversely the devs need to embrace that they've been enabled and actually do the work of watching performance trends, debugging problems, and thinking about not just features but how to roll them out and how they will affect the health of the entire system/environment.

Henk Langeveld
  • 1,294
  • 10
  • 25
cagenut
  • 4,808
  • 2
  • 23
  • 27
  • 2
    +1 "core of devops, empowering the developers to both break and fix the environment" – Ryan Gibbons Mar 26 '12 at 18:15
  • Which is in direct contention with providing reliable services and why devops can sometimes be developers playing operations without understanding. The skill is in finding the right balance between allowing free development and ringfencing change to hide breaks from the user behind staging, redundancy, etc. – JamesRyan Aug 16 '13 at 10:28
4

We are working on doing this very thing at National Instruments. You can read more about what we are doing at http://dev2ops.org/blog/2010/4/27/qa-ernest-mueller-on-bringing-agile-to-operations.html

The mix of tools that cagenut mentions here are basically going in the direction that we are moving in here.

wickett
  • 148
  • 1
  • 8
2

The best approach is to understand the environment in which you work. Start by talking to the developers and managers. Try to get them on board and bounce ideas off of them. Most likely they will have a good idea of how things are run and if your ideas for introducing devops will cause any problems.

From there, start looking at applications and introduce them one at a time to solve problems.

Matt Delves
  • 473
  • 2
  • 6
  • 12
0

While the tools and techniques are important, the critical path is in collaboration across the whole organisation. These days IT Operations is Business Operations. Etsy shows changes in revenue on their dashboards, visible for everyone.

Henk Langeveld
  • 1,294
  • 10
  • 25