30

In my organization, I work with a group of NOC staff, budding junior engineers and a handful of senior engineers; all with a focus on Linux. One interesting step in the way the company grows talent is that there's a path from the NOC to the senior engineering ranks. Viewing the talent pool as a relative newcomer, I see that there's a split in the skill sets that tends to grow over time...

  • There are engineers who know one or several particular technologies well and are constantly immersed... e.g. MySQL, firewalls, SAN storage, load balancers...
  • There are others who are generalists and can navigate multiple technologies.
  • All learn enough Linux (commands, processes) to do what they need and use on a daily basis.

A differentiating factor between some of the staff is how well they embrace scripting, automation and configuration management methodologies. For instance, we have two engineers who do the bulk of Amazon AWS CloudFormation work, and another who handles most of the Puppet infrastructure. Perhaps a quarter of the engineers are adept at BASH shell scripting.

Looking at this in the context of the incredibly high demand for DevOps skills in the job market, I'm curious how other organizations foster the development of these skills and grow their internal talent. Scripting doesn't seem like a particularly-teachable concept.

  • How does a sysadmin improve their shell scripting?
  • Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?
  • Are we simply to assume that some people will be left behind as these technologies evolve? Is that okay?
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • 14
    You practice. Try automating everything, build vms, etc.. – Doon Sep 11 '13 at 13:15
  • 2
    @Doon I've done this for 15 years, so I've had a lot of time to practice, break things and get to where I am. For junior engineers today, and with the level of complexity in some of the existing automated setups, there doesn't seem to be enough time or a safe place to allow that experimentation many environments. – ewwhite Sep 11 '13 at 13:20
  • Mentoring from the seniors, plus good documentation and other sustainable practices (not building technical debt) is a very good way to inculcate The Knowledge in your PFYs. – mfinni Sep 11 '13 at 13:58
  • actually I think the safe place today is in the vms, since you don't need all the physical hardware. Now time/etc. yes that is in short supply :) But given the availability of free / low cost hypervisors, and the malleability of *nix OSes You can build some pretty complex setups to learn. – Doon Sep 11 '13 at 14:37
  • 1
    Interesting challenge that applies to so many things in the IT world. No budget for training. No time or equipment for practice. VMs help a great deal but the gap remains. – Dave M Sep 12 '13 at 17:17
  • I wish I had this much opportunity to do scripting where I work. I have to make do with self-study. – bgvaughan Sep 17 '13 at 07:42
  • Take away SSH access. Seriously. If you have to manually SSH to the targets, your automation is insufficient. Make everything work out of a git (or equivalent) checkin and then have the build process do the damn thing. – dmourati Aug 19 '14 at 04:29
  • @dmourati My feelings on this change every few months. I don't think everything is or should be [web-scale and fully-automated](http://serverfault.com/questions/607985/what-types-of-systems-have-to-scale-up-rather-than-scale-out/607991#607991). I spend most of my time these days with snowflake systems... Yes, I've been able to Puppetize a LOT of the routine things, but I probably log onto 10-15 servers every day (out of ~300 or so running this application). – ewwhite Aug 19 '14 at 04:33
  • @ewwhite, mine too. The way forward for me is to remove the straight command line access and to do things "as code." It is a discipline. – dmourati Aug 19 '14 at 04:45

5 Answers5

21

• How does a sysadmin improve their shell scripting?

Practice, mixed with drive. It sounds trite, but you have to want to get better, in addition to practice. If you don't truly enjoy scripting, you can be forced to do it for years when you have to and never really get good at it. If you don't want to get better, you could sit next to the best scripter in the world every day at work and not pick up on a fraction of the skill that you could have.

I know those people that, despite working in IT, stubbornly refuse to learn any sort of scripting. There will soon be no place for those people in this industry. They're part of a dying generation.

(I'm not talking about old people, I mean that figuratively. :P)

• Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

Nope. Every thing that they do can be and eventually will be automated away.

I would contend that perhaps we never should have called them 'engineers' anyway. It's bad enough that the IT industry appropriated the word 'engineer' for ourselves, which in my opinion is kind of insulting to the actual engineers that spent years in higher education programs and getting legal certifications so that they could design bridges, skyscrapers, hadron colliders, etc... those are the real engineers.

But there is a similarity... If you want to call yourself an 'engineer' in the IT industry, then that at least means you create things. You are inventive and you connect the dots in new ways that no one ever thought of before. You build things that no one else knew how valuable it would be until you made it.

If you don't code or script, then there is no way you do much with computers besides just maintain them, and maybe install a software package or two. Maybe throw a new hard drive into the ol' MSA. And in that case, I'd call you an administrator, sure, but not necessarily an engineer. And I'd say much of your job is in jeopardy of being automated away.

• Are we simply to assume that some people will be left behind as these technologies evolve?

The market will adapt. It may be that some people will not be making 6-figure salaries when they don't actually deserve them, which happens quite a bit in this industry.

I find that creativity, and not just coding/scripting skill, is a key factor. It's that creativity that you need to say to yourself, "Oh, hey, I could automate this!" and then the skill only comes into play after that. If you find yourself scripting something only after your boss tells you to, then you might not have that drive or that creativity I was talking about... and those are two qualities that are very hard, maybe impossible, to teach.

Ryan Ries
  • 55,011
  • 9
  • 138
  • 197
  • Very good insight. I fear that the majority of people in IT are the types to be left-behind. I *am* seeing this now... But it also speaks to drive and motivation... – ewwhite Sep 16 '13 at 13:54
9

I have the benefit of understanding the size and complexity of your environment. Seeing as you work for a cloud/hosting provider, it's safe to assume that you have a large number of small-medium sized environments (10-100 servers). There are certainly daily tasks that are done by the jr. engineers and NOC staff that are repetitive (creating user accounts, configuring backup agents, etc). Similarly, there are probably some manual things that are done by the sr. engineers like installing ESXi on new hardware or configuring things like MPIO or installing VMware modules for specific sets of hardware. All of these things can and should be automated.

If your staff is capable of carrying out the bulk of their workload without automating, then you're overstaffed in my opinion. Any IT staff that can work a full day that consists of mostly manual processes has no motivation to automate. Why learn a new skill that isn't viewed as necessary and might even be scary? After all, necessity is the mother if innovation.

So, at some point in your organization, you will grow to a size where you'll flounder and fall apart, or you'll start automating almost everything and excel. Certainly, the senior engineers should be leading the charge here, and maybe even working with the junior engineers and NOC staff to automate some of their workload. This gives the jr. engineers the opportunity to have the framework of many scripts to work with, which they can tweak for each tenant and new hardware revision as necessary. This removes the daunting thought of "Oh my god, where do I even start?" from the equation and gives them a jumpstart to solving a real problem. Which brings me to my final point. Books and examples are well and good, but there's nothing that can replace the feeling of accomplishment of solving an actual problem that they face. Give them a goal, like all new servers for tenant x should have certain ESXi modules installed, and then work with them to accomplish it. Then adapt the script to work in a multitenant environment.

How does a sysadmin improve their shell scripting?

By needing to, as described above.

Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

Sure, there are plenty of organizations that cannot or will not shift to the DevOps methodology. They're seeming to be more and more boring options, but they're options nonetheless.

Are we simply to assume that some people will be left behind as these technologies evolve?

As with any new technology - yes.

tl;dr You'll never have anyone really invest in learning it until they see the value in it. If they can accomplish their daily tasks manually, then you're overstaffed and there's no incentive.

MDMarra
  • 100,183
  • 32
  • 195
  • 326
  • 3
    I read this : `you'll start automating almost everything *in* excel.` – mfinni Sep 11 '13 at 18:30
  • Yeah, 32-bit Excel VB macros are the things that clouds are built on. Didn't you know!? – MDMarra Sep 11 '13 at 18:33
  • 2
    I have a feeling that you may be correct... – mfinni Sep 11 '13 at 18:34
  • I can't conclusively say that everything should be automated, especially if the staff is at different skill levels. – ewwhite Sep 11 '13 at 18:58
  • If the staff are of various different skill levels, that's **exactly** when you want things automated. By automating, you ensure a level of consistency to the process that you don't get when you have people of different skill levels doing it manually. It's not all about time saving, it's also about consistency. Is a newbie to HP hardware going to know that there's a specific process to do to get the HP MIBs going on new servers? Or that specific HBAs need a specific MPIO configuration? Probably not, but it won't be an issue if there's a script that takes care of it. – MDMarra Sep 11 '13 at 19:00
  • But in a sense, that knowledge goes away when it's simply part of a deployment script. But that's my hardware-upbringing speaking. We have significant automation in the hardware build process, but maybe two people would ever be able to debug/modify it or do it by hand. – ewwhite Sep 11 '13 at 19:13
  • 2
    That knowledge *shouldn't* go away. Instead of documenting "Do these x steps" in your internal wiki (or whatever) you say "These x lines of code install $stuff" and you also comment your code heavily on things like this as well. Not scripting because of the loss of knowledge that might occur exposes a potential immaturity in your documentation process. It's not a reason to avoid automation. – MDMarra Sep 11 '13 at 19:15
  • 2
    @MDMarra What's a *wiki*? – ewwhite Sep 12 '13 at 14:36
7

How does a sysadmin improve their shell scripting?

How does one get better at anything? Read books, attend classes, and then apply the principles learned. (Or a combination of the methods.) This is over simplified intentionally since there isn't anything special about learning scripting over learning how to cook or how to repair a car.

Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

This is hard to answer within the scope of this site (where there is a requirement for clear/defined answers to questions asked.) We can predict that it will, but there are problems with the DevOps model. I feel that it's very hard for one person to be extremely proficient in both disciplines. The cost savings of a 2-for-1 employee is very attractive to businesses right now, but it's hard to say if this trend is here to stay. It certainly is for the short term.

Are we simply to assume that some people will be left behind as these technologies evolve?

At the current rate of how things are going, yes. Most of you are likely observing it in your own workplaces. You should definitely be keeping up with job listings and know what the market is currently demanding. (There's a lot of job listings for Hadoop in your area? Learn Hadoop.) If you don't keep up with the market, you are risking being left behind.

Aaron Copley
  • 12,345
  • 5
  • 46
  • 67
5

One generally doesn't send junior engineers into a complex production environment that is mission critical. You have senior engineers for that. Junior ranks should be allowed to work in dev / test sandboxes.

If you need an engineer for Technology X and want to fill the role internally, find someone willing to learn it, find structured training and combine the two.

Figure out what skills you need in a department. Find someone willing to learn them. Teach / Hand out money for training.

Falcon Momot
  • 24,975
  • 13
  • 61
  • 92
Daniel Widrick
  • 3,418
  • 2
  • 12
  • 26
  • Building skills in technology X in many cases is clearcut. There's a certification and training path for Cisco, VMware, EMC, Red Hat, etc. It's the scripting mindset and moderate development skills that seem to be less *trainable*. – ewwhite Sep 11 '13 at 13:26
  • 5
    Scripting is programming (I'm hoping the stack overflow folks don't come by to start a war). There is a way of thinking and a way of seeing and approaching problems that not everyone will be good at. 'Teaching the scripting mindset' is what people hopefully get from practice. ...And 'moderate development skills' is just generic enough to not mean anything. ---- As for teaching programming, Look at area universities that offer intro programming classes. An early Computer Science class can go a long way in teaching 'mindset'. – Daniel Widrick Sep 11 '13 at 13:32
  • 3
    Hell, UMass Lowell has "Bash Scripting" and "Unix/Linux Administration" courses. I took 'em both. Taught by old-school greybeards that no doubt wanted to show off their emacs profiles. (Online classes, so I'm simply assuming the greybeardedness.) – mfinni Sep 11 '13 at 13:54
  • @mfinni I had no clue! :) – ewwhite Sep 11 '13 at 13:55
  • I'm working on the UML BS in Information Technology program right now. All of it online, since I transferred in an AS in CompSci, with a freshman year's worth of lab science, Calc, etc – mfinni Sep 11 '13 at 13:57
1

Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

"devops" is just a new word for something sysadmins have been doing for decades.

Are we simply to assume that some people will be left behind as these technologies evolve?

Quite the opposite. As time goes on, there's only more and more of a need for technical people. Anyone with any sort of engineering knowledge and technical skills will have a place to work.

Michael Martinez
  • 2,543
  • 3
  • 20
  • 31