24

I just came across Something is burning in the server room; how can I quickly identify what it is?. In the comments I found the following quote:

you don't let a developer anywhere near your root passwords

As a developer but also someone very interested in sysadmin stuff, I'm wondering if there's anything to this phrase more than just the standard for everyone - that is, don't give away passwords? The commentor states this as though it's fairly common knowledge in the sysadmin community. Is it?

Community
  • 1
strugee
  • 901
  • 10
  • 25
  • 12
    I think its more of a need to know type of thing. Developers are known for making system changes to get things to work without regard to what else their actions may effect. I personally think it boils down to individual circumstance. – David Houde Apr 05 '13 at 08:47
  • 1
    Hmm, that makes sense. Not a complete answer but I'd still upvote your comment if I could (no rep or mobile, not sure which). – strugee Apr 05 '13 at 08:50
  • 2
    *shrug* We don't let sysadmins have root passwords on a regular basis either. They can *get* access, but if they do it gets recorded, and they shouldn't regularly need to do that. – kojiro Apr 05 '13 at 11:40

10 Answers10

29

Since the operational responsibility of a production system usually pertains to a Sysadmin, only a Sysadmin should have full administrative access to the system - it's as simple as that.

Now, you might say "Well, sometimes we make code changes that require system reconfiguration, shouldn't we have access to reconfigure the systems accordingly?"

NO

If your code changes require system reconfiguration, you should be able to explain to a sysadmin what requirements you have. That way, the sysadmin can review your change requirements and halt the change if it might have operational implications that you - as a developer - might not have foreseen.

In response to @NathanLong, I realize that in smaller businesses this is sometimes the case. In these situations, where admins and devs are the same users, an alternative is to divide system ownership between multiple persons, and make sure that no one - and I mean NO ONE - pushes their own configuration changes. Have another dev review your changes, let a third one carry out the reconfiguration.

Whenever doubt occurs, start over with change planning and reviewing - anything else would constitue horrible change management

Mathias R. Jessen
  • 24,907
  • 4
  • 62
  • 95
  • 9
    +1; we have a client who specifically asked that the devs had admin rights on the development system that hosts their git repositories. Despite raising objections, they still wanted us to go ahead. As the client is paying us to do as they ask, the devs got admin rights. Some time later (and fortunately before it was too late), we spotted that the devs had made some changes that completely broke the backup process on the server. That could easily have been several man years worth of development lost had it have gone unnoticed for any longer. – Bryan Apr 05 '13 at 10:45
  • As a developer, I appreciate the collaborative relationship you imply. Unfortunately, often the developer and sysadmin are the same person, so the idea of being "cut off" may irritate some developers. I'd say that if you have dedicated sysadmins, giving up root access in exchange for not having to do both things is a nice deal. – Nathan Long Apr 05 '13 at 12:26
16

I am not saying sysadmins are perfect, but there are a huge number of examples of developers that shouldn't even be developers. Let alone being permitted to having root access to systems with mission critical data.

But in any case it is mostly about taking responsibility for changes made. For many sysadmins it seems that developers will go do things on servers, and then they do not take responsibility for what they have done. Developers are often only focused on getting the single program/task they are working on going, and don't take the time to think about the big picture or long term state of the system they are touching. They haven't learned habits about how to make changes safely.

These aren't true for all developers, there are certainly some great developers that can maintain systems just fine. It just often seems like these are true 95% of the time.

Habits like:

  • Don't make changes to a system without testing on dev systems first
  • Don't make changes that you don't understand (blindly Googling, and doing things you don't understand)
  • Don't make changes without verifying that you know that the backups systems are working, and how to roll back anything you do.
  • Don't make changes that will make the system more difficult to maintain in the long term. (ie, don't add any technical debt)
  • Don't compromise the security of the system
  • Don't change the system in a way that will increase the potential regulatory risk (see PCI-DSS, HIPPA, FERPA, etc)
GAThrawn
  • 2,424
  • 3
  • 20
  • 38
Zoredache
  • 128,755
  • 40
  • 271
  • 413
12

I'll try to explain with a metaphor (I'm a developer, occasionally doing sysadmin stuff btw.)

Painter and Interior decorator

Let's say the developer is a painter, and creates a lot of fantastic paintings. Ok, maybe they aren't fantastic but he made what his employer asked him to make. He's good at that so all the paintings turn out fine.

Now the paintings need to be placed inside the respective buildings. The painter know's everything about paint but has no clue as how to properly mount something on a wall. If the painter decides to try it anyway (because he can), the interior decorator (sysadmin) might come in a few weeks later and respond like this: "What the f. happened here..?" "Everything's crooked, this.. this had to go on the ceiling! How the hell is this attached to the wa.. IS THAT GLUE!?"

Needless to say, the interior decorator could have done a better job. He knew where to put what, and how to mount it. He was consistent in using the same screws and occasional wires. He had done it a thousand times before and could, pretty much, do it with his eyes closed.

Long story short: The sysadmin knows how to handle his rig better than the developer (at least he should). He will keep his system clean and organized and know how the entire thing works. Also, a lot of developers are used to doing sysadmin stuff (at home for example), and they WILL try to do it themselves if they can, because it saves them time.

Of course there are situation's where one person does both, but they will most likely be in relatively simple environments or places where the manager failed to see that he actually needs two people.

Deruijter
  • 221
  • 1
  • 4
8

I advise turning it around around and putting yourself in the position as the other party.

So, in that vein:

Why shouldn't sysadmins have full access to modify the source code?

The long and short of it, to me, is that developers shouldn't have root access because they don't necessarily know what they're doing from the perspective of systems administration, and don't (or shouldn't) need it, so granting root is essentially increasing the risk of causing problems, for no benefit. Administering the systems is someone else's job.

Just as sysadmins shouldn't be making code changes, for the same reasons.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
4

This advice feels a little dated; It comes from an era where naughty developers who had a root password would go in and make changes on the live system.

These days no one should be making changes to a live system, neither developer nor sysadmin. Making changes on a server is the job of a system such as Chef or Puppet. Our industry moved beyond the concept of the singular server that can be hand configured years ago. Now we either need the ability to have many identically configured servers deployed at once, or at least the ability to reliably rebuild any singular server at any point. This rules out changes being made to a server by someone with a root password no matter what hat they have on.

So, back to the question: Should developers have the root password? Yes! Should sysadmins have the root password? Yes! Why? Because complex systems cannot be effectively diagnosed without smart generalists who understand them end to end - or at least a team comprised of developers as well as sysadmins. And part of the diagnosing can't be done without logging into the server and having the rights to inspect what's going on (for example, a developer might need to run strace or dtrace to understand what the webserver's actually taking all the time on.) Just remember - look, don't touch!

Mark Fowler
  • 141
  • 2
3

I have worked with a good number of developers who (rightly) have no interest in being sysadmins - they just want to Make Their Software Work. They'll take the fastest route to that end, which can often entail e.g. fixing permissions errors with chmod 777 -R .

This is, needless to say, A Bad Thing.

nickgrim
  • 4,336
  • 1
  • 17
  • 27
1

I can't speak for the sysadmin community but I can from experiences.

While they seem know their stuff they don't seem to have the attention to detail needed and random things get installed/changed, for example the other month I found out that one of our production web servers had KVM/QEMU installed!
The other thing that annoys me is they tend to mix server roles, so a Hypervisor host would suddenly also become a monitoring server and web server. There also seems to be a lack of consistency, for example 3 servers preforming the same role which should be setup the same way would be setup in 3 completely different ways.

Of course the commenter might have just meant they should be using SSH keys instead of passwords :)

Epaphus
  • 1,011
  • 6
  • 8
  • This all essentially boils down to the two separate roles of Dev and Ops. Developers primary goal is making something that works. Operators primary goal is keeping things working. Some people can manage to balance these, but not most. – Chris S Apr 05 '13 at 13:08
0

Typically it is for separation of concerns. Imposing a limit such as this is required in some environments for security purposes. It can provide business value by surfacing information about the applications that would not otherwise be available.

Greg Askew
  • 34,339
  • 3
  • 52
  • 81
0

Segregation of duties and change control.

Segregation of duties means that you never have a single person responsible for a single task. The best way to implement it is to make sure no single person CAN perform a single task alone.

As for change control, to properly implement it you have to make sure all changes to production systems are fully documented. The best way to achieve it is to have the dev perform a handout to the system administrators, giving them the code and the documentation and letting them perform the deployment.

Stephane
  • 6,382
  • 3
  • 25
  • 47
  • Segregation of duties is not exactly as you described. You _want_ someone to be able to perform a _single_ task by themselves or else nothing would ever get done. What you don't want is someone who can perform _multiple_, related tasks that could lead to escalation of privileges or breaking a critical application like a developer who writes code and is responsible for code review as well. – August Apr 05 '13 at 12:00
  • I more or less stand by my definition (And wikipedia as well as most of the documentation I read agrees). You're just using a more limited definition of the word "task" whereby I'm voluntarily leaving that definition open. – Stephane Apr 05 '13 at 12:25
0

This is an over simplification of a common problem in the IT industry. There is simply too much for any single person to know everything, so people specialize their knowledge. Ideally only those with the specialized knowledge to do a task should do so.

In the real world, we are often asked to "wear multiple hats" by management structures that still don't know all the complexities of how IT works and are dealing with the complexities of the company operation and finances they do know (and often the IT staff doesn't).

This often results attitudes like those asked about by the OP when two of these IT fields overlap. Like that attitude, as a network person, I know how easy it is to have similar thoughts about sysadmins, as often sysadmins are asked to manage the network as well.

In a relatively small number of cases they do a great job and in another relatively small number of cases they do a horrendous job. Most of the time, they fall into doing a decent job somewhere between the two. However, it is not the great jobs that are remembered, it is the horrendous jobs and to a smaller degree the decent jobs that are remembered.

When a network person is brought into the mix (either a consultant for a specific reason/project or because the IT staff has expanded and now includes one), they often find ignored recommended best practices, misconfigurations, security holes, etc. They spend a great deal of their time trying to figure out how the blank things got the way they did, troubleshooting problems that should not be occurring, and fixing misconfigurations and problems.

But this goes both ways and includes a number of overlapping disciplines. Specific to my example, I know as a network person that I cannot set up a server as quickly as a knowledgable sysadmin and/or that there will probably be a number of things I would overlook or do incorrectly (or at least not in the best way). Give me enough time and I can do at least a pretty decent job, but often the job/work does not give that time. If I have the choice, easily my preference is to hand it off to a good sysadmin.

YLearn
  • 1,237
  • 7
  • 17