26

When it comes to something like a production webserver, what is best practice for responsibilities for the sysadmin & developer? Specifically, I am thinking about updating/installing software. (In my understanding, the dev should not have root access on the production server.)

So a production webserver is running Wordpress, and it needs to be updated to the latest version. Who is responsible for keeping it updated?

What if the devs have custom hacked plugins, or custom core files on the app (in this example, WP) ?

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Josh Brower
  • 1,659
  • 3
  • 18
  • 29
  • 2
    Where is the dev's docs on the plugins and where are the backups of the core files and when was the last time the backups and documentation was tested in your dev environment? – ForgeMan Sep 07 '09 at 05:40
  • What about role-based privileges? give them secure, audited access to what they need to do without screw your server. – allruiz Nov 05 '09 at 17:35

8 Answers8

21

I've found that in most cases, if YOU are the one responsible for the physical server its best to NOT give the devs root access.

This is a bit of a "holy war" debate as I'm sure you will find developers who disagree. I've personally been on both sides of that debate.

My MAIN reasoning for not giving the devs (even 100% trusted devs) root access is because more often than not there's some package they need in order to make XYZ work correctly. They go ahead and install it... or reconfigure something that is already in place so it works... or... well... you get the idea.

Months go by... the server needs to be reinstalled or recreated... and suddenly nobody knows why "It works on the old server but not the new one."

The answer of course is that the documentation you're looking at doesn't include all those little packages and tweaks that the developers did to make the system work the first time around.

It can be a pain in the a$$ for both sides... but if the sysadmin is responsible for the server, packages, and documentation... and the developer is responsible for the development and software... I think you'll find it was worth it in the end.

If the developer needs a custom plugin, module, configuration, tweak... no problem... do it for them... but DOCUMENT IT so you can reproduce it next time.

KPWINC
  • 11,274
  • 3
  • 36
  • 44
  • So what about a situation like I described, where root access is not needed to update an application (like Wordpress). Who should be the one responsible to keep it updated? – Josh Brower Sep 06 '09 at 21:31
  • In that particular instance its basically politics (ie. management decision). In my organization it would be the sysadmin who is responsible for keeping the servers patched and up to date, and in this case keeping Wordpress updated. The sysadmin's job is to keep the server secure. Sounds to me like (especially these days) keeping Wordpress up to date is part of that. – KPWINC Sep 06 '09 at 21:39
  • if you had a managed server at a hosting company they would be unlikely to cover Wordpress. The rule of thumb would be whoever installed the software is responsible for it. Aysadmins would install OS, webserver etc. Developers would install Wordpress. – ollybee Apr 07 '11 at 17:32
16

Golden Rule: Don't let non-admins touch anything you don't want broken and for which you will be held responsible.

Devs should have access to a test environment. Once their work is ready to be put onto the production machine it should be handed over to the sys admin. If the devs have done their job and properly documented the procedure all will go well. If not, they need their backsides kicked for not adequately testing.

John Gardeniers
  • 27,262
  • 12
  • 53
  • 108
  • 5
    Amen! A magical way to get a developer to go away... Ask "Well does it work on the Dev environment?" or "Where is your build sheet?" – ForgeMan Sep 07 '09 at 05:39
12

I've been in this battle as well. My answer is that whomever is responsible for the uptime of the server is the one who should be responsible for all updates, changes, etc., etc. Nobodoy else should have the ability to perform these types of functions on the server. If it's your job to make sure the server is up and running and if the boss holds you accountable and responsible for the server then it's your responsibility to maintain and secure it.

Most developers are going to tell you that they need admin level access to the server and most of them are going to disagree with me, but I'm the one who has to reboot it at 2AM when it hangs up, has to rebuild it after a failed update, the downtime is charged against my department, etc., etc. I have to answer to the CIO for anything that impacts our SLA, so therefore I'm the only one who gets admin level access to the server and I'm responsible for all components, updates, changes, etc.

joeqwerty
  • 108,377
  • 6
  • 80
  • 171
  • Here! here! :-D 2 - 3 a.m. is not a fun time to be dealing with some dorked up change. (Been there seen it... visudo solved it). – ForgeMan Sep 07 '09 at 05:37
  • 1
    I really like the idea that "whomever is responsible for the uptime of the server is the one who should be responsible for all updates, changes"--It makes alot of sense. – Josh Brower Sep 08 '09 at 12:28
1

Developers should not have root on production; everyone except the developers agrees on this. But the developers can sort of have their cake and eat it too. I am somewhat surprised that nobody explicitly mentioned this:

One of my very long time small business clients has a web site with a Drupal installation, several WordPress sites, an SMF forum, and a few other random small web apps. I am the contract sysadmin (and for historical reasons also update/hack WordPress and SMF when needed) and my client has his own contract Drupal developers. The environment is several VMware virtual machines on a public cloud provider.

The developers really want to have root access and sort of need it. It's their responsibility to write the nginx rewrite rules to make all their custom Drupal stuff work, for instance. But no way in hell am I giving them root access on the production server, and my client agrees with me on that.

So we compromised: They get root access on the test web server (which is generally identical to production except for its IP address and is on the same cloud). Which, like production, has etckeeper so I can see whatever changes they needed to make and any packages they installed. I can then either pull the changes into production or tell them what's wrong with whatever they want to do. And if they really screwed up (they haven't yet, thank gawd) I can easily revert their changes.

They have no access at all to the production database server; they don't even have user logins. Only my client and I do.

(The web app itself, they deploy directly with git, and if they break it, they get to fix it and explain to my client why they should continue to be his developers. Though my client would CC me on such email so I could either laugh at them or facepalm.)

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
1

There's a trend to blur the distinction between dev & operations. Make your developers sysadmins and your sysadmins developers.

In this sense wordpress could would benefit from some work on automated (and programatical) provioning of blog. Many wordpress users maintain more than a few WP/WPMU instances and updating them timely is cumbersome at the least.

You can find a nice (and fun) overview of the philosophy at Agile Infrastructure slides from Agile 2009

Denilson Sá Maia
  • 561
  • 4
  • 8
scyldinga
  • 178
  • 1
  • 6
1

I 100 % agree. Dev are most of time non aware of how syadmin things work. If they need anything, they ask you,that's all. You think about how and when and you deliver them a useable package. (like they need to send emails, YOU are the one who's gona configure postfix). Moreover, they'll tend to think that in most of times, root access will make the things work if they are not able to do it with normal access. I agree with others here, they won't be with you at 2 am when u'll see a problem. I had the case few weeks ago, a developper wanted to update his wordpress. I told him to RTF changelog, for him, that was useless, the update process is made through a beautiful interface. Well the update did not work, I had his application saved, I made the backup script not him. Without me he could not have been able to restore the site. So take care about these things, we spend time to setup strategies (chrooting, backup, configuration) that's why we are here (and we love it :) ) and dev should NEVER interfer with these things (give back to Caesar whats belongs to Caesar)

Razique
  • 2,266
  • 1
  • 19
  • 23
0

Root == System Admin.

Users == Developers, DBAs, or Users.

Root knows no sleep when a server is down, root protects users from themselves, root protects the users' data from the net, root puts the server's health above all users. Root's ass is on the line when the server is offline. Server happy, root happy!

Common Reasons for unscheduled down time: Users, Undocumented changes in the environment, and backhoes. Servers do exactly what they are told they don't just randomly break. Hackers you ask, its not if, its when... thus the need for a "root".

00:33 CDT do you know where your backups and disaster recovery documents are? :-p

ForgeMan
  • 391
  • 1
  • 8
0

Sysadmins should have admin access (just like the title says). No one else needs access to production servers. If the developers need to troubleshoot a problem on a production system, that problem should be reproducible in the development environment. If it isn't they can sit with the sysadmin and look through the system.

Developers don't like not being able to touch production, but that isn't there job. There job is to write the software and hand it over to sysadmins for a production release. If they have documented everything (and keep in mind that in most shops documentation is a dirty word) correctly then the releases should go fine.

In public companies here in the US, you have SOX, HIPPA, etc to deal with. Most of these godforsaken regulations actually help with this argument. SOX dictates separation of duties which requires that developers keep there hands off the production systems.

mrdenny
  • 27,074
  • 4
  • 40
  • 68