25

I am lead dev on a team of five people. Only three are coders. And only the coders are really technical enough to do a roll-out of our application to a production server.

Our app is pretty successful and we have attained some high profile clients. These clients typically put us through pretty brutal security audits. For the most part, we pass these audits. But one area that trips us up is separation of duties. e.g. They want one person/team to handle dev environments, while one person/team handles the production environments. And furthermore, sometimes they even want the person who does the work (like pushing to production) and who monitors/supervises/approves the work to be separate roles as well.

The problem with only having three tech people is that we can't keep our bus factor high while keeping our separation of duties that some clients require.

Thoughts? Solutions? Have you faced similar resource problems?

p.s. Tried to tag this devops and bus-factor, but I don't have enough rep to create new tags.

Angew is no longer proud of SO
  • 374
  • 1
  • 4
  • 10
Neil N
  • 359
  • 2
  • 6

2 Answers2

19

I haven't personally had to make this decision, though I do work in an environment where such separation of duties is implemented.

Are you the owner of this business or are you working for someone else?

Reason I ask is because realistically this comes down to a business decision. (barring the obvious infosec perspective of - "SEPARATE ALL THE THINGS!")

enter image description here

To find the answer, you need to ask yourself/owner/CFO this question:

Is the cost to add more employees to implement separation of duties less than the amount of business that will be lost to clients who require such, or can you raise the cost of the software sold to cover the salaries of more employees without losing additional business due to the price increase?

If so, then you can add more employees without putting yourself in the red, financially.

k1DBLITZ
  • 3,933
  • 14
  • 20
  • This answer nails it, IMHO: All business boils down to profit and loss. – Panther Modern Sep 03 '14 at 16:45
  • Unfortunately it's a little bit of a chicken and egg problem. We can't land giant contracts without more people. We can't pay for more people without big contracts. These audits take place BEFORE a large client commits. And even when they do commit, it could be 90 days before we see a check from them. – Neil N Sep 04 '14 at 19:54
  • It sounds like you're at a crossroads. Are you happy with the status quo or do you want to take the business to the next level? With great risk comes great reward. You've reached a point where you need to spend more money to make more money. If the capital is not on hand you may consider taking on some investors or perhaps a loan. Another option may be to hire interns, or find people willing to accept a job offer based on deferred compensation. I don't envy the decision you have to make one bit, that's tough. – k1DBLITZ Sep 05 '14 at 13:48
16

Welcome to the world of enterprise IT security. As you've found out, it's easy for security departments to make recommendations waving vaguely at "best practice". It's much harder to implement this on the ground. But you can address this, and you will get some benefits.

One-click deployments - new releases should be packaged so they can be installed easily. You can combine this with automatic deployment using something like Puppet. At that point, deployment should be simple enough that you can leave it to a less-technical colleague. One benefit of this approach is that it avoids deployment mistakes. The developer can test the full deployment in pre-production.

Log correlation - have your servers log to a central log correlation server, using something like Splunk. You can give your development team read-only access to this without violating separation of duties. You can also do centralised monitoring of CPU, memory, disk, etc. Ideally you will have enough automated analysis of logs that a less-technical colleague can judge whether things are ok or not.

Automate as much as you can. Routine maintenance should be scripted and scheduled. You may also want to script certain responses to problems, such as restarting an app server.

Emergency access - sometimes your app will break in a way that you need a developer to login to live to fix it. This is acceptable even with separation of duties. You probably want a process that looks like: "ops guy identifies fault, requests assistance" "manager approves emergency access" "ops guy issues developer a temporary password".

If you do all that, you can use your two less-technical colleagues as your ops staff. Your bus factor (nice term BTW) is not seriously affected because you've de-skilled the ops role.

Are you're running your app on premises, or in the cloud? It will be easier for you to comply with this requirement using a platform-as-a-service cloud.

paj28
  • 32,736
  • 8
  • 92
  • 130