At what point does a development company need a dedicated sysadmin?

Question

When a company's primary focus is developing, deploying and maintaining software, at what point would it be prudent to hire a dedicated sysadmin? What are the warning signs that you need one?

Answer 1

The person or persons who are handling system administration tasks are spending a significant percentage of their time on those tasks. The percentage may vary from organization to organization, but it's a tradeoff on that person spending time on system administration versus their primary responsibility. Another factor is the quality of the network service and up time. If you have someone who is barely managing the network or is having trouble keeping it up, then you should have a dedicated person who knows the network and can act in a proactive manner.

So some basic rules of thumb:

1. Do you have outages or degraded network service that could be handled by an experienced system administrator?
2. Are the current people handling system admins utilizing more than 30% of their time on system admin tasks (the % may vary).

Answer 2

As a contractor who provides sysadmin services to small companies, including ASIC and hardware design companies, I have this perspective:

A sysadmin's value is not primarily the difference between his direct cost and the direct cost of engineers who are sysadmin'ing rather than engineer'ing.

In my experience, most (but not all) engineers are good engineers but lousy sysadmins. They know how to do the engineering side, but building networks, servers, and services are not what they do all the time.

A sysadmin can increase his value if he can improve the productivity of the engineers -- for one particular industry I work in, we doubled or more the units of work done through a regression and simulation server farm by changing their networking infrastructure, increasing the value of their server farm, engineers, and very expensive tools licenses. This was a far greater value to the business than merely freeing the engineers from sysadmin duties.

Of course, not every company has low-hanging fruit like that!..

Value is also increased by avoiding downtimes that prevent larger numbers of people from working. I always say, if your IT resource never has to be on-site, then he's doing a pretty good job...

What you need is an evaluation from someone you trust, not just someone looking for a job or looking to sell you something (although they will be doing that, too). Someone who will come in and look at your site, front to back, and ask you questions about what you do, how you do it, and what you are trying to accomplish -- as these are all very different things!

Finding these people can be easy or hard, depending :) I suggest that you talk to your network of peers in other industries and ask them who they have used in the past, good or bad. Do your research. Ask for, and check, references.

What you also need to do is recognize that this will cost money. Time for a FT/P or a consultant (or team of consultants). I personally prefer customers to engage my company with a small fixed number of on-site hours per week. They then have access to my whole team and their varied expertise so that our responses can scale up (or out) when required without them having to have a large FT/P staff constantly on the payroll. At some point it makes sense to go with a FT/P of your own, if only as a senior IT or IT manager. But you don't need to go there right away if you get the right team.

Also realize that any new body will want to change the way some things are done, and initial changes can be bumpy. This is why you have to trust who you get to take you from where you are to some tradeoff between where you want to be and what you can/are willing to afford.

If you are asking the question, then the time is right to start doing research, and maybe have one or two consultants in to do an evaluation. Then take what they say and decide what the next step is.

Answer 3

Probably just before something bad happends:

• You realize you dont have a regular backup of a Line of Business app - such as something tracking licenses/customers

• You can't remember when production servers were patched or scheduled for maintenance

• You discover your file server is dead becuase TWO drives failed in your RAID array

• It takes longer than 1 day to get a new machine provisioned to replace of an old/existing machine

• Printers are out of toner or say "maintnenace required" on the LCD readout

• "All your base are belong to us" or similar network intrusion

• E-mail server is overwhelmed by the latest spam attack

I am sure there are other WTF moments that I havent listed in my short list.

Answer 4

When the amount of productivity you're losing by having developers and admin staff filling in for the absent sysadmin is greater than or equal to the sysadmin's salary.

Answer 5

Don't wait until your developers are getting bogged down with sysadmin-type tasks. Just about every developer I've met has some capacity to satisfy some level of system administration without giving any visible indication that they are becoming consumed. By the time you notice that your development team is being visibly impacted by the sysadmin load, you've probably already spent more of their time on it than you would care to.

Conversely, early on your system administrator is likely to be underutilized in the sysadmin role. Consider integrating them into the development team. I've never seen a software product suffer because it had the early input of an actively participating IT resource contributing feedback on technical requirements. It's usually the case that IT considerations get short shrift without that resource being involved.

If you're asking the question, my guess is that you've already observed behavior that indicates the tipping point has been reached.

Answer 6

I would recommend creating a log for a month. Every time you or an employee spends time on a sysadmin task, log who it was and how much time it took. At the end of the month calculate how much money it cost total for all that time, and double it. By doubling it you represent the amount of time you paid to have those people do sys admin tasks AND amount of time lost that could have been used to generate revenue (i.e. coding).

Once you have that number, ask yourself "Can I get a full time sys admin for that price per month?"

If yes... hire someone.

Also, look at service companies in your area. There are many that work on a retainer situation. This would be a good solution for that period between when you need someone but not full time. Know that a full time system admin is not your only option. Also, you don't have to deal with benefits and other personnel issues. NOTE: I work for a company that provides these services to small business so I may be a bit biased.

Answer 7

When your computer room looks like this. =)

_{(source: tvsnob.com)}

On a more serious note, you should when your developers are spending the time they could be developing on administration and operations. These tend to be very different skill sets so it is good to have someone come on board (even on a part time basis) ahead of the crunch so that you avoid situations like the one above.

Answer 8

It's also worth considering that a Sysadmin will bring experiences and knowledge that most developers won't have. They'll know how networks work, how to get the most out of databases, how to make applications scale. They tend to have a slightly different mindset to most developers, and that knowledge can be invaluable.

Answer 9

Depending on how big you are, I would start looking when

• one or more of your developers starts regularly spending ~40-50% of their time performing sysadmin related tasks
• your configuration has grown beyond your technical comprehension

Answer 10

Simple: 5 to 1.

Doesn't matter all the 'if' clauses. On average, you'll find a 5:1 in emerging companies. there are many metrics out there, such as "30% of developer time spend doing sysadmin work".

My personal (18 years) experience is that, by the time you have 5 folks developing code, someone needs to come in to help build centralized systems/services so those 5 good folks can continue being productive at writing code and not managing servers/software systems, and provide return on the (salary) investment.

Answer 11

Whever you are concerned about security and you know none of your current employees can secure and maintain servers providing the required uptime.

Answer 12

I think if you have to ask this, you probably need the sysadmin.

The business cases made above are great, and useful for getting approval. I'd say, however, that if someone is griping, or upset about the amount of time they spend doing this, that's a warning sign. If things are not done in a timely manner, then you need someone.

Don't forget that security, maintenance, basic tasks are often ignored, and you're gambling here that it won't bite you. If you can afford one, meaning it won't put the company out of business, you might want to just get someone. At least a part timer, maybe a college student, to help ensure that things run smoothly.

Answer 13

Start small if you need to, hire a consultant at an hourly rate to take care of your needs.

If you're paying a developer 200 dollars a day and he's spending a quarter of his time taking care of the network you have 75% of that developer's skills being utilized.

Use the 50$/day budget to hire an IT consultant to come out once a week for 2 hours and give your network a once-over. They can also give you a quote to get you into shape if you don't have the internal resources.

It's really about 1) do you have people who can do the job? 2) Can they do it correctly with experience or are they "winging it"? (winging it will kill you) and 3) Is that developer's time worth more than the cost of an IT consultant?

Hire somebody part or full time when it makes sense to do so. If you don't have somebody in house with experience in Systems Administration then do not use them as if they do. It WILL bite you in the ass later.

Answer 14

In reality, the answer to this is more of a business case than anything else.

You need a sysadmin when the total potential revenue lost for the time when the developers are doing sysadmin jobs rather than actual development is greater than it would cost for you to hire a sysadmin

Answer 15

When the developers are having to spend a large amount of their time maintaining the systems. While your developers will all be competent with computers, they may not have the skills needed to manage a network, and if they are doing so, won't be spending time developing, which is what they're good at, and what they're being paid for.

It really depends when your budget allows you to hire in a sysadmin, and there are lots of other factors to think about, such as if you have a domain set up, if it's managed well at the moment, how many servers you have and how much maintenance they need, and the state of your network.

Because your computers are the core part of your business, if they go down, you are losting time and money fast, so once you've got more than about a dozen machines, I'd advise hiring in a professional.

Answer 16

Analyze how much developer time is spent deploying and doing routine maintenance on the production server. You may need a dedicated sysadmin if this gets to be more than the typical developer's hours worked in a week if this is consistent for a few weeks. So, if there is enough time spent deploying and applying patches that would be an indicator to me.