87

Through reasons that don't warrant exhaustive discussion, I find myself in charge of 10 servers:

  • A domain controller-~500 hosts/~350 users
  • IIS web server-This is where we make our money
  • SQL server-The crown jewels
  • Exchange server
  • Linux box for data entry
  • AV server
  • Backup server
  • A few others tossed around

The company where I work believes everybody is replaceable and therefore believes they can pay a minimum wage for any position. The IT manager and Sysadmin quit recently and I think I was the only person who did not take a big step backward when the call went out for volunteers. This also explains why someone with my background is in this position. That is the reality, as much as I wish it otherwise.

What are the things I should be doing to keep those systems running? There is no written procedure left behind and I crammed the A+ and Network+ certs in the last two months but that leaves me with some theory and no practical experience.

I am in the process of teaching myself powershell but from here to there is a long way. I have no scripting or programming experience.

What tasks should I be performing? What practices should I implement?

I understand I am probably hosed but a lifeline to get me through would be helpful.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
chipandachair
  • 471
  • 4
  • 6
  • 68
    After reading that, I think your first step is to find a nice cushion to put in the corner. You're going to have many hours rocking on it. With a box of kleenex to wipe the tears. You might also want a pad of paper to start updating your resume while you're at it. My condolences on the spot you're in... – Bart Silverstrim Dec 09 '11 at 15:53
  • As @Bart Silverstrim says. Good luck. Not a good place to be – Dave M Dec 09 '11 at 15:55
  • Yes. Second step is finding a job you are qualified for. Point. – TomTom Dec 09 '11 at 15:55
  • 86
    Honestly, this is an entirely serious suggestion. Find another job asap. I've been in IT for 7 years, 4 of them in Systems Administration and I'd find managing all of that by myself to be a handful and I know these applications inside and out. For someone with no background, it's suicide. If you're to be held accountable for their well-being, you'll probably end up losing your job over it. Get your resume out ASAP, or ask to be relieved of these duties. – MDMarra Dec 09 '11 at 15:57
  • 13
    http://serverfault.com/questions/9766/what-a-beginner-should-know-learn-for-sysadmin-job – Zoredache Dec 09 '11 at 16:03
  • 2
    Plus one'd for the history major comment. I'm a IT professional with 10 years of experience, and my brother is a history teacher, and we often joke about what would happen if we traded jobs for a week. The kids wouldn't be all right, but it wouldn't matter because there'd be no working computers to store their failing grades. – Basil Dec 09 '11 at 16:10
  • For what its worth, if the company doesn't value its employees, I'd be hard pressed to stay around. Especially with the high stress situation you've just been put in. With the attitude they have, they will not have your back when s*** hits the fan. There's already some good answers to get you started if you decide to ride this out though. – Holocryptic Dec 09 '11 at 16:26
  • 10
    I am sure this will read as self-serving since I work at a managed service provider, but you may want to see if you can acquire an emergency budget/funds for outsourced help. Find a vendor in advance that can handle your needs. I would have a short list of paid help should it be needed. Given the climate, I doubt this will be possible but certain something I would try to do in your spot. Good luck. – jeffatrackaid Dec 09 '11 at 16:31
  • 15
    @Bart Silverstrim- I am on my third pallet of kleenex – chipandachair Dec 09 '11 at 16:53
  • `What tasks should I be performing`...well, I'd work on your CV and head towards to the door, personally. Although, seriously, best of luck to you. – tombull89 Dec 09 '11 at 17:03
  • 8
    If your CEO allowed this to happen in a company reliant on their IT, how can you trust the company will still be in business for very long, let alone getting fired yourself? – JamesRyan Dec 09 '11 at 17:35
  • 10
    For all the negativity (and I agree with most of it) IT and server admin are good jobs to be in if you can find it in yourself to do it, which, based on your question, appears to be the case. If you can do even a passable job handling these systems and the company for a year or two, keep that CV updated and start actively applying for roles at real companies doing comparable stuff. You could be making a load more doing much more rewarding work if you tough this out for just a little while. Don't jump ship just yet unless you absolutely have to. – music2myear Dec 09 '11 at 17:36
  • 1
    I agree with music2myear, think of this as an opportunity to break into a new career field. If you end up enjoying the work, it can be lucrative and rewarding to be a sysadmin. – Bigbio2002 Dec 09 '11 at 18:12
  • 2
    +1 Your history degree has helped you better filter authoritative from non-authoritative sources. – Doug T. Dec 09 '11 at 18:29
  • 1
    Simply put, you can't replace two experienced IT professionals with one untrained volunteer. It's irresponsible of management to create this situation. – bgvaughan Dec 09 '11 at 19:41
  • 2
    Everyone already beat me to this, but the canonical answer is **Seek. Professional. Help.** -- Production servers aren't the place to practice and learn, and usually when the last team made a hasty exodus the environment is hanging by a thread anyway. Knowing that you're out of your depth, if you think system/network administration is something you want to do then get a consultant in to help you make sure your environment won't fall apart, then treat THAT process as a learning experience. – voretaq7 Dec 09 '11 at 19:42
  • 1
    Run. Just...run. – Ben Brocka Dec 09 '11 at 20:36
  • I'm just curious here... What were you hired to do? – Jacob Dec 09 '11 at 21:50
  • I'd like to express my sincere sympathy by upvoting this posting. – Adrian Grigore Dec 09 '11 at 21:53
  • I know one history major who is copying very well with server admin duties. Don't give up! – matcheek Dec 09 '11 at 22:38
  • 1
    Just realized the title *should* have been "10 servers to administer and I'm history". ;) – NotMe Dec 09 '11 at 23:48
  • 3
    No one is saying the OP *can't* do it. He's studying like crazy and seeking advice; he's obviously putting up a helluva struggle before drowning. The fact is that he's still in way over his head, with expectations on him he shouldn't have, and he has no one on site to help guide the history major into the admin role. He needs at the *very minimum* outside consultants to come in and help him...while he updates his resume. Ideally a change of career happens as a padawan to an experienced sysadmin...not thrown 10+ production servers and no docs! – Bart Silverstrim Dec 09 '11 at 23:49
  • 2
    If most of the servers are Windows, "Have you tried turning it off and on again?" That's a pretty good troubleshooting start. (hope that brings a smile or two...if not, that's something else a sysadmin needs to learn. Geek references.) – Bart Silverstrim Dec 09 '11 at 23:51
  • Oh Man! How dumb can you get and still breath. Jump ship! sounds like they aren't paying you much anyway and expect the earth. If it all turns to custard they are probably going to fire you anyway. – hookenz Dec 10 '11 at 00:58
  • 1
    Last time I was in a situation like that, I applied for other jobs. Once I found one, I asked my current boss for a $5 dollar raise and he said he'd give me 50c (he was paying me min wage). I decided then was a good time to put in my two weeks. Fast forward 5 years from there and I was making ~7 times as much as I was at that old job. Once I had my bases covered, I've worked very hard and haven't had time to look back (nor do I want to). – Jacksonkr Dec 10 '11 at 03:53
  • He won't find a job for a month at least, which leaves him with 20 business days with this problem. So life-coach style "find a new job" isn't much practical use. – Chris S Dec 10 '11 at 13:58

13 Answers13

56

Honestly, I would find another job, unless your current task is just to keep everything running until they hire a new SysAdmin. You are being setup for failure. You are doing the job of at least two people if this is all hosted locally and nothing is documented.

Don't worry about the scripting or programming anything just yet. Get a handle on keeping everything running.

Are you in charge of the corporate firewall too?

The quick and dirty daily tasks I see you needing to do are (in no particular order):

  • check the nightly backups
  • check the exchange queues to make sure they are processing
  • check the SQL backups
  • check the AV server for alerts if anything failed to scan or update
Wipqozn
  • 103
  • 5
Top__Hat
  • 962
  • 1
  • 7
  • 12
  • I'll also add to learn the simple `IISRESET` command to potentially fix problems on your IIS server. – NinjaBomb Dec 09 '11 at 20:08
  • 5
    The line "IIS web server-This is where we make our money" says it all. A lot will depend on the programmers you are surrounded with. But given "everybody is replaceable and therefore believes they can pay a minimum wage for any position", I think jumping ship is the sensible thing to do. – DutchUncle Dec 09 '11 at 22:37
51

First off, look at Zoredache's link to the Beginning Sysadmin question. It's a lot of info but will get you where you need to go. That being said, it's a lot of data dumping.

Second, seriously, look for a new job. If your employer regards everyone as interchangeable widgets you're better off not being there anyway. But it's a bad economy, and that might not be an option.

In your specific case, I would:

  1. Find a local IT firm you can escalate labor to if something breaks and you're over your head.
  2. Make sure your maintenance contracts with your software vendors are up to date. Chances are most of your software packages (including whatever's on the IIS server that's making you money) were purchased rather than written internally. Make sure you have support with them - if anything breaks, you'll be calling them first.
  3. Make sure you have good backups - both data and system state. If one of your servers toasts, you'll need to be able to restore it quickly. This may actually be the most important priority.
  4. Do a hardware audit and make sure your servers are under warranty. If they're out, point out to management that there will be significant downtime if they die, and you only make money when they're up.
  5. Pray
  6. Perform a software audit, hope you find a lot of out of compliance software, report it to the Business Software Alliance (if you're in the US), and use the money you make from the finders fee to tide you over until you find a new job.

...6 may be optional.

Community
  • 1
Driftpeasant
  • 3,207
  • 2
  • 20
  • 28
21

Find a new job. Let that company fail. They clearly need a lesson.

Your stepping up to help will quickly become unappreciated and chances are, won't lead to anything else but more stress and abuse.

thinice
  • 4,676
  • 20
  • 38
  • My company took similar lesson while they wanted to hire cheap Solaris Sysadmin by hiring guy with some Linux administration experience and... I found that I've got better knowledge of system administration that the guy (I am a programmer and he was sacked three after three months...). – Xaerxess Dec 10 '11 at 10:55
12

Start drinking? :-)

Besides that, look at getting a spare box or two that can be used to set up non-production servers. That will give you the opportunity to learn how to deal with some of this software in a slightly less dangerous environment than "if I break it we all stop working and making money".

Even if you run them inside virtualization software like VirtualBox, it's something that gives you hands on with the software without jeopardizing the production work.

Finally, if you do have to touch the production servers before you're more comfortable with them, document every step you take. Make sure you always know how you got to the screen you're on and how to get back.

afrazier
  • 710
  • 4
  • 7
  • 9
    +1 for "Document every step you take." I would also add notes documenting the "why" as well as the "how." This will be helpful when backtracking (to say nothing of its importance in an unhealthy organization). – Roy Tinker Dec 09 '11 at 17:46
12

Honestly, find a mentor!

Not knowing where you are, it's tough to recommend a forum, but meetups are pretty common with IT folk so google your local area for a Sysadmin (or specific techs) meetup. You'll get a whole bunch of new contacts out of this too; which might help in unexpected ways.

Update:

I'd also highly recommend you start writing a blog about this (pseudonym required IMO)!

hafichuk
  • 762
  • 2
  • 5
  • 18
  • +1 for find a mentor. Two mentors, actually: a career mentor, yes, but also a paid outside contractor to consult when you have questions or problems. If your boss won't approve paying an hourly rate in the neighborhood of US$100-200/hour to bring someone in when you absolutely need help, run like hell. – Skyhawk Dec 09 '11 at 22:04
11

So, you will be learning on production servers and this is not your choice. In this case, you should not be blamed on any service interruption. It is the company responsibility to find a replacement for the sysadmin.

You can learn many things, but it will take time and it may hurt the running services. You have to be extremely careful before doing any change and you should do it on a development/testing server first.

There are many related questions on this site such as: SysAdmin & Developer: Responsibilities, https://serverfault.com/questions/51006/how-did-you-become-a-sysadmin, and What's the difference between sysadmin and an IT Consultant?. I hope these will give you an idea what should be done as a sysadmin if you could not get out of this ASAP.

Community
  • 1
Khaled
  • 35,688
  • 8
  • 69
  • 98
  • 21
    "Should not" and "will not" be blamed are two entirely different things. – Driftpeasant Dec 09 '11 at 16:21
  • 3
    What @Driftpeasant said -- Invariably when the fecal material hits the rotary air displacement mechanism whoever is in the SA chair gets blamed, even if you made them sign papers holding you harmless. – voretaq7 Dec 09 '11 at 21:47
  • 1
    I'd pay extra for a fan named that ... – thinice Dec 09 '11 at 22:16
  • Based on what the OP has already told us about his company, I'm fairly certain that anything that goes wrong with the server will result in the blame being placed on him, regardless of whether or not it should be. – Wipqozn Dec 09 '11 at 23:03
11

Protect your jewels!

Brent Ozar has an excellent "Oh crap, I just inherited a database server" script and even a training video now. He calls it the "60 minute database blitz" but I think my title is implied.

Remember this

Your backups are important but nowhere near as important as your restores.

What's that mean? Yes you need to take backups to have a chance of unbuggering a situation but if you don't test those backups, all you're doing is burning disk space.

billinkc
  • 276
  • 2
  • 16
7

If for some reason you decide you actually WANT to do sysadmin work, I heartily recommend Tom Limoncelli's Practice of System and Network Admistration. He provides excellent examples of how a well-operated organization might be run and if you get an opportunity to suggest changes, it's tough to go wrong with his recommendations. Book link to Amazon: http://amzn.to/sNUmsr

Magellan
  • 4,431
  • 3
  • 29
  • 53
7

You never want to be in a position where you are forced to learn on production systems. Learning involves making mistakes, and mistakes on production systems cost your company money. They may not care about anything else (which is clearly indicated by the very fact you are in this position to begin with) but they will care about this.

  1. If it is at all an option find another job. This will not end well. And honestly do you really want to work at a company that treats everyone as replaceable?
  2. Request an emergency budget so you can find a consulting company to keep things running and act as a resource while you get up to speed. If this is denied, make sure you have it in writing somewhere for CYA purposes.
  3. Read any documentation left behind by the previous IT staff.
  4. Perform an inventory of both physical hardware and devices. Figure out what is in warranty and what isn't.

Your goal should be to find another job and leave without anything breaking. If something does break (and it probably will) be prepared to walk out. This means you should be extremely careful regarding CYA procedures. You need to assume that no on at the company has your back - make sure you're not in a position where you need to rely on someone's support.

While you're looking for a new job setup a environment to practice and learn with and do your best to familiarize yourself with the current infrastructure.

And for what it's worth... my degree is in Philosophy. You can succeed in this field if you want to, but not at this company. Good luck.

6

Reading everybody else's comment I would say:

  1. Take this opportunity to learn
  2. Tell your management without sugar coating that they are in a dire situation
  3. Take this opportunity to learn more. After all sysadmin / DBA related jobs are well paid.
  4. If you do get fired, collect some money from them and find a new job with your newly acquired skills.

I love the write a blog comment, you have pretty good writing skills. Make sure to put a link where we can find it.

Ben Pilbrow
  • 11,995
  • 5
  • 35
  • 57
Nicolas de Fontenay
  • 171
  • 1
  • 4
1

There are quite a bit of scattered answers here. Here are some of my thoughts.

Some people say you should do it and others say to completely avoid it. My question to you is are you ready to devote hours on and working on these servers? Not only that, but are you also willing to learn things completely new?

Server administration isn't tough once you grasp it, but I've ruined servers by running the wrong commands, not following the "standards" (like taking daily backups), and et cetera. Every system admin will fail sometime, hey, it's how we learn, but you have to be prepared on how to handle it. It's a good amount of work and the biggest question here is if you're willing to take on the job.

Sure you can learn something new to put on your resumé, but if you are not willing to learn or devote yourself to you job, you're just going to be another person who "hates their job".

Taylor Jasko
  • 686
  • 4
  • 13
1

I will be frank, this is a handfull, even for more experienced guys, also learning most of the skills in server administration is learning from mistakes. I doubt you want to make mistakes on live-systems.

The danger of you not having any or few experience in server administration also makes it dangerous for your company. You are the victim of ignorant managers it seems. The only way to wake them up would probably be a sudden disaster, where disaster recovery is nescesary and nobody will be around who has the knowledge to make it run again and their crucial business processes stop working.

I'm not implying you should create a disaster, but just go and find another job, because you don't want to be held accountable when things go bad.

Lucas Kauffman
  • 16,818
  • 9
  • 57
  • 92
0

Seize the day! This is a great opportunity. Get a subscription to the Safari Tech Books database and read up on all of the how to books and certification books for the systems you administer. Start by focusing on backups and restore procedures. Get yourself test servers to play around with before laying hands on the production models. Test servers can be cheap desktop units, they don't have to be expensive or durable like production ones. Make yourself a goal to be successful in this position for two years and then go get a new job with a good company that is willing to pay you for your great knowledge and experience. Also, earn certifications for all the technology you are responsible for. You are poised to have a lot of fun and earn a lot of money (way more than a degree in history would get you).

MattChorba
  • 119
  • 8
  • 3
    One important question for our history major to ask: "*Is* this fun for me?" If not, that might impact the feasibility of an enthusiastic approach... – Skyhawk Dec 10 '11 at 01:41