Which changes that you have implemented did have the biggest impact on saving time in you daily sysadmin workload? What are your tricks to work more efficient and get more things done or work less for the same results?
I'm thinking about automation, changes in workflow/processes, new tools, stop doing some things altogether, outsourcing, better delegation, changing software/hardware, reducing bureaucracy, etc.
monitoring+alerting - which is great safety net. just as developers write unit tests to make sure things don't get messed up when they update code, i rely on monitoring as additional safety net just in case i screw something up [ that is disconnect a server, deny production traffic on firewall etc ]. it gives a peace of mind - if things break i will know before customers call.
For me, it has been centralized authentication.
I got to the point that I was administering 40 or so Slackware machines, and each one had local authentication PLUS local Samba authentication. I also had a VPN solution where each account needed to be setup, plus an internal jabber server and an internal email server. Everything had its own account. MAC (Moves, Adds, Changes) were insane.
So I switched from Slackware to CentOS, created an Active Directory infrastructure, and used Likewise Open to authenticate all of my Linux machines against AD. It probably saved me 20 hours a month without joking even a little.
Now, I've got everything authenticated through AD that I can, and it works tremendously. I can't recommend centralized authentication enough if you're still doing things the bad old way.
I agree with the obvious choices here; Automation and central authentication. However, it appears that I have to be the guy to mention documentation.
By documenting as many problems, workflows, installations and guides as possible people were able to work through some of their issues without the need to get our department on it.
Another great time saver is issue tracking.
Being able to prioritize tasks, assign them to team members and getting rid of all the clutter of people sending in requests by email, msn or simply coming to the office. This also helps our good friends, the managers with seeing how efficient you are (if you want).
Then of course, the icing on the cake would be my 'RTFM' (Read the Fine Manual) mug that gets raised a lot.
Infrastructure automation with a tool like Chef or Puppet is the best thing I've implemented on systems I manage. Monitoring is great and all, but often, getting the various bits to play nice with the rest of the infrastructure requires a lot of work. Chef and Puppet are both great at automating the entire infrastructure, providing a lot of glue that used to be written by hand. Particularly piecing together which servers provide what services.
Chef has, built in, the ability to query the server for attributes and recipes applied on other nodes, so you can ask who the production web servers are, or who the database master is, making automation much easier. Puppet can do this too, but it requires an external node classification tool like iClassify.
This should have obvious implications for monitoring and trending tools like Nagios and Munin. It can also, for example, provide automated configuration of load balanced environments, so the LB's can query all the web servers that need to be covered for a particular app.
The other big time saver I've implemented in a variety of environments is automated builds, like kickstart (redhat/centos) and preseed (debian/ubuntu). This should be obvious for most people, but it can be surprising how many sites still build systems from hand off CD. It's even better if the automated build gets the system ready to run Chef or Puppet to get all the other goods ready.
Multiple monitors, with a window open on the console of each server I was responsible for.
Monitoring is great of course, but not sure it's a time saver. For my money it was centralized logging, with a viewing system that filtered out the mundane, highlighted the dangerous (disk failure, virus scanner finds) and displayed everything else for categorization.
syslog (and perl) for the win.
It basically allowed me to read the event logs of all the computers on the network while eating my bagel; at least a cursory check to look for anything scary. Huge time savings.
Removed local admin rights for all users. (if I can)
This has had the effect of now I don't have to deal with any of the, how do I get X application to install (cause you are not allowed to now), my computer has a virus/spyware, my computer is running slow and pretty much anything related to that.
I never relized how stable this made the workstations until I came across one workstation that had been completely missed in the audits, windows updates, etc. It had been running for about 4 years without a single update done to it. I think I it was on Windows SP1. That site never once complained about any issues with it and when I did discover it, I found it running great.
Virtualization (VMware)
Between deploying servers from templates, managing servers from a single interface, detailed hardware monitoring built into the infrastructure client, it has really changed how we administer our infrastructure.
And the impact it has had on how we think of our "hardware" has really made it a game changer. Clusters are no longer "too expensive" because we can deploy them virtually. Need more Citrix servers, clone it. As long as we keep our physical hardware farm providing adequate resources (and those servers are truly commodities now), everything is peachy.
cfengine for config management under debian/linux.
openvpn to connect the satellite stations
tighvnc through the vpn to reach the 2300 Windows Clients in the satellite stations.
munin and nagios for monitoring and reporting
smartmontools on the servers for automatic healthy checks (and status mails, if something goes wrong)
raid1 (mirroring) on the servers. If one disk fails, the server is still running - this ist really major for my time management.
TeamWork - It's useless to do all the things on your own. share the work and together your are more productive.
working atmosphere respect your mates, do jokes and laugh - this is enormous usefull for an positiv and creative atmosphere.
Enjoy your job! man, I love my job! This makes me really goog (I think ;))
Puppet. The idea of changing one place and having all the systems that are affected is fantastic.
Couple that with standard installs, and, it's very fast to bring a new system up. You netboot and run a stock install, and then puppet takes over and everthing is configured.
Finally, standardize. No, you really don't want 35 different linux distros and 4 different Solaris versions. Work to move to one standard install. Each unique system that you turn off saves you loads of time.
Add another vote for monitoring. The principle is quite simple: I want to know what's happening before the users are affected. System Administration should ideally be a transparent role. Users should neither know nor care about what you're doing. From their perspective it should just simply work. Happy and satisfied users should equal happy and satisfied admins.
One thing that is often overlooked in IT is that the computers are there to work for us, not the inverse. Nevertheless I know admins who spend a significant part of their day manually checking their servers and the logs. Why? Computers can monitor each other and with a little scripting you can have just the interesting part of the logs delivered to you. Really, you don't need to wade through a few million informational entries, such as Fred's print job or the DNS transfer was successful. Just tell me when they're not.
The biggest time saver I have implemented was Disk Imaging of our production workstations. They are all the same and no one stores anything locally, so if there is a problem I just re-image the machine and it's all set to go, good as new.
Time saved is most important when systems are down.
I documented all support contract info in standardized text files in a standardized directory structure. I had one central and kept more than one copy around.
Each bit of information (web portal, phone number, point of contact, expiry date, contract number, phone menu shortcuts, etc.) where preceded by a standard tag in old .ini format (tag:data).
Finding a phone number was as simple as going to the top level directory and running:
grep Phone */*support.txt | more
Where the first wildcard expanded to the vendor and/or product name.
I did not use Excel, Word, OpenOffice, a database, etc., etc., simply because when something is down, that something might be the very thing holding your support information. Also these are not easily viewable from a text mode console screen.
I also documented all how-to's in the same manner.
My biggest time saver, so far, has been SSH keys + ssh-agent + keychain as described in this IBM article on OpenSSH key management
No more passwords for shells, scripts and scp.
NOTE: I still have a passphrase on my private key.
I implemented an IT Department Wiki (using Mediawiki for those interested) several years ago. When we started getting comfortable using it, the reply to many questions asked around the office was "Did you check the wiki?" It took a little time for us to get used to checking the Wiki for specific information, but once we did we realized it's great potential. We have all the information we need right at our fingertips -- if there is something that isn't in there, we have the ability to add/change a page quickly.
My biggest time saver was preseed scripts for the installation of our linux workstations. we have contractors coming and going all the time, so we have a pool of workstations that get re-used on a regular basis. When they come back to IT, we shove the install CD in, add the preseed file to the install command and within 20 minutes (and no further keystrokes from anyone of us) the box is back to a fresh, working base install, with all the tools and pre-configured to run in our network. Plug'n'play.
I have to agree with pQd about monitoring.
Although we haven't seen the full benefits yet, implementing configuration management using Puppet is promising to save us a considerable amount of time and massively reduce mistakes.
I found AutoIt and later AutoHotkey and fell in love.
The last job I had was for a custom vehicle manufacturer. The assemblers were minimally proficient computer users and managed to crash the program they had to use to enter which job they were on. Everyday, several times a day, I had to go around to 15+ workstations in 3 different buildings and kill the crashed program, relaunch it, and get it back to the data entry screen. I eventually installed VNC so I could do it remotely which cut down on travel time but still involved me remembering to go in and reset the machines every so often. When I found AutoIt, I realized I could set the program to watch the computer and if there hasn't been any input for 5 minutes, it could reset the program and type and click everything needed to get it back to the input screen. Doing this saved me at least an hour a day and made finance very happy since less people complained about the computers being down and more people were entering their job data.
++ for central auth and account management including account creation AND termination handling. We have AD (two forests) and LDAP (and, until recently NDS) with various groups accessing resources existing in either directory. The time we put into getting the directories in sync and getting all resources managed in one or other of the directories has been worth it in spades.
The next-biggest win has been any amount of automation whether it's account cleanups, config centralization, or what have you.
I'm not sure how much time actually monitoring saves but it's essential. It doesn't take all that much for an environment to get big enough or complex enough for "manual checking" to be impossible and ineffective. Plus, it's nice to sleep sometimes. ;)
This may be a bit off out of the main vein of thinking on this but we also had a huge win when we standardized our hardware platform. We picked a server platform that was workable for all our OS in-house and have stuck with it for several years. We learned the hardware and we learned the remote management of it and it saves time and energy in various ways:
Same goes for standard, documented, and reviewed OS builds. May seem basic but I bump with shops frequently where the builds aren't standard and there's no end of messing around to see if this tool or that tool is present or whether a particular server has its settings right. That kind of chaos can turn even the most basic tasks and problems into firedrills.
Learning to delegate and trust my colleagues - once you know that you can hand off bits of work to other people life becomes much more relaxed. And not because I'm lazy and have everybody else do my work for me; it's the peace of mind that comes with knowing you have good backup. And, of course, well-monitored, standardized OS configurations on standard hardware. Goes without saying.
My goal for automation has always been that now again I get an email saying "Such-and such broke on server foo. It's been fixed." and then once I've sent the bug to the developers I can go back to reading the paper and drinking coffee. We're not there yet, but we've come a long way from the reactive chaos we used to fight through every day.
I recently implemented AntHill Pro at work and now have all our builds and deployments for a number of projects completely automated and tracked. This included creating a shared Tomcat deployment Ant library that all the projects use, simplifying the maintenance of those projects in AHP. Next up is creating a similar library for site CDA deployments.
While it doesn't save me time personally, it saves the time of our developers and our operations staff. I enjoy being the oil can for other people's wheels. :-)
I'm hoping to look at Chef and Puppet next to help out on the infrastructure side of automating things.
Oh, and documentation is HUGE help. It saves a lot of time to just point people to a well written document rather than answer the same question 20 times.
