What tools should every sysadmin use that no one's heard of?

Question

I'm wondering about tools that are powerful and that most system administrators either don't know about or don't use (but should).

For one thing, I like the possibility of finding out about a tool that is good and that I should be using - or at least, trying out. I also find that giving these tools their time in the sun (again) can be a positive, letting others know about the wonderful tools that are out there.

Thus, things like sudo, vi, emacs, dtrace, ps, and top are out. I have some ideas but I just hate to skew the statistics...

I'll just wait and see if anyone mentions my favorites.

Answer 1

iPerf

Iperf helps you run tests that measure maximum TCP and UDP bandwidth performance. It allows the tuning of various parameters and UDP characteristics reporting bandwidth, delay jitter, datagram loss: http://openmaniak.com/iperf.php

---

,And

MTR

MTR (My Trace Route) is also a pretty good tool. It combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. Like traceroute except it gives more network quality and network diagnostic info. Leave running to get real time stats. Reports best and worst round trip times in milliseconds... It shows latency, jitter (average/best/worst): http://www.bitwizard.nl/mtr/

use:

mtr server.domain.com (or IP)

• Add more fields with "O", type "LDRS NBAW V JMXI" and hit ENTER
• Type "n" to toggle DNS Off/On

alt text http://locobox.googlepages.com/mtrsavedw.jpg

Answer 2

Paperclips: I've had to use paperclips to pick the locks on the front of a sun and dell server We've all used them to open a cd rom

HotPlug This thing is awesome. Want to move a server to the next rack over without turning it off? http://www.wiebetech.com/products/HotPlug.php

Blocks of wood The idiot before you not mount the server properly? The server too old to have rack mounts? Blocks of wood are your friend. It's ghetto, but works.

Answer 3

A bent paper clip to pop open a CD drive.

Answer 4

Sysstat / Sar - superb historic monitoring for servers.

Network/CPU/Memory/IO/etc/etc brilliant if you need to examine why a machine went down.

For example load averages between 6am and 7am this morning:

hcooper@localhost:~$ sar -q -s 06:00:00 -e 07:30:00
Linux 2.4.27-3-686 (localhost)  06/06/09

06:05:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
06:15:01            2       200      0.00      0.05      0.04
06:25:01            4       199      0.01      0.05      0.04
06:35:02            0       208      1.74      1.39      0.79
06:45:01            1       201      0.12      0.52      0.70
06:55:02            1       197      0.04      0.14      0.40
07:05:01            2       203      0.13      0.10      0.24
07:15:01            2       200      0.00      0.05      0.15
07:25:01            2       200      0.01      0.04      0.08
Average:            2       201      0.26      0.29      0.30

Answer 5

A Paper Bag: To breathe into when you discover that the SQL script you wrote wasn't actually mashing up the test database after all. Oh, and your backup system crashes and now you have to wait on the phone with tech support because you don't have a support contract.

Origami Paper: To pass the time when you're on hold with tier 1 tech support at said backup software vendor.

A Stress Ball: To help when discussing the problem with Tier 1 support after they finally answer the phone.

An Ice Pack: To soothe your aching fist after pounding it into the desk when you realize it took 90 minutes just to get handed off to Tier 2 support.

A Spare Cell Phone: After you throw yours into the ground when you get disconnected just as Tier 3 support answers your call.

An Account at Monster.com and Experience in Laying Carpet: =(

--

On a slightly more serious note, having a small refridgerator can be handy especially for putting crashed hard drives in to try and recover data.

Also, it's even better if your camera phone can take video. I take video when I'm messing with BIOSs and other things that I don't have time to simultaneously document while editing.

Cough drops and nose spray for extended stays in the dry server room.

A P-Touch Labeler to label EVERYTHING even if you're certain you'll remember which patch cables connect to the security cameras.

Fingernail clippers to remove the sheath from the 4 pairs of wires in a Cat5 cable.

Silver Sharpies for when you have to mark on something that has a dark finish.

Answer 6

I use 'ss' which is part of the iproute package (which also includes the great ip and tc commands) to get information about tcp/udp sockets. It supplies more information that the regular 'netstat' and can be used with filters, for instance:

$ ss -o state established '( dport = :www or sport = :www )'
Recv-Q Send-Q Local  Address:Port                  Peer Address:Port   
0      0      192.168.22.100:52430                 209.85.173.118:www     
0      0      192.168.22.100:51115                 209.85.229.101:www     
0      0      192.168.22.100:49105                 209.85.227.104:www     
0      0      192.168.22.100:37792                 209.85.227.101:www     
0      0      192.168.22.100:60337                 69.59.196.212:www     
0      0      192.168.22.100:37794                 209.85.227.101:www   

Answer 7

I already mentioned this tool in another answer on SF.
sysv-rc-conf

DESCRIPTION:
sysv-rc-conf gives an easy to use interface for manag‐ ing "/etc/rc{runlevel}.d/" symlinks.

alt text http://blog.ubuntu-tweak.com/wp-content/uploads/2007/09/sysv-rc-conf.png

If you have ubuntu : sudo apt-get install sysv-rc-conf

Answer 8

"arping" to determine if a machine in the LAN is really up. Bypasses all blocking efforts of ICMP echo (ping), by using ARP packets.

Answer 9

couple of network tools:

tshark - real time text version of graphical network traffic analyser - wireshark. when raw output from tcpdump is just not enough for you tshark can do the trick. sample :

1041488.938623 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [SYN] Seq=0 Len=0 MSS=1460
1041488.964593 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=1 Ack=0 Win=65535 Len=0
1041488.997561 192.168.10.14 -> 123.13.42.143 SMTP Command: EHLO s72f30c9a2c784
1041489.034541 192.168.10.14 -> 123.13.42.143 SMTP Command: AUTH LOGIN
1041489.064026 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.095757 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.304390 192.168.10.14 -> 123.13.42.143 SMTP Command: MAIL FROM: <someone@somewhere.cx>
1041489.375849 192.168.10.14 -> 123.13.42.143 SMTP Command: RCPT TO: <anotherperson@wherever.se>
1041489.409579 192.168.10.14 -> 123.13.42.143 SMTP Command: DATA
1041489.470060 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.503278 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.529797 192.168.10.14 -> 123.13.42.143 SMTP EOM:
1041492.660752 192.168.10.14 -> 123.13.42.143 SMTP Command: QUIT
1041492.726452 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [FIN, ACK] Seq=2362 Ack=281 Win=65254 Len=0
1041492.734770 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=2363 Ack=282 Win=65254 Len=0

httpry real time passive http analyser. sample output:

06/06/2009 13:36:16     74.125.77.104   192.168.10.138  <       -       -       -       HTTP/1.1        204     No Content
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     wp.pl   /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        302     Found
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     www.wp.pl       /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        200     OK

both tools are very useful for troubleshooting some connectivity issues reported by users in remote offices / networks where you have access only to edge gateway.

and usual monitoring favorites:

• nagios for alerting, bash or any other scripting language to write your own checks of anything you want to monitor
• munin for plotting trend charts

Answer 10

pv pipe viewer. Insert it between commands in a pipeline to get an indication how fast data is moving, how long till it finishes, etc. It can also act as a rate limiter.

Answer 11

strace is useful in a surprisingly large number of places.

Answer 12

Windows: Nbtstat (has a variety of tricks, but -A to get the machine name of a remote IP can be really handy assuming you're in an environment where permissions will let this work)

All of the sysinternals tools, (process explorer, debug view, etc...). Particularly debug view, you'd be surprised how many server programs write interesting/useful things to the debug log and that's the only way I know to view it. Those are must-haves though.

Unix: strace/ktrace/truss Are my favorite tools (best non-obvious use, if a there's a process running that has it's stderr redirected to /dev/null and you don't want to stop it, monitor the calls to write on fd2.

lsof: (list of open files) I always forget about this one, but when you need it, you need it.

Answer 13

I would say the oddest tool I've had to use was a Putty knife to open a Mac-Mini (as directed by Apple).

Think Different!

alt text http://locobox.googlepages.com/putty.jpg

Answer 14

Earplugs for longer actions in a noisy data center.

Answer 15

A lot of the suggestion above are Network related. But for a SysAdmnin:

htop - an interactive process viewer for Linux

This is htop, an interactive process viewer for Linux. It is a text-mode application (for console or X terminals) and requires ncurses. Tested with Linux 2.4 and 2.6.

Or Human Readable Top

http://htop.sourceforge.net/

Answer 16

A magentized paperclip on a string. I dropped a screw in a full rack once. It was w-a-y down there. I had a strong magnet stuck to my desk but it didn't have any way to attach a string to it. So I found a thicker paperclip and magnetized it after tying some pull-string to it. I g-e-n-t-l-y fed it down the left side of the rack in the space between the server rails and the cabinet door. It took some poking, but it found the screw I needed. It took a few tries before I could pull it all the way back to the top without dropping it again.

Delicate work. At the time I was swearing vociferously that I clearly hadn't played enough 'Operation' as a kid.

Answer 17

Pipe Cleaners -- They are better than twist ties for tying off cables. They are longer, softer on the fingers, and color coded.

Answer 18

Unix centric answer (simple tools) :

• ntop (can't believe I haven't seen this yet)
• tcpdump/snoop
• double recommendation for sar
• rsync (life would suck more than anything without it)

Answer 19

Multitail is a must. Monitor multiple files on one screen. Can also follow file names instead of descriptors.

Answer 20

vnstat comes in handy - a little traffic monitor that gives output in the same way as vmstat, e.g:

> eth1
> 
>            received:       1.40 TiB   (48.4%)
>         transmitted:       1.50 TiB   (51.6%)
>               total:       2.90 TiB   since 20.11.08
> 
>                         rx      |     tx      |   total
>         ------------------------+-------------+------------
>         yesterday      6.15 GiB |    8.50 GiB |   14.65 GiB
>             today      6.98 GiB |    4.77 GiB |   11.75 GiB
>         ------------------------+-------------+------------
>         estimated      7.82 GiB |    5.35 GiB |   13.17 GiB

Answer 21

I second the flashlight comments, though I prefer a headlamp over the minimag or such. Leaves your hands free.

I also find a click-style ball point pen to come in handy. Useful for straightening pins on VGA cables (when the pen is retracted), pressing reset buttons, and jabbing in your own throat when you find your file server has crashed and everything is lost.

Answer 22

Systemtap rocks. Systemtap provides you with very simple-to-grasp tools to write a kernel module that inspects various parts of the kernel. Pretty deep tech, but on occasion very, very useful.

Apart from that I'd say: a combination of sysstat and rrdtool for long term trend analysis.

Answer 23

Looks like your post was slanted towards Linux but here's one for windows even I didn't know about until recently:

getmac

C:\Admin>getmac

Physical Address    Transport Name
=================== ==========================================================
42-35-7C-4F-85-1b   \Device\Tcpip_{72338DC1-13A4-8514-2C1B-60FC3B4559DB}
00-11-05-86-D2-C0   \Device\Tcpip_{CCD25CFB-7765-1BE2-C59B-57C05FD32B67}

C:\Admin>

Answer 24

od - octal dump

strings - find printable strings in files

Answer 25

iftop, this gives a much better output than ntop in my opinion.

Answer 26

Programmable Bash completion functions. Make life less error-prone. They're not that hard to write.

Some of the less obvious options to GNU grep, especially --color, -P, and -o.

My standard quick reporting one-liner:

grep something /some/logfile | sort | uniq -c | sort -n

Simple frequency analysis from a logfile:

grep something /some/log | grep -o '^... .. ..:..' | uniq -c

(adjust the second bit for how granular you want the frequency breakdown to be).

Answer 27

A good Minimag flashlight with fresh batteries is good to have when you need to peek inside a cabinet/server case/under a raised floor/etc.

Answer 28

Facter is a particular favourite.

It's frequently just considered as part of the larger Puppet configuration management framework, but extremely useful in its own right; giving a consistent cross-platform way of finding out core system information. Great in shell scripts, essentially.

For example:

  facter operatingsystem => ubuntu
  facter lsbdistcodename => hardy
  facter domain => mydnsdomain.com

It's also very easy to extend with your own (or other people's) 'facts', eg:

  facter local_postgresql_port => 5434
  facter has_hardware_raid => LSI

Answer 29

Windows

Although all SysInterals tools are great, the tools which helped me the most, were the monitoring tools:

• Regmon *
• Filemon *
• Process Monitor
• Autoruns

Nothing's better to quickly determine, why something is not working.

(* Regmon and Filemon are integrated into Process Monitor for newer Windows versions)

Answer 30

I'm going to go with a hacker tool, that might as well be useful to diagnose troubles: Cain. It can sniff a network and do a lot of attacks(arp poisoning, man in the middle,etc) on a network as well as breaking(or recovering) passwords.

What better way to know your network is secure.

Answer 31

Someone bought a very expensive scissor-jack that can lift about 300Kg to a height of 3m - a waste of money I thought - until we needed to fit 5 HP C7000 blade enclosures into one rack - boy it came in useful for that top one :)

Answer 32

Here's my list of things I didn't see mentioned yet:

sticky stuff (like wall tack or putty) to pick up little bits in that fall into tight spots (little screws on motherboards)

non-oily lube (like teflon bike chain lube) for tight or frozen bolts on racked gear (esp. great for tight-back-of-rack situations where there's not much room to get good leverage)

strong knife, v. stiff putty knife, or very thin-tipped prybar for popping the head off stripped bolts. Drills work OK too but there's the vibrations and the metal shavings all over.

locking vice grips including small & needle-nosed. In some cases, almost as good as an extra set of hands.

velcro strapping not just permanent cable management but hold things out of the way in packed racks w/o creating tangles or rats' nests.

slim-edge rack tool for popping the rack nuts in and out. I keep several in my bag "just in case"...they are commonly included with new rack-mount hardware but not always and they rarely are handy when you have to move something months or years later. Saves me a lot of ripped up fingers. They are also often just right for depressing the lock-clip on a tight network cable (esp. one with a boot) or a lock-clipped fibre patch for those of us with stubby or snausage fingers.

another use for a cellphone: some (easy) way to light up the screen as a quick flashflight for checking stuff out in the backs of cabinets

Back in my desktop days, I had kept a handful of CMOS batteries in my bag. This was esp. helpful when dealing with labs and offices where the PCs were 2+ years old and resolved many "head scratchers" in short order.

++ paper clip...tape monkey's best friend when a robotic library is good and jammed :D

Answer 33

Absolutely vital: A dentist mirror for those tight spaces!

Answer 34

One tool that I absolutely find indispensible is tcptraceroute 1 - this is a traceroute that does not use ICMP packets to perform timings, but uses TCP instead. This allows you to traceroute without regard to the typical blocking of ICMP: it works well.

Answer 35

I used this much more as a helpdesk support person than an SA, but I always keep a push/pull spring hook in my toolkit. It's a hook on one end, and a sort of s shape on the other. Perfect for getting bent pins on connectors straightened out.

The only link I can find to one is this: spring hook - but it should give an idea of what it looks like.

Answer 36

A good knife. Not a multi-tool, a KNIFE, as in something made by SOG, Cold Steel, or Boker. You'd be astonished at how useful it is.

Answer 37

I once used office chairs in place of one of the fancy scissor-jack devices Chopper3 mentioned. I needed to relocate the UPS's in a rack, I do not know why they were half way up in a wheeled rack, but they were. I'd scheduled the maintenence window on a Sunday. My helper had a family emergency, and could not make it. I had my 9 year old duaghter with me. I could not manhandle the UPS to the ground by my self, and she wasn't strong enough. So I pulled the UPS out as far as I could on the rails, but a chair under it, and had her hold the chair while I lowered the UPS on to the chair. After resting and re-gripping, I lifted the UPS, and she pulled the chair out, so I could lower it the rest of the way down onto a set of blocks I'd brought to help hold the bottom UPS at the right height for the bottom of the rack. Then the remaining UPS's when onto the stack so they where roughly at the right heights as well.

Answer 38

Here is a few tools I have come across:

SYDI

Auto network documentation tool.

Orca

Messing with .msi files.

RichCopy

An advanced alternative to Robocopy.

ImgBurn

Best free burning software hands down.

PowerGUI

A nice PowerShell editor.

Your Brain!

A lot of people seem to forget they have one.

Answer 39

Voice coil magnets taken out of dead hard drives. In case you don't know, these are very strong rare earth magnets. I use them for multiple things:

• Wiping hard drives that are being tossed out that I don't have time to take apart. They're strong enough to wipe the servo tracks from the drive, making it essentially unusable.
• Holding notes or wires in strange places: most of the magnets have holes in their backing plates that you can thread a cable tie through and they stick to any small bit of steel.

I've used CDs as mirrors to see behind boxes (usually desktops crammed under someone's desk) when I didn't want to pull them out.

Answer 40

For Windows:

PATHPING is nice (and built in) for some advanced ping stats

I also like QCHECK : http://www.ixchariot.com/products/datasheets/qcheck.html It's a great GUI based simple network analyzer.

File Unlocker is a life saver at times even on servers: http://ccollomb.free.fr/unlocker/

Answer 41

fldiff is a graphical diff program

Answer 42

All our apps are clustered, so cssh is pretty handy.

scp - only mentioned because so few people use it, relying on FTP instead. I've always loved the ability to grab files instantly from remote hosts. Secure, quick, and doesn't need a dedicated daemon running.

Answer 43

In my server closets, I always put one of those cheapie push-on battery operated lights. It's great if(when) there's a power outage.

Answer 44

Swiss Army Knife.

I have two, a big one I keep in my desk and a standard one on my keyring (just remember to take it off when you go to the airport!).

It even saved my life (or at least bad trip to the hospital) one day when I was cutting through some 240v wiring our electrician swore was disconnected (but wasn't).

Answer 45

12" prybar I have used it for lifting stuck tiles, grabbing cable bundles just out of reach, drifting a half loaded rack over a half inch so it was squarely on the correct tile.

Vice Grips Handy for extracting mounting screws some other monkey stripped out with a power drill.

The little slim jim A piece of bent spring steel that has one end about 3mm narrower than a square rack hole. This little guy with a hook in the end makes setting clip nuts in and getting them out a dream.

Rubber Chicken, and a Sledgehammer. I menaced a recalcitrant server with them during a reboot late one afternoon after a long and hard day of troubleshooting a problem the vendor had NO clue about. It lead to the best moment of my day when I scared the crap out of my director by walking into my managers office where they were meeting returning the sledge. Director asks "What's the sledgehammer for?" I said "I used it to fix the ServerX," The best part was, that menacing the server worked (that or the 5th reboot as recommended by support with nary a config change...did the trick)

Answer 46

Sledgehammer... for things that aren't working by around 4:45.

(from coworker)

Answer 47

ratchet and webbing (kinda like a tie-down for a truck) to 'lift' servers high enough to place on shelves or racks.

cart/wheeled chair for a 'crash-cart' console when you don't have a Cyclades port for each server.

indoor/outdoor thermometer to check for periodic hot-spots suspected in the datacenter.

garage-style retractable power-cord located semi-centrally for when you need power but dont want to hunt under the floor for a receptacle/plug.

Answer 48

Patience.

I know it sounds corny, but it takes a lot of willpower to just sit on your hands and wait or refrain from use of excessive strength (how many connectors have you destroyed due to losing patience and pulling harder?)

Answer 49

A MagnoGrip (http://www.thinkgeek.com/gadgets/tools/917f/). It holds onto screws and other small metal items for you. I tend to leave one stuck to the first cabinet in the room.

Answer 50

Over the years, I've used the tip of my favorite mechanical pencil for all manner of poking and prodding - everything from reset switches, to RS-232 connector pins, setting dip switches. I've even used it for pulling jumpers on numerous occasions.

jumper http://netstrata.com/images/jumper.jpg Pentel Mechanical Pencil http://pentelstore.com/images/products/P205A.jpg

Answer 51

Ok, I am baffled no one has mentioned Duct Tape yet. I cant be the only person that uses duct tape to hold things together.

Answer 52

grep, awk, netcat - can't live without them

For monitoring I use NetMRG and Nagios.

Uptimed is also a nice tool to monitor uptime and avalibility.

Answer 53

I try not to use too many "nonstandard" tools. That is, tools that aren't easily available between operating systems or out of a particular distribution's software repository. I work on a Macbook, and I do testing on a variety of different Linux/Unix platforms.

That said, I like ohai(1). It is the node data collection tool used by Chef. It outputs data in JSON, so it can be manipulated with a variety of JSON parsing libraries. Since I work on Chef :-), it's a "nonstandard" tool available on every system I work with.

(1) Disclosure. I work for the company that wrote Chef and Ohai.

Answer 54

Odd answer in a way as it's not a tech tool directly - but I couldn't live without EverNote. It's a centralised note taking system, it has Windows, Mac, iPhone clients and lets you clip pits of web pages (with links to the original), has catergories, records images & voice notes.

It's what I use when my brain breaks :)

Answer 55

It's not exactly a small cli tool as such, but I can heartily recommend Zenoss Core for network and system monitoring. It has restored my faith in monitoring-and-alerting software.

Where tools like Nagios, Cacti, etc. all have steep learning curves and seem to be good at only one thing, most often you have a need for an all-in-one solution, and Zenoss Core provides that for free (and Free).

It's hard to summarize, but basically it's an application with web-interface that autodiscovers devices on your network, then monitors tons of parameters, logfiles, ports, services, software, hardware, amount-of-pages-printed, etc. If anything fails or crosses a threshold you get an alert. It's easy to get started: enable/install snmp on each device you want to monitor, install Zenoss on a server, and open the web interface.

The free Core version is good enough for small and medium businesses, while the commercial Professional and Enterprise versions provide some advanced features.

If you ever thought about Nagios, Cacti or the like, be sure to evaluate Zenoss as well.

Answer 56

I personally never go anywhere without UnxUtils. It's got a very large amount of Nix command line utilities. Some of my faves include: find, wget, and sed. And of course it's nice to not have to remember to type dir/copy/etc when I'm switching to a Windows machine after a long stint on a nix machine.

Throw in a batch file to quickly add it to your PATH and you're good to go.

http://sourceforge.net/projects/unxutils/

Answer 57

sdiff. It displays two files in a split screen, showing the direction of change in the middle. It can also merge files interactively.

#sdiff -w 40 /etc/fstab /tmp/fstab
/dev/md0                /dev/md0
/dev/md1                /dev/md1
tmpfs                   tmpfs
devpts                  devpts
sysfs                   sysfs
proc                    proc
LABEL=SWAP-sdb2    |    /dev/sda2
LABEL=SWAP-sda2    <

Answer 58

I've once used a butane soldering iron to reseat a flapping component from a PCB whilst in the field.

It worked, I might add. But I avoid repeating if possible.

Answer 59

Paperclip (other use) - if a system is hung (particularly on a single processor, single core, CPU) and you don't want to reboot it, and all else fails, shove a paper clip in a USB port. I've found this to create a fault that allows the CPU to handle user input.

Rubber band - our office has dozens of printers, many of which are connected locally to their computers for security reasons. The parallel cables often pop out ... tying a rubber band around them holds them in place.

Fist - a quick slap on a rattling AC unit, or even a spotty fax, is often all that is needed to keep it going until a Service Tech can arrive.

Old analog answering machine - we have a few analog lines coming into our data center for telecom system for backup purposes ... surprising how often Verizon forgets to reroute things when the T1 is down so we just hook up the old Record-a-Call so clients don't get a dead ring.

Awl - basically an ice pick. Will punch a hole through anything. We use them to destroy old hard drives, open up a machine with stuck screws, and other uses.

Answer 60

Did anyone mention sharpies? You can write on anything with them!

Answer 61

I use this almost daily a good ol' leatherman tool for 12 years. (sorry can't post images )

It can cut cables, turn screws, cut cardboard, plastic and wood... Waaaay better than a swiss army knife IMHO.

Answer 62

dsh

Distributed SHell. With dsh you can perform a command via ssh on multiple servers. I find it quite handy when doing a lot of the same things on a serverfarm, at least when it's not complicated. For instance, doing a reboot of the whole farm is just 'dsh --all reboot'. I wouldn't recommend it for using it interactively. With dsh you can make lists of servers, like a list of all your webservers or all servers located in .uk, and only perform the action on that list.

ClusterSSH

Sort of the same, it spawns multiple ssh sessions and you can input in multiple sessions at the same time. It's extreme usefull when you've got interactive commands, like aptitude dist-upgrade. I found this very usefull when upgrading a serverfarm from etch to lenny.

Answer 63

tcping

From the author: "tcping.exe is a small (Win32) console application that operates similarly to 'ping', however it works over a tcp port."

It's one of the best (OSI Layer 3+) ways to determine whether a host is powered on and connected to the network.

Answer 64

HTOP as yet another TOP. For Windows; PortQueryUI as a replacement for NMAP.

Answer 65

Ncat (part of Nmap 5) is a great replacement for Netcat, OpenSSL's s_client and telnet. I used it recently to test and verify an HTTPS connection over IPv6.

Answer 66

I had a switch go down one night after a huge surge, I used some phone cable wire to bridge the connection. Hmm I still need to get that fuse...

Answer 67

A vaccum cleaner ! It's always better when you can work in clean environment or remove all the dust that jam the cooling system.

We go this one : alt text http://www.realself.com/files/imagecache/blog/dyson-vacuum-cleaner-25080.jpg

It looks ghostbusters enough so you don't have to be ashamed while working with !

Answer 68

Big fan of the camera phone over the last couple of years for documenting consoles/error lights on switches/servers. My new favorite tool to carry in my backpack is a FLIP camera. I find it very useful for documenting wire racks/trays, or remote maintenance closets that I need to have a 'view' of from my desk while on the phone with a tech. Plus, it is great to document when someone in the office goes nuts.

Answer 69

Windows PowerShell.

Until Windows PowerShell I always was envious about the powerful Linux shells.

Answer 70

What? Nobody mentioned the dippy bird in front of the photo-cell thingy that switches off the lights when you're not moving?

But ok, one thing that has been very useful to me until only a few years ago was a small coin (pick your nomination, mine was mostly a 1NOK coin simply because that's what I would have in my pocket, living in Norway). Because our Thinkpads had this nice screw for fixing the hard drive assembly that was designed with a head for just a coin like that. Works better than the Phillips or flathead screwdriver I have to use for newer models. Dammit, it was designed that way, and it was good. Thumb screws done right!

Oh, ok, and then there was the one time I wanted to extract the innards of a USB pendrive to put into an ASUS EEE and got (unplanned) help from my tumble dryer. Works sweet on that cheap glue holding the electronics in their plastic casings, better than the knife I tried at first ...

Answer 71

Another set of tools I am surprised isn't used more - or heard of more - is Performance Co-Pilot. This is an amazing performance monitoring tool, and is available on a number of UNIX platforms. It is a distributed performance monitoring tool with historical recording and history recall - and with an amazing set of rules for alarms and so forth.

The rule engine allows you to do things like: notify me if 80% of the disks in the system are busy 90% of the time over the last 5 minutes; notify me if disk space grows at a rate of more than 1Gb/min; notify me if all processors are more than 75% busy during work hours; notify me of swap space used is increasing more than 5% per second for the last minute.

The tool also allows you to run the programs not only in real-time - even against hosts across the network - but also over an archive file acting as if it was in real-time.

Answer 72

If you want a cheaper version of hotplug you just need a server with bonding and redundant power supplies =)

get your extension lead going to your new rack location and a network cable that will fill that length, plug them in then disconnect the localised rack power and network, move the server then plug back into the racks new network/power =)

Answer 73

PuTTy connection Manager! Tabbed Putty!

http://puttycm.free.fr/

alt text http://puttycm.free.fr/images/puttycm.JPG

Answer 74

DBAN

Excellent bootable CD (or floppy) for completely wiping the contents of a hard drive. I use it in two ways:

1. Zeroing out a hard drive before installing or reinstalling an OS.
2. Securely wiping a hard drive before shelving it or destroying it.

It's great, and it's easy to use. It runs in about an hour on most of the disks I have ever tried it on, it may require more time for terabyte-class drives, I'm not sure.

This is one example of a good tool that performs one function and performs it right.