Best sysadmin WTF?

Question

You sure witnessed it with your own eyes (or are going to) sooner or later: that dreadful project/system/situation where something got SO screwed up you just can't believe it actually went like it did.

Mismanagement? Misbudgeting? Misunderstanding? Just silly, plain ignorance? Name your cause, it sure happened (and keeps happening a lot, sadly; see here).

Describe it here, for amusement (although somewhat of the cynic kind) and learning (hopefully).

Some rules:

• This is not the place for random (even if utterly devastating) admin errors, so please avoid "argh I mistyped that rm -r" or "OMG I JUST COPIED THE CORRUPTED DATABASE OVER MY LAST GOOD BACKUP" (been there, done that); those things are better here. This is about "what sort of drugs was exactly under the influence of who designed/implemented this system?".
• One WTF per post, so they can get properly commented.
• Please post something you actually witnessed :-)
• If it was you who did it, it still qualifies :-)

I'll be adding some material soon, feel free to add your own; and please do :-)

Answer 1

I was called from a company I never heard of before, which had been tasked with implemented an Exchange 2003 mail server for a customer and had no clue at all about how to do it; nothing too strange, right? I work as a freelance consultant, so I'm perfectly fine doing jobs you don't know how to do for you (and getting your money for it).

So I went at the customer site, and discovered something quite strange: every single server in the network was a domain controller; all 15 or so of them.

Then I discovered something even stranger: no one of them was properly replicating with any other, Active Directory overall behaviour could only be described as "erratic", users were having about any network issue you can imagine and Exchange just refused to install with unknown-to-mankind errors.

So I had a look at the network configuration on the server, and I saw... it was using the ISP's public DNS servers. Then I look at another server... and it was the same. Then I look at a DC... same thing. Then I asked... and it was officially confirmed: each and every computer on the network (about 1500) was using the ISP's DNS instead of a rightful domain controller.

I proceeded in explaining DNS is quite critical for proper Active Directory operation, and was able to reconstruct the back story:

• Someone originally set up the AD domain correctly, using a DC as the DNS server for every computer.
• He/she/it didn't know anything about forwarders and/or firewall configuration, so computers were unable to resolve Internet public names.
• So came the idea of using the ISP's DNS servers on the computers; they configured it on everyone of them.
• They started having lots of "can't find a domain controller" error (who would have guessed?).
• They thought the problem was caused by not having enough DCs, so they proceeded to promote every single server to that role.
• Needless to say, this only worsened things, as those new DCs too used the wrong DNSs, so they were also unable to replicate.
• This went on for months, they just "got used" to the network being totally unreliable.
• On top of this, they try to launch Exchange setup, which crashed miserably; only then they decided to call some external consultant, and until them they had absolutely not clue something was totally, definitely wrong in their network setup.

Answer 2

Once upon a time I had a client that was a small business (10 people) with an electronic health record. (Not a medical doctor). I noted one day that the backups had been failing. Upon testing, the tape drive was not working at all. I mentioned this to the owner, who said he was well aware that the drive was bad, but was too expensive to replace.

Sure -- thats not very WTF.

The WTF is he had his staff rotating the tape daily, taking it to a safety deposit box, and all that jazz for the 6-9 months since it died.

"Don't tell the staff, it might worry them"

Answer 3

I was working as a sysadmin for a Big Government Agency (one of the main bodies of Italy's governement), and had been managing their data center for some months. One evening, my phone rings and my boss tells me something Very Bad is happening: total power outage.

Ok, we have UPSes, right?

Yes, but they won't last long, so better go there and shut down everything until power returns.

I go there, make my way through the dark corridors, arrive at the server room... and am greeted by what can only be described as pure hell. Literally. The room was so hot you could have baked cakes in it. UPS power was ok, but half of the servers had already shut down from overheating and the remaining ones were screaming in agony.

The reason?

Servers were on UPS power... air conditioning was not.

Answer 4

Email response from a Microsoft support engineer to a reported issue:

"As far as my opinion of your issue, I have one word: WEIRD."

Gold!

Answer 5

HP ProLiant ML370 G3 fan failure... The fan sensors on the motherboard of this model tended to go bad after 5 years. The server can't boot when the wrong combination of fans is not detected. I had to walk the customer through jumpstarting the machine with a shop-vac (to get the fans spinning at boot) and this is how they kept the server running until I could arrive with a new system.

Answer 6

I used to be an accounting software consultant for Dac-Easy accounting. One time I was called into the main office of a local business and was told by the accountant that if I couldn't resolve why the program was full of accounting errors each weekend, they would have to find another application and consultant. Going through the administrative log files, I discovered all of the entries were usually made on Friday or Saturday nights. I then found out that the owner's wife was logging in to the accounting system computer from home using PC Anywhere and trying to balance the accounts with her checkbook after several glasses of wine. Once the numbers looked good, she'd log off.

Answer 7

I lifted a computer I received from our parent company's office when I heard something rattling around inside. When I opened the case, I found it 1/2 full with Frosted Mini-Wheats. I figure a mouse was living in it or using a food cache. Likely entry point was the gap between the case and the DIN keyboard jack.

Not exactly what you asked for, but a definite WTF.

Answer 8

Another customer, another horror story.

In the main post I talked about having erroneously overwritten a good backup with the corrupted database it was meant to replace; it happens :-(

So a restore from backup was neeeded. Luckily, there actually WAS a backup there: it was done daily, on a central backup server with a Really Big tape library attached to it; this server managed backups for the whole company, it was Really Expensive and had a Real Backup Software installed on it.

So far, so good. We look up the backup job, load the proper tape, start the restore operation, the tape is loaded, the restore starts... and just nothing happens.

We try again, some thing.

We unload, reload, reboot, try restoring previous backups... nothing changes.

We assume some long operation was going on, and leave it on the whole night... next day, still nothing changes.

Ok, time to call the Real Backup Software vendor's support... but it can't be done, we are on Sunday. We try look up the vendor's support site, but a special access code is required, and only one manager has it... the same manager which will be Really upset on discovering the system is still down on Monday, when he comes to work.

Another day of pain, and I discover the bug is well-known, and it has been fixed by a vendor patch, which (obviously) nobody bother applying. So I go to apply it... but it can't be done: management doesn't want to risk breaking anything unless the vendor confirms the patch can be safely applied; the fact that the backup server wasn't able to restore anything apparently didn't look enough "breaked" to them.

Only after four total days, various support calls and the vendor sending a support engineer onsite we were finally able to apply the patch and restore the backup; the backup server hadn't EVER been able to do restores, but nobody ever tested it, so nobody noticed.

Answer 9

Terminal Emulation For Fun And Profit

I work with an older system -- essentially a bunch of text terminals which connect to a Unix box, with a few Windows computers thrown in just to complicate things.

A few essential background facts

• Primary software application uses it's own extended termcap file. System-wide termcap and terminfo are ignored.
• Primary software application allows various keyboard mappings to be selected according to environment variables.
• All text terminals have programmable keyboards.
• Windows PC's connect using proprietary client software with no built-in support by primary software application.
• Proprietary client software allows various keyboard mappings to be selected on load.
• Only half the staff use standard QWERTY layout.
• At least one, possibly more, of the system administrators who have worked on this system have lacked a proper understanding of terminal emulation.

I think you can see where I'm going with this.

There'd been half-hearted attempts at adding proper terminal support to the terminfo and termcap files, but these are only partially functional. The proprietary termcap file used by the primary system application works, but it's mostly irrelevant since the $TERM is never set properly anyway.

Logging in, each user has to basically select where they're logging in from and what keyboard layout they want to use -- there's no autodetection involved. This will set $TERM to ANSI, regardless of which terminal is being used on the client side, and an environment variable set so the primary software application remaps some of the keys appropriately.

In the case of people using the embedded keyboard layout on the text terminals, a script is then run to reprogram some of the keys to the embedded layout (as physically printed on the keyboard itself), but not all of them. On the Windows PC's, a client keymap is loaded up to remap most of the function keys to ANSI. Can't remap all of them, or else some of the other keys will stop working.

Logging into anything else from any of the terminals is an exercise in failure, and can only reasonably be done directly on the server itself. Since everything works, it's difficult to justify the time and effort required to fix it. But adding anything newer than the 20-year old text terminals we're stuck with now would essentially require a rewrite of the entire system.

I cry a little inside every time I see it.

Answer 10

I knew someone who decided to reorganize the files on their computer by putting all the .exe's in one folder.

Answer 11

A network with ~60 (SIXTY) PCs.

A security-fanatic boss.

Some new switches with VLAN capabilities.

A "network reorganization plan" involving ~20 (TWENTY) VLANs.

Thanks to some unknown upper power, I left before all of this could get actually started...

Answer 12

Oh, this is an easy one...

I had a Linux system installation for a company situated in a warehouse in rural Kentucky. This system is the accounting/inventory application for the organization. I was replacing a 1998-era SCO server with a new CentOS 4 server. As I result, I expected the worst when it came to connectivity and clients. You know, old PCs, bad cabling, etc. I shipped new switches, cables, thin clients, etc. to the site...

However, I did NOT expect the wiring "closet" to be in the BATHROOM! Actually, the cabling was located in the space above the dropped ceiling directly above the warehouse bathroom toilet. This included the DSL modem, 10-megabit HUB and a 66-block for the phones...

Answer 13

We gave one of our Unix admins a Windows box. This was shortly after we had given Windows admins Linux boxes to admin in an attempt to increase our familiarity across different operating systems. I as a Windows admin trying not to screw up my Linux box I had asked a ton of questions of the Unix team. I expect this was the objective of the exercise. After a stern lecture about making sure I sized all my volumes right and not putting everything on one volume, or on a single partition, I went forth and built a perfectly adequate box, that is still running in production two years later.

When the Unix guy who had given me the lecture about different volumes and not building things so that a full disk could crash the OS built his Windows box, he put everything on C:. When I said "I can't back that up with any policy we have and when those logs fill your box will crash." He said "Well I figured it was Windows, everything on C: makes sense." He didn't make another partition at all. I was seriously boggling at this. Absolutely WTF were you thinking sort of thing. Apparently it didn't occur to him that some best practices are the same regardless of OS.

Answer 14

Massimo's reply reminds me of another WTF...

A small office has Windows 2003 server with Exchange, Symantec AV, etc. The company usually uses us to do all their IT work.. but this new hotshot is in the office and bought a new PC. We had them on a unified Dell Optiplex platform, this guy decided to replace 3 of them with "great machines" that were on sale at Best Buy.

Regardless -- they somehow manage to join it to the domain.

First call:

We can't find outlook. The machine came with Word (Microsoft Works), so where is everything else.

Second call:

How do we use the network antivirus?

This is the wtf. I went onsite, and 7 of their machines had been configured as Symantec Antivirus SERVERS, instead of installing the client. Apparently the gentleman didnt like that we had locked the settings of the client, and decided to "reinstall" the client so he could change when it scanned. Apparently the first CD for Symantec AV that we had left out somehow persuaded him to install the "server", rather than the client.

Answer 15

A client called me to help with some server builds. During the build I was informed that these were replacement servers for the bottom two from the rack. I thought it was strange that the rack position was relevant and asked why. Turns out the server room flooded after the water pipe that runs through the middle of it burst.

So what is a water pipe doing in the middle of the server room you ask? Well, the ladies bathroom is next door and the server room is conveniently out of sight. What better place to put the main feed to the bathroom?

Answer 16

Same customer involved in the DNS From Hell thing.

Their Exchange server kept running for years gracefully and even survived some disk failures (thanks, RAID, and thanks more, backups). One day, I was called again for an unrelated problem, and they told me en passant something was troubling them: some times before, they discovered everyone was able to open everyone else's mailbox; they thought it was strange, but didn't do anything, because they don't know how to fix that, and anyway e-mail was working and (almost) no one had disccovered this.

I was quite shocked here: everyone was able to open any one else's mailbox, and they thought this was ONLY A MINOR NUISANCE and didn't bother doing anything for THREE MONTHS. Any ordinary people would have yanked the network cord from the server immediately, but not them.

The source of the problem was... strange: someone had set "Everyone/Full Control" at the Exchange Organization level in Active Directory, so everyone immediately became a full Exchange administrator; their only luck was that very few people noticed it.

As of today, nobody knows yet who actually did that.

Answer 17

Worked as a full-time programmer and part-time admin for a small company making doctor's office accounting (read: customer billing and insurance claims) software. Got a call from a big customer that their server was down; it was the new product running Altos Unix SysV/386. Since I was all they had, they sent me and the new gal.

Got there, and there was a room full of data entry operators typing in WordPerfect documents as fast as they could. There was a long-haul line going out of state where another set of operators were doing the same.

The documents directory had about 10,000 word perfect documents, all of them following the same format. ALL IN ONE DIRECTORY. So-and-so has been injured and suffers from such-and-such. Seems that this doctor was the go-to guy for lawyers filing injury lawsuits.

I tried to boot off of floppy and got only lots of seeking noises. I pulled the floppy out and it was completely covered with cobwebs, I'm talking handfuls here. It was a doctor's office, so I demanded swabs and alcohol (it was the only Unix boot disk we had, and the only chance of fixing things on that service call), and got started cleaning up the floppy, drive and server.

There were two problems, one was the server was full of dirt from the carpeted office and its corner-of-the-room location (on thick plush pile carpet), and was overheating in the 80 degree office. The filesystem needed serious FSCK'ing and I divvied up the 10K files into a new set of alphabetical subdirectories and suggested they break it down further.

The doctor was a perfect ***hole who ripped us a new one when we came in the door. When we left we explained that buying the WordPerfect for UNIX option from us didn't mean we had sized the server for intracontinental document creation; also, he needed an A/C, a carpet-free server location, a UPS, an external RAID and a better directory filing protocol. Suddenly he was all ears as this was how he made all his money. I gave all the information to Sales but left that job soon afterwards for other reasons.

Answer 18

My best WTF originated at my current employer, when I first started here.

Pretty much the first couple months on the job involved a constant state of panic and outright horror as I discovered the state of the servers. At my current level of experience, I might have just recommended setting fire to the place and collecting the insurance. But the very best part was one company policy:

There was no accounting for dialup customers. At an ISP that got half its income from dialup customers. This had actually been a policy that had been in place for some 6 years by the time I started, and it just boggled my mind that nothing had been done about it. The standard line was "we just trust our customers not to use too much." The fact that the place was running on a shoestring budget and didn't have a real admin at the switch for all that time didn't help matters I don't think.

The end result of this policy of course, was that spammers had basically commandeered the dialup pool. Nearly every night, they would use one account to dial into multiple modems until the whole pool was full, then blast out spam as fast as they could. The version of AAA RADIUS didn't help matters in that there was no quick "allow only one user at a time" switch at the time. I hacked something together in Perl to disconnect anyone with two logins until such time as the RADIUS server could be replaced, and that fixed that problem sometime in the first week or two. Part of the other problem was that I wasn't allowed to have enough time to fix things (see: shoestring budget) since I was also doing tech support at the same time. But part of the reason I'm still here was that I had free reign over it all. I don't think I've ever really been told not to go ahead and implement some sound idea. The only restriction I've ever had was money.

Answer 19

Ten years ago, I interviewed for a company whose primary server room's cabling looked like spaghetti soup. The networking equipment was in 3 racks near one corner of a 20ft x 60ft room. There was a cascade of network cables of all sorts of colors tumbling down the racks and ending in a pool of tangled cables about 7ft wide and spilling out about 5 ft from the racks. There wasn't any documentation and no one knew what went where.

The company had a 95% market share in a field that dealt with safety of life and property. All I can say is that that can give a real pause.

BTW the company is still in business and has apparently cleaned up a lot of things since then.

Answer 20

Was in a Demarc closet not so long ago and found this sweet setup on a site:

Elite fiber cable install= TWIST TIE

alt text http://locobox.googlepages.com/Fiber.jpgalt text http://locobox.googlepages.com/Fiber2.jpg

Answer 21

A few years ago, the company I was working for acquired one of our competitors and I had to audit their machines (Centos) to ensure that they would meet our standards in terms of configuration (their sysadmin had declined the opportunity to join our company in favour of starting his own company that developed turn-key solutions).

In /etc/rc.local (the home of all great WTFs) I found:

swapon /dev/sda3

and

route add -net 10.0.0.0 netmask 255.255.0.0 eth1

Note: eth1 was configured in ifcfg-eth1 with a netmask of 255.255.255.0

Apparently real sysadmins don't use those cliched config files like /etc/fstab and /etc/sysconfig/network-scripts/*

Answer 22

Earlier this year we hired a new girl. I was asked to make sure that all the company fonts were installed on her computer. I knew they were but decided to pick my battles and just install them again. I got a call from the girl about 5 minutes afterwards and she wanted me to come over and show her how to use the new "software" I installed. I walked over, opened Word and walked her through how to change fonts....which is what I installed.

This girl had a PHD!! She no longer works here but not because she didn't know how to use fonts.