77

We have a couple of tower servers in a small server room. The carpet is wet as a result of the cooler and no-one else really seems concerned about this but I'm not too happy. I'm only a lowly developer, but I seem to be more concerned than the hardware guys!

Is this dangerous? What's the worst that could happen? My instinct says water + (electric * allOfOurData) = dangerous.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
JMK
  • 778
  • 1
  • 7
  • 19
  • 5
    If servers are racked and no cables are touching the floor you should be fine. Get your site team to look at the cooler and send an e-mail to the lead SysAdmin detailing your concerns to cover your ass and for a "I told you so" moment. Also...why carpet in a server room? – tombull89 Nov 04 '11 at 10:54
  • 30
    I think the carpet was put in to absorb the water... – JMK Nov 04 '11 at 10:57
  • 40
    @Heisenburg...you have **got** to be joking. – tombull89 Nov 04 '11 at 10:58
  • 3
    I'm honestly not! Thanks for the answers guys, going to go show this to management! – JMK Nov 04 '11 at 10:59
  • 21
    So, instead of solving the problem, there was *carpet* put in to *soak up* the water. The carpet where your servers are standing on. *I don't even...* – tombull89 Nov 04 '11 at 11:16
  • 1
    Also, are you sure it is water and not coolant fluid? – Simon Richter Nov 04 '11 at 11:44
  • 12
    If the carpet was put there to soak up the water, you need to update your resume. That place is a ticking time bomb. – MDMarra Nov 04 '11 at 14:53
  • 16
    Surely you need some hot air blowers in there to dry out the carpet. – dunxd Nov 04 '11 at 15:51
  • 3
    You could use a fishbowl to collect the condensation from the cooler like [these guys did](http://thedailywtf.com/Articles/A__0x26_quot_0x3b_Priceless_0x26_quot_0x3b__Server_Room_0x3a__Priceless.aspx) – Doug Harris Nov 04 '11 at 15:52
  • 2
    I would also be concerned about humidity in the room. You need to fix that air-con and put a dehumidifier in that room pronto to remove the water. – Yeodave Nov 04 '11 at 15:58
  • Sure a carpet in the serverroom can cause accidents, but mayhaps they want some accident to occur to the machines? I guess it's not their money on the line. Maybe they even collect some insurance money on the go? – Totumus Maximus Nov 04 '11 at 16:15
  • 3
    After reading this, just thought I would add a bit of general knowledge - Did anyone know, during World War 2, They worked on colossus in a room that was flooded due to a radiator leak... The technicians wore wellies as an insulator! – William Hilsum Nov 04 '11 at 16:41
  • 2
    So the towers get hot, the swamp cooler cools the towers, the cooler drips condensation, the condensation is absorbed by carpet. I like dunxd's suggestion to use hot air blowers to dry the carpet! But you might want to add a few more swamp coolers to cool the hot air blowers. Oh, wait... – artlung Nov 04 '11 at 17:30
  • 1
    All I have to say is: RUN DUDE!!! Don't think about it just run! – NitroxDM Nov 04 '11 at 22:32
  • 4
    In addition to the other concerns, I'd be worried about the nasty things that are growing in the carpet. Mold alone could be reason enough for OSHA to shut the place down. – Paperjam Nov 04 '11 at 22:52
  • 1
    The carpets are there to absorb electrons on dry days. – Bart Silverstrim Nov 04 '11 at 23:37
  • You laugh, but an ex girlfriend of mine who worked as a teacher once told me a story about their IT department. They put the new server (with all the migrated data) under a leaky aircon unit, didn't last very long! – ChrisFletcher Nov 05 '11 at 00:08
  • Is there a raised floor, and if so, what's under it? – Joe H. Nov 05 '11 at 13:04
  • 1
    I asked this question on a whim and it's had an incredible response! Thankyou all so much for the advice. – JMK Nov 05 '11 at 15:25
  • I think everyone here has come up with some great reasons to fix it. In order to get your management really interested in fixing it, you should write out a petition with a big list of the problems (from most dangerous to least) and pin it up on a noticeboard, with spaces below for people to write their names. Including a link to this page might help too. It shouldn't take long before the list is completely full of signatures and management can't ignore it. If they still do ignore it, contact the health and safety authority for your location. Not only is it dangerous, it's downright illegal. – Polynomial Nov 05 '11 at 17:18
  • Somebody might slip on it and crash into some hardware? – user712092 Nov 05 '11 at 19:22
  • @JoeH.: you *really* don't want to know *that*. – Massimo Nov 06 '11 at 16:35
  • 4
    You are in the UK. The much maligned Health and Safety at work act makes health and safety everyone's responsibility. This is a dangerous situation that should be reported to your H&S rep immediately. – user9517 Nov 06 '11 at 16:44

16 Answers16

68

Carpet is a big 'NO! NO!!' for a room hosting equipments that are of high value, because of the fire risk. Water is too, for obvious reasons. You should straight call maintenance immediately and have them repair the drainage system. The water could really cause problems, inform your superiors right away and draw their attention on the matter.

Community
  • 1
Gaumire
  • 825
  • 6
  • 12
  • 26
    "but... but... the carpet is water logged. So no fire hazard!" I can just hear someone thinking this is a jenius solution. – albiglan Nov 04 '11 at 14:24
  • 14
    don't forget the increased static (ESD) risk of carpet. – RyanTM Nov 04 '11 at 14:35
  • 6
    @Ryan not such a concern when the carpet is wet... – user229044 Nov 04 '11 at 14:36
  • 7
    Fire, ESD, Dust -- All sorts of reasons to avoid carpet in your server room. Sometimes though you're in an office where there's no room/budget to do a proper server room. Work one problem at a time though (and water is worse than carpeting - especially if you have both at the same time because now you have to think about mold too) – voretaq7 Nov 04 '11 at 14:58
  • 1
    ;) COnsider this room a hazard and in violation of laws. Server rooms run a lot of power. There are legal sides for environments with high electrical densities. A conductive floor is a NOT. Water on floor? ;) The building inspector may want to tell your employer no people are allowed there. – TomTom Nov 04 '11 at 15:33
  • 5
    If you can't sell it because of the safety hazard concerning water and electricity, consider this. A constantly wet carpet is a breeding ground for mold, stink, and much worse. Legionaries disease was tracked back to accumulated standing condensation from an air conditioner. I mean seriously, wet carpet anywhere? – Bill Nov 04 '11 at 16:11
  • 5
    Why is a BS answer so highly voted? We have flame-resistant, anti-static carpet installed over our raised floor. Normal household carpet would be a bad idea, but a blanket "NO CARPET" statement is BS. – Ward - Reinstate Monica Nov 04 '11 at 19:41
  • @Bill's concern is quite important and other considerations are also worth mentioning in the appropriate order. The biggest concern is health and safety of staff. Also, if this situation is located in the US, one of the most litigious places on Earth, the previous two concerns create doubly large and unnecessary financial gambles for the owners. –  Nov 05 '11 at 12:13
53

Nobody has mentioned "why" water in a server room is dangerous. It might be obvious, but just to put it out there...

Server rooms have a good deal of electrical equipment running on AC and DC current. Usually they are well grounded, but sometimes insulation breaks down, or a wire gets pinched. A person touching an electrical potential is protected by their skin or clothing (gloves). This provides a resistance that prevents a current running from the higher potential piece of equipment to ground. Adding water to this situation decreases the resistance allowing more current to pass through the body. (in both the AC and DC forms of Ohms law Current = Voltage / Resistance(Impedance))

If you are wet, a 60V potential difference can send enough current across your heart to kill you. Not 120V, not 240V, 60V. You don't need a main electrical cord to cause the short.

To keep beating the horse... You are likely legally permitted to refuse to enter the server room. In the United States, OSHA and the DOL regulations may apply to this situation. OSHA standard 1910.22(a)(2) is what you can reference. Not that it is a good idea to throw around OSHA regs, but they exist for a reason.

If you do need to work in that environment, then invest in some good PPE (gloves, shirt and shoes) to afford yourself some protection. But seriously, this is a bad situation and should be fixed.

Skyhawk
  • 14,149
  • 3
  • 52
  • 95
albiglan
  • 1,033
  • 8
  • 11
  • 5
    +1. Given the fuse size in some server rooms it may be ILLEGAL for a person to go into the room while the electricity is on (not even servers running - you will ahve to remove the fuses). You get into an area where legal regulations exist and ignore them. – TomTom Nov 04 '11 at 15:31
  • 1
    "Is this dangerous?" Anyone in the modern world who asks that about a room full of electric/electronic equipment sitting on damp carpet or above damp carpet needs to question their education. It's why we install GFCI sockets in bathrooms. Standing in a puddle of water and touching a computer case on a system that has a bad ground and an electrical fault is a potentially deadly situation. – Fiasco Labs Nov 07 '11 at 02:12
  • 2
    Death or physical injury trumps data loss any day of the week... – Fiasco Labs Nov 07 '11 at 02:20
48

If these machines are directly on the carpet (carpet in a server room ? seriously ?) then you might want to lift them off a few inches.

Any non-conductive material will do - wood, stone, plastic, foam, whatever.

In a pinch, put some cardboard under them.

adaptr
  • 16,479
  • 21
  • 33
  • 2
    No idea why this was down voted. In a pinch there's nothing wrong with this advice. They should be raised regardless, to be honest. – Dan Nov 04 '11 at 10:59
  • 1
    I think it's better to solve the root of the problem then to put the servers on a piece of wood. – duenni Nov 04 '11 at 11:07
  • 35
    Sure - when ? Tomorrow ? Next Week ? I would advise him to put them on some sort of elevated support RIGHT THE F NOW. – adaptr Nov 04 '11 at 11:24
  • 12
    +1 for realism - some people on this forum forget that there is a real world out there that everyone has to deal with, rather than some miraculous instant implemention of the perfect solution. – dunxd Nov 04 '11 at 12:35
  • 9
    What good does cardboard on a wet floor? Will it not just soak through? – jva Nov 04 '11 at 14:25
  • 3
    +1 -- I wouldn't recommend cardboard at all myself, but definitely get your servers at least a few inches off the floor - ESPECIALLY if it's carpeted. They will suck in a lot less dust and be much happier in the long run. – voretaq7 Nov 04 '11 at 14:55
  • Cardboard is pretty conductive when wet. – ceejayoz Nov 04 '11 at 14:56
  • 2
    @ceejayoz: I find that most things are pretty conductive when wet. Or, rather, that water is pretty conductive when wet. – Adam Robinson Nov 04 '11 at 15:29
  • Foam, stone, etc. won't soak the water up, though. – ceejayoz Nov 04 '11 at 15:33
  • @ceejayoz: True enough; just a joke ;) – Adam Robinson Nov 04 '11 at 15:35
  • okay, I can make batteries out of cardboard, wood and foam, if there are the right electrolytes in the water it's worse than just being conductive with a porous material. – ewanm89 Nov 07 '11 at 00:51
  • If they are sitting on wet carpet then get them remotely turned off first. Then shut off all the power at the wall before picking them up with your bare hands off the wet carpet. :) – hookenz Nov 07 '11 at 02:43
  • I would have upvoted, if not for the 'cardboard'. – Andrew Barber Nov 12 '11 at 03:28
29

We have a couple of tower servers in a small server room.

Put them on a table.

paulmelnikow
  • 453
  • 3
  • 9
20

You're 100% right to be worried, you need to inform your management and recommend that the servers be shut down and the room powered off until the problem is resolved. Anything else is negligence.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • 11
    Have you ever worked for a business ? You know, the kind that makes money ? – adaptr Nov 04 '11 at 12:36
  • 6
    @adaptr, note the of the word "reccomend". Not "demand". – tombull89 Nov 04 '11 at 12:51
  • 2
    I meant that planning things, getting approval, budgeting, getting experts in, and applying large(r) data center changes are often luxuries of time we simply don't have. If this is a problem NOW, he needs to deal with it NOW. Today. – adaptr Nov 04 '11 at 12:58
  • 8
    @adaptr Yes he has/does. Also the kind that doesn't have water flying all over their DC, nor carpet, and has redundant systems so if one has to be shutdown there's no disruption of service. Something about an ounce of planning and a pound of solution... – Chris S Nov 04 '11 at 13:01
  • 5
    I think this was more of a "How to avoid being fired when this blows up" answer: lodge protests now. – Sorpigal Nov 04 '11 at 13:03
  • 7
    @adaptr - Have you ever been fired (or worse sued) for professional negligence? You make the recommendation to the appropriate person(s), in writing, and keep a copy. If management ignores the recommendation and then comes after you when everything blows up you produce the copy. – voretaq7 Nov 04 '11 at 14:53
  • 2
    Youc an possibly also call the authorities / union. A server room runs a lot of electricity, there are legal regulations for worker safety. You may have to turn off power and get it fixed NOW by LAW. – TomTom Nov 04 '11 at 15:32
  • 3
    @adaptr Last I knew, Chopper has consulted on the creation of **many** large scale data centers across the world. If I had to pick only a single person whose advice I would take about data centers for the remainder of my career, it would be his. – MDMarra Nov 04 '11 at 19:18
  • This would be something to make a high-enough leader aware in a way that cannot possibly come back to the messenger. –  Nov 05 '11 at 12:18
  • @adaptr Sounds like the OP is very close to working for an ex-business. You know, the kind that **loses** money? – BCS Nov 07 '11 at 00:38
  • I think there is probably quite a list of regulatory authorities one could threaten to tell if something is not done instantly (off switch), as well as getting lawyers involved. – ewanm89 Nov 07 '11 at 00:55
  • 1
    In the UK you can be considered professionally negligent if you don't report Health and Safety problem that you notice. Once you've reported it to your H&S representative (which every company must have, if it has employees IIRC) then you can relax as it's no longer your responsibility/liability. – Mark Booth Nov 09 '11 at 15:31
13

$10 says that it's "leaking" because nobody is venting the AC unit to the outside - given the ad-hoc sound of this set up, I wouldn't be surprised if they hadn't installed the unit properly. That means, of course, that the room is likely also overheating on a regular basis.

Renee
  • 131
  • 2
  • Short term: Plumbing a drain to the unit perhaps. An indoor unit that is not optimized for cooling the air mass/temperature gradient is also likely to be inefficient.Cooling is a major cost. I've seen capital planning reviews on these size nanodatacenter closets costing way more than renting servers (CRAC/HVAC installation, power not counting servers costing $50-100K for merely 1-5 racks). –  Nov 05 '11 at 12:27
10

Turn everything off and go to the pub ... if asked simply state that you will begin taking things seriously when everybody else does.

Things to do:

  • Get rid of that carpet
  • Plinth for your kit (or rack would be even better if you can afford it)
  • Environmental monitoring (temperature, humidity etc or dehumidifier if you're on a budget)

But please for the love of all things binary do not leave this kit sailing towards the iceberg.

Moog
  • 201
  • 1
  • 4
9

Ummm. As you said Water and electric = BAD... in a server room = Dangerous.... I would soon rather turn everything off clean it up than someone get hurt.... Is cheaper in the long run.. Talk to the "IT manager" if not go higher... that is dangerous.

Zapto
  • 1,824
  • 6
  • 23
  • 39
7

Even if this is not a problem short term, the chances of this causing other problems like rusting cases and rotting floors and walls makes it a must fix. The first is a pain but the second could end up forcing you to completely vacate the room to fix.

BCS
  • 1,065
  • 2
  • 15
  • 24
  • +1 For mentioning corrosion which was my first thought. The water from an ac is basically "destilled" water, and not very conductive to electricity until ions from salt or metals are added, but the moisture will speed up corrosion a lot – Stein G. Strindhaug Nov 05 '11 at 12:02
  • The "not very conductive" bit is rendered somewhat moot as soon as it hits the floor as there will be plenty stuff added to it at that point. – BCS Nov 07 '11 at 00:29
  • If the carpet is over concrete, your distilled water assumption goes away pretty quickly. – Fiasco Labs Nov 07 '11 at 02:16
6

One word: mold

Now you not only have a safety issue (shock hazard) and a reliability issue (potential equipment failure), you have a health issue.

Document EVERYTHING, signed and dated, with pictures. First alert your immediate superior, and failing that, go up the chain of command. Make a record of every communication. If necessary, consult your company's legal counsel (if their attorney says they need to fix it, they might jump). If still nothing happens, go to OSHA or your local equivalent.

Then when they fire you for whistleblowing, take your mountain of documents to your attorney and sue their pants off for illegal termination of employment.

Jonathan J
  • 584
  • 1
  • 5
  • 11
5

WOW -- I can't believe this situation. I wouldn't even want water on the floor of the bathroom let alone an office or server room environment. Many of the risks/hazards have been addressed here but honestly, the "cooler" which I assume is an air conditioner, shouldn't have leaked condensate onto the floor in the first place. When the portable air conditioner is replaced, be sure to get one that has an automatic safety that cuts off the unit if the condensate is full or the condensate line is blocked. Then you avoid the water on the floor issue.

Tina Behnke
  • 71
  • 1
  • You turn the "floor is wet" into a "server room is very hot" issue. That's great when management take environmental issues seriously, but at this poor chap's place? They'll probably wait for the silicon to melt out the CPU cores before fixing the blocked condensate line. – MadHatter Nov 04 '11 at 16:13
  • Wait until this short circuits the high amps cabling that may be there. And someone dies. – TomTom Nov 04 '11 at 18:11
5

I was in a similar situation where the roof had a leak and there was water in the server room. There was a power distribution panel in the room. I talked to a person that used to work for the electric company. I asked him what would happen if I would have accidentally touched the power distribution panel. He said it probably wouldn't kill you but it would explode the arm that you touched it with and the opposite limb.

Greg Finzer
  • 163
  • 5
4

In my early career I visited a site where the cleaner used the server room to store her mop bucket. The cantankerous elderly lady cast me sideways evil glances for the next 2 weeks after I issued an ultimatum to the Managing Director of the site. I left site with guarantees a locked door would be installed and the cleaner would be forever banished from the server room.

Needless to say a year later I was back on site - no locked door - soaking wet server and the UPS had short-circuited. Needless to say the tape backup had not been changed for 8 months and the a lot of people lost a lot of work. At least the cleaner had somewhere convenient to store her bucket in the interim.

The moral is bad things usually happen when people don't give a cr*p. Your situation is so absurdly bad you need to keep on moving up the chain until you can find someone who gives one. Try the hardware guy's boss- same response - up a rung until you get to the top. If you still can't find anyone who cares you need to look for another job.

leonigmig
  • 317
  • 2
  • 4
  • 8
  • Your story sounds as a perfect example of the Broken windows theory (http://en.wikipedia.org/wiki/Broken_windows_theory) – jao Nov 05 '11 at 12:44
4

In addition to all of the other answers I'd be very concerned if there was a false floor in the equation, who knows whats under there and what state it's in. If your in a warm climate you might have all sorts of critters nesting next to your servers by this point.

I've heard of putting servers in the cloud, I don't think they meant raincloud... but seriously, start migrating the software functions out, if it's damp and has been for a while, you might not know whats gotten rusty and prone to failure. Backups would be a good idea too.

Antitribu
  • 1,709
  • 3
  • 23
  • 37
2

Water aside, carpet + movement = static. I wonder how long before some shoe-dragging sysadmin goes to replace a component and doesn't wear his ESD strap (if he even has one)?

Common sense dictates here -- if you think it's a problem, it probably is.

Garrett
  • 1,272
  • 10
  • 16
0

I know this may seem like a pretty obvious solution, but I'll mention it anyway: Install one of those old metal clawfoot tubs; leave a hair dryer and a toaster (include pop-tarts for style) balanced on the rim; then on the outside of the server room door post a sign that reads: "Executive Suite".

Oh yeah, and while you're at it, wring out your resume and hang it up to dry somewhere.

aculich
  • 3,520
  • 1
  • 25
  • 33