122

Our building is located approx. 100 meters from the explosive charges. They happen several times per day, and really shake the entire building a lot. This is going to go on for many days and the blasts are supposed to get stronger.

Our server rooms are nothing fancy; one of them has all the racks on hard concrete while the other one has a raised floor (the one which allows the cables to go beneath it).

Does anyone have any tips, countermeasures or best practices for us?

Currently we are thinking of the following countermeasures:

  • Daily report of the server rooms status lights (HD lights, power supplies and so on).
  • Nightly check disk scan on the most important servers
  • Order in extra supply of spare harddrives

Edit: Many good answers here! However one needs to be accepted. The highest voted answer at the time of this edit will get their answer accepted.

Chris Dale
  • 1,553
  • 2
  • 12
  • 22
  • 84
    Make sure you've got good backups! – Shane Madden Sep 19 '11 at 19:07
  • 49
    Ask if you can borrow a couple sticks of dynamite, handy to have around for the zombie apocalypse... – Ward - Reinstate Monica Sep 19 '11 at 19:19
  • 37
    Ask if you can 'push the button' for one of the charges! – RobW Sep 19 '11 at 19:26
  • 18
    Use Howitzer in combination with anti-tank missiles. – xpda Sep 19 '11 at 20:47
  • 31
    Deploy some Smooth Jazz. – Andrew Sep 19 '11 at 23:12
  • 2
    I really don't think you'll find any "best practice" for dealing with this, other than to stay under cover. As for countermeasures, consider using a fire hose on them. – John Gardeniers Sep 20 '11 at 04:20
  • 2
    I'd suggest if you have to take special procedures to counter a known threat to your equipment you should consider double checking your arrangements for the unknown threats. Dynamite doesn't hold a candle to an earthquake or flood. That said, some spare hardware is always a good idea regardless. – Sirex Sep 20 '11 at 06:59
  • Deploy chaff and flares...turn ECM on – EMBEDONIX Sep 20 '11 at 13:36
  • 1
    sabotage their gear during the night, with some elaborate pranking. i advise moving this thread to diy.SE or pranks.SE. actually: create pranks.SE! – JoséNunoFerreira Sep 20 '11 at 15:00
  • Maybe you could absorb those shockwaves by putting those server on some rubber stuff. I wonder what is the frequency of a blast shockwave. Ask XKCD :p – jokoon Sep 20 '11 at 15:16
  • 16
    Until I read the first answer, I really, honestly thought you were asking what the best weapons were. – Trevoke Sep 20 '11 at 15:42
  • 1
    Drawbridge up, snipers on the walls. – XTL Mar 20 '12 at 14:13
  • 1
    Counter-mine; molten lead and baskets of stones on the battlements; stand by to repel scaling ladders; Greek fire for the siege engines; repair wall breaches overnight. – user207421 Jul 31 '12 at 23:57

13 Answers13

127

If I were you, I'd call the company's insurance company, and have them place out an accelerometer.

This way - the insurance company will know that you're not the one shaking the disks, and the insurance company will know for sure when every blast has gone off, in case your SAN dies at the exact same moment.

(We did this.)

pauska
  • 19,532
  • 4
  • 55
  • 75
  • 15
    It's not clear from the OP where he is, but when blasting is taking place close to other buildings, the contractor is sometimes required to monitor the accelerations felt in the other buildings, i.e. by placing an accelerometer in them. – Ward - Reinstate Monica Sep 19 '11 at 19:23
  • 1
    When you did it, did the insurance company pay for any damages? – Phil Sep 19 '11 at 20:48
  • 1
    We did not get any damages afaik. – pauska Sep 19 '11 at 22:47
94

Now would be the time to make sure you've verified your backup solution. All the replacement hardware in the world won't save you if your backups are corrupt or have otherwise been rendered useless.

squillman
  • 37,618
  • 10
  • 90
  • 145
  • 8
    You don't know the quality of the backup unless you test the restore. Just don't check the logs to ensure backups are complete - verify that you can restore. – Raj More Sep 20 '11 at 15:11
  • 2
    I suspect that he meant that when he said "verified your backup solution" but I guess it's good to say it explicitly. :-) – pc1oad1etter Sep 20 '11 at 15:49
52

In addition to all the other excellent suggestions (particularly off-site backups) you should consider dust proofiing your room to the extent practical: Weather stripping around the doors, tape around the windows, etc.
If you have external air intakes plan on changing the filters when the blasting is over.

All that said, I wouldn't waste time/resources on nightly disk scans, or money on spare hard drives (maybe buy one or two for your most critical machines, but remember that you can overnight drives if needed).
Server equipment is surprisingly robust, and will continue to operate for years under some pretty awful conditions. Chances are you'll be fine.

voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • 12
    I have to +1 you simply for *Server equipment is surprisingly robust*. We had builders on a site across the road and they were hammering the ground for about 6 weeks and our entire building was shaking and everything (to the point things would get shock-vibrated from your desk onto the floor). I was panicking, but it was a total non-event. This question would have been nice back then :-) – Ben Pilbrow Sep 19 '11 at 20:07
  • 2
    Computers in general will take a pretty severe beating (think about how many people drop laptops every day that keep working) - these things aren't as fragile as our vacuum-tube-and-ferrite-core history has lead us to believe (though they should still be treated well, lest they die at inopportune times) – voretaq7 Sep 19 '11 at 21:27
  • 4
    Silicon Valley is one of the most earthquake-prone areas in the world. Stuff that works there _will_ survive a few shakes. – MSalters Sep 20 '11 at 11:24
  • Pro tip: Don't trust anyone if they say that they have dust-proofed your server room. Never. Ever. – pauska Sep 28 '12 at 00:55
  • @pauska agreed - "dust proofed" is a pretty vague term (and even if they say they met the relevant standard (in the US that'd be MIL-STD-810G 510.5) assume they're lying, because properly dust-proofing a ROOM is a pretty huge undertaking with airlocks, positive pressure, two-stage HEPA filters, etc.) – voretaq7 Sep 28 '12 at 02:23
27

Mount your server racks on ISO-Base platforms. This will isolate your server rack from the ground shocks and allow your systems to function even in the event of movement.

It's not easy, but is effective. Especially in earthquake zones.

To see the ISO-Base in action, check out this video showing the effects in a server rack during an earthquake simulation. One rack is mounted on an ISO-Base, the second rack is not.

Stefan Lasiewski
  • 22,949
  • 38
  • 129
  • 184
  • 2
    Bit late for that now. But interesting idea. – hookenz Sep 19 '11 at 21:49
  • 2
    I'd think it a bit of overkill for short-term blasting personally, but it looks like a cool system. – voretaq7 Sep 19 '11 at 22:29
  • The problem's the same whether it's short-term local shocks, or long-term low-incidence earthquakes. There's a fair case that since you know 100% these shocks are coming, and don't know 100% that an earthquake will hit, that it's a better investment right now than it would be for earthquakes, even in prone areas. – Chris Thorpe Sep 20 '11 at 08:10
  • I know it is too late for buying new stuff, but you could get yourself a good ammount of anti-vibrations rubbered screws for your hard drives. They are cheap. – Mister Smith Sep 20 '11 at 13:27
18

Power: If you lost power, will all of the servers, storage, san (etc) in your rack reboot on their own? Or will they come up only after you press the 'On' button?

This for two reasons:

  • If the power is unstable for a couple of hours, you may want to leave your servers down until things stablize - less chance of a bouncing server being taken out by a mis-timed power spike.
  • When was the last time you did the math on the amperage required to reboot all devices in the rack at once? Do you have the amperage to handle it?

You might also suggest to your users to turn off, and unplug computers each evening.

RobW
  • 2,766
  • 1
  • 17
  • 22
  • 10
    Three little letters: U P S (and I don't mean the shipping company). Well-conducted blasting is unlikely to knock out power, but there's still no excuse for not having good, stable power for everyday problems... – voretaq7 Sep 19 '11 at 19:56
  • 1
    Ha! Right, and I agree with the use of UPS and other line conditioners. But, Karrax noted that the blasting "...shakes the entire building...", and will get worse. I dont know what you mean by 'well-conducted', but if my building is shaking, I'd imagine my 30 year (or 1 year) old substation half a block down the street is shaking too... – RobW Sep 19 '11 at 20:50
  • "well-conducted" as in "directed charges and blast netting so flying rock doesn't go through a transformer" - No promises about loose connections, though if your substation cant handle some shakes your utility company may need a "firm talking to" about routine maintenance :-) – voretaq7 Sep 19 '11 at 21:25
13

Most people talk about backup - not many talk about the restore. Make sure that restoring from backup is as easy as possible. In particular if time is critical.

If you have a co-location replication, switch those to be the masters (if timing permits) since the ones in your building are much more vulnerable. They are also more likely to go down during work hours (evidently).

I also advise working from home.

evan
  • 233
  • 2
  • 7
  • 2
    Very, very good point. I've seen many 'backup plans' failing because they don't include an effective 'restore plan'. Two days of business blackout can mean huge fees to pay to your customers. The restore procedure should be considered from an economical point of view and integrated into the emergency plan. – gd1 Sep 20 '11 at 07:49
12

Play heavy metal to replace the hard rock as it explodes. Hard rock is so passé anyway. ;)

More seriously, maybe there's a DJ gear shop around the corner where you can pick up some shock absorpbtion/isolation stuff. Many (dance music) DJ's have a similarly hard time when they're playing records (i.e. keeping a tiny needle in a tiny spinning groove, somewhat comparable to sensitive equipment like disk drives) at extreme volume while the crowds around them are jumping and stamping on the floor in sync (the "in sync" is important; if you have never heard of the Tacoma Narrows Bridge incident, you should consider watching this clip). Rip a few trash bags open to provide additional protection against dust.

Sjoerd
  • 223
  • 1
  • 5
  • 2
    I was going to suggest this. The most clubs just use concrete garden slaps sitting on foam. Simple but very effective. – Iain Holder Sep 20 '11 at 11:02
9

I'm surprised no-one has mentioned the process side of things.

This is the perfect opportunity to go over what you have planned for business continuity. What is the plan if you have to move offices for a day or a week? Do you have up-to-date plans on drawings and which systems have priority for restoration? Is management briefed that you do have a plan and aware of

The acceleration from the blasts causing havoc on the server room is probably the least of your worries. Your utilities could be at much higher risk unless you are self sufficient with on-site power and robust connectivity (assuming you are not self-contained and only supporting local staff).

If there is a water main or power or internet access failure, can you survive that? Have you called your internet provider to see if they are aware of the blasting and have prepped to restore service through an alternate route if your utilities are interrupted. You'll know your specifics better than we can guess, but you should have a list of everything you need to function and addressing "What if this goes away unexpectedly?" for each.

Just going over this in your head / on paper will help you know if you have any weaknesses that need work later and perhaps communicate this up the chain if your organization doesn't have anything written up. Start with a two page, executive summary - just a FYI so that everyone knows what you're doing.

Yes - getting a few extra hard drives / spare parts on hand is good, but I would be more worried about the things I can't see or don't directly control.

The real benefit of this process exercise is a reality check for your current monitoring system. Once you've planned out some basic scenarios, you'll be better prepared for the unexpected. Having a short summary of what you expect to survive and what you don't will come in very handy no matter why you suffer an outage, and also assist in driving your efforts to improve monitoring 24/7 rather than when the foundation starts shaking.

bmike
  • 283
  • 1
  • 19
9

I used to run computers on a mine site that was being reclaimed. As the reclamation got closer and closer to the office, the data room would shake almost daily at the end of the day.

My counter measures included moving the racks on top of some hard rubber feet designed for vibration dampening. They were made by Manson and I just estimated the weight of my rack and bought the appropriate feet. This seemed to fix any movement within the rack. The tough part was lifting the rack gently enough to slide the feet under.

Also, as other had mentioned, I had backups that I tested and kept off site.

Those servers lasted 3 years with constant shaking an no hardware related crashes. The desktops in the office did not fair so well as 2 of them had catastrophic hard drive failures.

Jeremy Stein
  • 307
  • 1
  • 5
  • 14
kaptk2
  • 366
  • 2
  • 11
8

I have not seen this mentioned about backups yet, but make sure you take them off site! You may also want to make a backup of switch, firewall, etc. configurations.

jftuga
  • 5,572
  • 4
  • 39
  • 50
5

Ask the engineering company doing the blasting. Odds are very good they have their own gear onsite or have dealt with blast damage to a neighboring server room in the past.

Freiheit
  • 201
  • 1
  • 2
  • 15
5

Reread the BOFH chronicles. It will give you some practical ideas on how to make the explosions stop. Everything else will just be a countermeasure.

As far as countermeasures go, everyone else has mentioned the backups, so I'll avoid repeating that good advice.

Check your raid 5 arrays daily, and have spare disks available now. All that vibration can't be good for the spindles, and it's amazing how often save-able arrays are lost due to an alarm that's heard by nobody and a second spindle failure.

Perhaps I'm optimistic, but the engineers doing the blasting typically don't target vibration levels for "destroy the neighbor's foundation" energy levels, so I'd be most worried about the moving parts of the computers, aka the disks.

NotMe
  • 3,772
  • 7
  • 30
  • 43
Edwin Buck
  • 257
  • 1
  • 4
2

Put some protection beneath the racks, specially the one that is in hard concrete. It'll (or probably is) shake(ing) a lot. If you can't buy (specially because of the time) buy some earthquake-resistante floor, put some foam underneath it, to reduce the shaking.

woliveirajr
  • 166
  • 2
  • 9