21

We have a 16-drive RAID-6 that has three problem drives. Two are already dead, and the third is giving SMART warnings. (Nevermind how it got in such a bad state.)

Obviously we want to replace the dead drives before the one that is still working, but is it better to:

  1. replace one dead drive, let the RAID rebuild, then replace the other, and let it rebuild again; or

  2. replace both drives at once and let it rebuild both in parallel?

To put it another way, will we get back to a state of redundancy faster by reintroducing one drive or two? Does rebuilding two drives in parallel slow the rebuild process?

In case it matters, the controller is a 3ware 9650SE-16ML.

Warren Young
  • 472
  • 1
  • 5
  • 19
  • 10
    Cross everything you got that can be crossed and send your favourite $deity a large donation! – user9517 Aug 30 '11 at 17:56
  • 1
    Can I just ask one question regarding this; can you let us know the EXACT make and model of disk in this array please - if my suspicions are correct you may very well see this question become a useful referal point for future users asking certain questions. Thank you. – Chopper3 Aug 30 '11 at 17:59
  • I don't know the exact model, and can't query the controller to find out, because someone else decided the machine should be turned off until the replacement drives get there. From data captured before then, I can say they're 1 TB Seagates with serials beginning with 9QJ. – Warren Young Aug 30 '11 at 18:08
  • Probably ST31000340AS's then, they'd be desktop disks then. Thanks. – Chopper3 Aug 30 '11 at 18:11
  • 8
    @Warren - AIEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! They turned it off? Bad JuJu my friend! It's too late this time, but generally speaking shutting the drives down (especially if they're older drives that have been up and running a long time) gives them an opportunity to throw errors on spinup (and cause the controller to say "Yeah, that drive is fried too now") – voretaq7 Aug 30 '11 at 18:14
  • Well, bonus on that: it means we can pull drives and check the type now: they're Barracuda ES.2 drives. – Warren Young Aug 30 '11 at 18:19
  • 2
    @voretaq7: I've once sent all the disks from an MSA-20 box to hell by shutting it down after ~3 years of uptime and continuous use. Will never do that again :-) – karatedog Aug 31 '11 at 00:53
  • 1
    The array is up and rebuilding now, so I can get exact models, for anyone who cares. The original hard drives were ST31000340NS, meaning they're the server-rated version of the one Chopper3 was asking about. So are these notorious for failing or something? (The new ones are ST31000524NS.) – Warren Young Sep 02 '11 at 18:51

4 Answers4

28

!!!!! ONE !!!!!

Do one at a time, seriously dude, don't think of doing this ANY other way ok.

Anything else will test your full system restoration skills.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • 3
    The two items I would add to this answer are (1) **PRAYER** (to whatever deity you like) and (2) **MONITORING** once you get everything back to a safe state (so you'll know when drives fail in the future and can address the problem before you have two and a half failures. Optionally you can also configure a hot spare in the array for the future. – voretaq7 Aug 30 '11 at 18:22
  • 3
    Or use RAID 10 – Chopper3 Aug 30 '11 at 18:24
  • 1
    **Re: prayer**, no comment. :) **Re: monitoring**, I've been advocating that for years; maybe this will light a fire under someone. **Re: RAID-10**, too much data in a bid market; when 3 TB disks came out, we didn't triple redundancy, we cut the number of disks by 1/3. Sigh. **Re: hot spares**, we do that now that drives are big enough to allow it, but this particular server was 16 drives in a 16 drive enclosure, when 1 TB drives were the biggest you could get, and we really needed all 14 available TB. Going to a 24-drive system wouldn't have worked; see previous. :) – Warren Young Aug 30 '11 at 18:29
  • It might not even be a good idea to replace anything yet as a rebuild might discover/trigger a media error (possibly on the SMART indicated disk). – pfo Aug 30 '11 at 18:41
  • 2
    If the drives have already failed, then there is no reason to keep them around -- rather, I'd expect that two consecutive rebuilds are more stressful for the other drives than a single one. – Simon Richter Aug 30 '11 at 23:26
  • 1
    +1, This. While two consecutive rebuilds add more stress and more likely to cause the third drive to fail before you finish both, it's also a faster rebuild, and if the marginal drive fails while rebuilding the 2nd disk, you still stay online. So the fastest, safest way to a fault-tolerant state is one at a time. – Joel Coel Aug 31 '11 at 03:21
  • 1
    I've upvoted this answer, which I agree is the most conservative approach. However, in principle, replacing two disks at once could be less stressful on the remaining drives: each RAID-6 stripe will have enough redundancy to reconstruct data for any two failed drives, and so replacing two at once could result in only a single scan of the surviving drives. Replacing one at a time guarantees two scans, but with increased safety sooner. All of this depends on implementation details, though, so I still recommend one drive at a time. – Dale Hagglund Aug 31 '11 at 03:26
  • 1
    Absolutely pointless answer: no reasoning, just "do that", and traditionally for QA sites bunch of upvotes. Seriously. – poige Jun 04 '19 at 07:43
14

Do you have good, recent backups? If not do you think you can get them in reasonable time?

I'd honestly be more concerned about tripping the bad drive offline during a rebuild than anything else - If you're already throwing SMART errors you're more than halfway there.

My suggestion would be to confirm your backups, then rebuild one drive at a time to try to recover to a state where you can replace the one throwing SMART errors (dead drives first, soft-errors last).

If you have no backups it's a crap shoot: Backing up may create enough soft errors to mark the marginal drive as failed, as may trying to do a rebuild.

voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • 2
    Most or all of the data on this array is a kind of cache, to avoid the need to pull terabytes of data repeatedly through a slow link. This cache data is all replaceable, either by downloading it again over months (once) or by shipping it to a site that can copy from another array. So, backups aren't the issue. What we're trying to prevent by saving the array is the days to weeks of downtime shipping the server to a service depot, re-populating the array, and shipping it back. – Warren Young Aug 30 '11 at 18:05
  • in that case, what @chopper3 said is pretty much The Law Of The Land: Rebuild one drive at a time and PRAY REALLY REALLY HARD that you don't trip the marginal drive offline with the extra read load. – voretaq7 Aug 30 '11 at 18:12
  • Phew - glad to hear it. – Chopper3 Aug 30 '11 at 18:19
1

I see no point in changing it as "one disk a time".

Obviously, if RAID is capable of "resilvering" both disks simultaneously (that are failed anyways) you only win allowing the whole RAID to regain its ability to sustain up to 2 failures faster.

poige
  • 9,171
  • 2
  • 24
  • 50
-1

My 0.02.$

Since the server is already offline, run ddrescue on the drive that is about to fail, to clone it to another, sane drive.

Then put the new, sane drive in the array instead. If the cloning is successful, you'll avoid the risk of seeing that drive fail during 2 rebuilds.

Guillaume A
  • 11