6

I would like to know about your strategies on what to do when one of the Hadoop server disk fails.

Let's say, I have multiple (>15) Hadoop servers and 1 namenode, and one from 6 disks on slaves stops working, disks are connected via SAS. I don't care about retrieving data from this disk, but for general strategies for keeping cluster running.

What do you do?

wlk
  • 1,643
  • 3
  • 14
  • 19

2 Answers2

3

We deployed hadoop. You can specify replication numbers for files. How many times a file gets replicated. Hadoop has a single point of failure on the namenode. If you are worried about disks going out, increase replication to 3 or more.

Then if a disk goes bad, it's very simple. Throw it out and reformat. Hadoop will adjust automatically. In fact as soon as a disk goes out, it will start rebalancing files to maintain the replication numbers.

I am not sure why you have such a large bounty. You said you don't care to retrieve data. Hadoop only has a single point of failure on the name node. All other nodes are expendable.

Amala
  • 529
  • 2
  • 14
  • Well, I am a different guy (one person asked, and a completely unrelated person put up the bounty months later, as they have the same issue). I know the data is replicated 3x, so nothing is lost, but losing a disk seems to be much worse than it should be for us. New team who inherited a poorly documented system. Our node tends to go nuts (load over 200) when they lose a disk, which seems unreasonable. I am hoping that a better config will fix that. – Ronald Pottol Sep 01 '10 at 06:42
  • Are you interested in keeping a node alive after a disk goes bad? I agree the node should gracefully die instead of going nuts with a load over 200. So is this only a Hadoop bug of why doesn't a node die when a disk goes out? Hadoop's architecture is simple, if a node dies, take it out. Add another one later if you like or put the same one back in when it is fixed. – Amala Sep 01 '10 at 13:38
  • Yes, after all, they have something like 6+ disks for hadoop, each on it's own file system, so loosing a disk just means some data needs to be replicated from the other two copies, but no reason for the node and the other 5 disks of data to go off line. – Ronald Pottol Sep 01 '10 at 17:30
  • A datanode shouldn't die with the loss of a single drive. I suspect that somebody did something silly on your machine like striped the disks. Hadoop is much better off if you mount all drives independently with no RAID whatsoever. In this situation, losing a drive causes re-replication, but doesn't take out the datanode. – Ted Dunning Dec 05 '10 at 20:22
  • No, no raid (well, the OS is on raid, but not the hadoop data). – Ronald Pottol Dec 22 '10 at 00:40
3

You mentioned this system was inherited (possibly not up to date) and that the load shoots up indicating a possible infinite loop. Does this bug report describe your situation?

https://issues.apache.org/jira/browse/HDFS-466

If so, it's been reported as fixed in the latest HDFS 0.21.0 (just released last week):

http://hadoop.apache.org/hdfs/docs/current/releasenotes.html

Disclaimer: To my disappointment I have yet to have the need to use Hadoop/HDFS :)

Rob Olmos
  • 2,220
  • 1
  • 15
  • 25