Moving the SecondaryName Node in a Cloudera HBase Cluster

Question

I deployed the secondary namenode on the same machine is my main namenode:

This is wrong for performance and durability reasons (the secondary name node isn't a hot spare, but it does have a copy of needed metadata). I have found documentation on how to move the namenode, but nothing about moving the secondary namenode.

Does anyone with experience know how to do this safely?

Is there any reason why you can't just remove the Secondary role and then add it back again on a different host? You will be temporarily without a secondary namenode, but that shouldn't matter too much. — growse, Dec 11 '14 at 17:21
@growse: No idea - if I could answer that I wouldn't be asking random people on the internet how to manage my HBase cluster ;-) — Kyle Brandt, Dec 11 '14 at 17:42

score 4 · Answer 1 · answered Dec 11 '14 at 17:45

It should be safe enough to simply remove the Secondary Namenode role, and then add it back again onto a different node on the cluster. In the intermediate period, you may see a warning from Cloudera Manager that the role doesn't exist (which may cause long term issues with the namenode), but not having a Secondary doesn't put your data any more at risk.

score 3 · Answer 2 · answered Dec 11 '14 at 18:11

The job of the 2NN is to read the changes on the HDFS filesystem and add them to the fsimage. This decreases NN startup time as during startup the NN reads the fsimage file and then applies all the interim log changes on top of it. The naming is somewhat unfortunate as it is indeed not a standby/backup NN but only a utility to increase NN performance.

In CM there's an option to "Roll Edits" (depending on your CM version) on the 2NN, see also "Checkpointing". Make sure to do that before moving.
To be safe stop all services
Move the 2NN role to a new machine.
Restart all services
(Optional but worth it): Implement HA

Moving the SecondaryName Node in a Cloudera HBase Cluster

2 Answers2