1

I'm having weird issues with the Hadoop Namenode and Secondary Namenode. Our HDFS cluster runs smoothly most of the time. But every now and then, either the Primary Namenode freezes (crashing the whole cluster) or the Secondary Namenode freezes and stops making checkpoints.

Whenever this happens I try to restart the hanging service which fails because the port is still blocked:

# service hadoop-namenode restart
 * Stopping Hadoop namenode: 
no namenode to stop
 * Starting Hadoop namenode: 
starting namenode, logging to /var/log/hadoop/hadoop-hdfs-namenode-HOST.out
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 26100; nested exception is: 
        java.net.BindException: Address already in use

However, checking the output of ps auxw, no Namenode is running anymore. Checking which process is blocking the port it get:

# netstat -tulpen | grep 26100
tcp        0      0 0.0.0.0:26100           0.0.0.0:*               LISTEN      6001       20067       -

which isn't helpful at all. It says, the port is in use, but the process is -.

ss isn't any more helpful either:

# ss -apne | grep 26100
tcp    LISTEN     34     50                     *:26100                 *:*      uid:6001 ino:20067 sk:000015c1 <->
tcp    CLOSE-WAIT 224    0              127.0.0.1:26100         127.0.0.1:56770  ino:0 sk:00000527 -->
...
tcp    CLOSE-WAIT 13     0              127.0.0.1:26100         127.0.0.1:56762  ino:0 sk:0000078f -->
tcp    CLOSE-WAIT 17     0              127.0.0.1:26100         127.0.0.1:56772  ino:0 sk:000007b1 -->

The only thing both commands tell me is that the process is owned by UID 6001 which is the hdfs user. Checking ps auxw for any processes owned by hdfs I can see that there is one zombie process:

hdfs      4947  4.8  0.0      0     0 ?        Zl   Feb23 435:50 [java] <defunct>

So for any reason, restarting the Namenode service leaves a zombie which continues blocking the port. Unfortunately, there is no way I can get rid of this process because the only parent process is init:

# pstree -ps 4947
init(1)───java(4947)───{java}(9957)

The only workaround is either restarting the operating system (out of question) or temporarily using another port of the Namenode (or Secondary Namenode).

Do you have any idea what the reason for this weird bug may be? I couldn't find any hints in dmesg.

The cluster has 130 nodes, each running Ubuntu 14.04 Trusty with Kernel 4.2.0-30-generic #35~14.04.1-Ubuntu. Hadoop version is 2.7.1.

Janek Bevendorff
  • 186
  • 6

0 Answers0