How does hadoop decide what its nodes hostnames are?

Question

Currently the urls generated by the jobtracker & namenode return either hostnames like bubbles.local or just bubbles. These end up not resolving unless the client machine has specified these in their /etc/hosts file.

When I run the hostname command on these machines it returns a hostname complete with the domain (E.G bubbles.example.com)

Running a small java test on these machines

InetAddress addr = InetAddress.getLocalHost();
byte[] ipAddr = addr.getAddress();
String hostname = addr.getHostName();
System.out.println(hostname);

Produces output just like the hostname command.

Where else could hadoop be grabbing a hostname to use in its jobtracker / namenode UI?

This is occurring in clusters with Hadoop 1.0.3 and 1.0.4-SNAPSHOT from early august. The machines are running CentOS release 5.8 (Final).

The generated URLs I'm referring to are like this

http://example:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/

or http://example.local:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/

score 3 · Accepted Answer · answered Sep 15 '12 at 06:31

I managed to get the WebUI to use the full hostname by doing the following.

On each slave node and the master node the /etc/hosts file was modified to be something like this:

127.0.0.1 machine1.example.com machine1 localhost
#These 4 lines appear on all of the machines
1.2.3.3 machine1.example.com machine1
1.2.3.4 machine2.example.com machine2
1.2.3.5 machine3.example.com machine3
1.2.3.6 machine4.example.com machine4

Now when I click on URLs in the task tracker or the hdfs web tool they contain the nodes full hostname and resolve.

score 0 · Answer 2 · answered Sep 04 '12 at 17:48

0

Where else could hadoop be grabbing a hostname to use in its jobtracker / namenode UI?

conf/core-site.xml

<property>
    <name>fs.default.name</name>
    <value>hdfs://hmaster90:9000</value>
    <description>
    The name of the default file system. Either the literal string
    "local" or a host:port for NDFS.
    </description>
</property>

conf/mapred-site.xml

<property>
  <name>mapred.job.tracker</name>
  <value>hmaster90:9001</value>
</property>

answered Sep 04 '12 at 17:48

quanta

50,327
19
152
213

Those are HDFS protocol URLs. Is there something similar for the http urls, E.G `http://datanode:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/` – Dan R Sep 04 '12 at 18:27

How does hadoop decide what its nodes hostnames are?

2 Answers2