hadoop-config.sh in bin/ and libexec/

Question

While setting up hadoop, I found that hadoop-config.sh script is present in two directories, bin/ and libexec/. Both the files are identical. While looking onto scripts, I found that if hadoop-config.sh is present in libexec, then it gets executed. If not then the one under bin/ is called. What's the purpose of keeping the same file in two directories?

Answer 1

Thanks for pointing this out.I tried in my system

[root@prashant bin]# hadoop version
Hadoop 1.2.1

and you are absolutely correct

20:39:57.038706 open("/usr/local/hadoop/libexec/../conf/hadoop-env.sh", O_RDONLY) = 3 <0.000253>

When I moved this file then its referring to file inside /bin

20:42:51.024234 open("/usr/local/hadoop/bin/../conf/hadoop-env.sh", O_RDONLY) = 3 <0.000332>

Then I configured the latest version of hadoop

[root@hadoop libexec]# hadoop version
Hadoop 2.4.1

[root@hadoop libexec]# pwd
/usr/local/hadoop/libexec

[root@hadoop libexec]# ls -l hadoop-config.sh
-rwxr-xr-x 1 67974 users 9793 Jun 21 11:35 hadoop-config.sh


[root@hadoop bin]# pwd
/usr/local/hadoop/bin

[root@hadoop bin]# ls -l hadoop-config.sh
ls: cannot access hadoop-config.sh: No such file or directory

So as far I see most of the script is moved out from bin in the latest version but still I dont have answer to your question

Answer 2

Historical accident? Developers building out things just to make them work? There hasn't been a huge amount of effort in the past in making sure the operational shell scripts used to start up all the java programs are well thought out and designed in a functional way.

Allen Wittenauer has been undertaking a monumental task of consolidating and fixing some of these issues in HADOOP-9902. You may wish to watch that JIRA issue.

If you're using a release from Apache, you'll probably seen the changes coming "soon". If you're using one of the commercial releases, I would expect for them to get the changes much later; at the very least, they'll probably wait until Allen's work is fully committed.