i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to install HADOOP in cluster using windows
Asked
Active
Viewed 120 times
0
-
Hi @SadimNadeem, could you provide some more info on what you are trying to do, what you now have, and what issues you are seeing? You might find it useful to check this help article on questions: https://serverfault.com/help/how-to-ask. Your question right now is a little light on details and research, and it would help get an answer if you could provide more information. – iwaseatenbyagrue Mar 14 '17 at 08:41
-
You do not need YARN to run a Spark Cluster. Try setting up a standalone cluster. – OneCricketeer Aug 30 '17 at 08:22
1 Answers
0
how to install hadoop2.4.1
That in itself is a separate, broader question. Read the Hadoop Cluster installation documentation.
I would recommend getting a single node, then pseudo-distributed, and then you can setup a cluster.
With Spark 2.0.0
Head to the Spark download page, choose the option for "user provided Hadoop"
For the most part, Spark will just work with minimal editing. It's Hadoop that needs configured.
Ensure that
HADOOP_CONF_DIRorYARN_CONF_DIRpoints to the directory which contains the (client side) configuration files for the Hadoop cluster.
You can edit spark-env.sh and spark-defaults.conf if needed.
If you are having a specific issue, please edit your question
Or you can setup an Ambari Server, and it'll setup and install everything for you, post SSH key management.
OneCricketeer
- 170
- 1
- 9