we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,,
spark.dynamicAllocation.initialExecutors = ?
spark.dynamicAllocation.minExecutors = ?
spark.dynamicAllocation.maxExecutors = ?
I guess we cant give exactly the values so it will help to find the range for this parameters
for example
we have ambari cluster with
3 master machines
21 worker machine ( each worker machine have 32G memory & 16 CPU CORE )
3 kafka machines
please help me to calculate the values for the parameters ( I will happy to get then range of the value for each parameters since I know we never shut the value exactly )
I just give here my opinion ( based on the site - https://community.microstrategy.com/s/article/Best-Practices-for-Spark-Thrift-Server-on-YARN
can we set both parameters to 21 ( as number of workers machines ) ?? , is it OK?
spark.dynamicAllocation.initialExecutors = 21
spark.dynamicAllocation.minExecutors = 21
spark.dynamicAllocation.maxExecutors = 21
