How to balance elasticsearch (java) on a very powerful machine?

Question

On a 16 core XEON class, 128GB RAM RHEL server I want to deploy elasticsearch.

What is preferable performance-wise?

1. Have a huge elasticsearch process to use all resources on the native host?
2. Break the host to e.g. 4 equal virtual machines (KVM) and deploy an elasticsearch cluster with an elasticsearch instance on each VM.
3. Create docker containers on the native host and deploy elasticsearch cluster on them.

Thank you!

Answer 1

Option 4: Run multiple intsances/nodes on the same machine.

This is like options 2 and 3 except it's simpler because there's no virtualization or containerization. It's all running natively on the host.

Here's a list of caveats and recommendations when doing so (from here).

• Max heap size for each node instance should be < 32Gb. This is because a heap size above 32Gb will actually be counterproductive for the JVM will stop compressing pointers.
• Leave 50% of the memory to file system cache for Lucene.
• While you may have enough RAM to run multiple instances on the same machine, test to see if there is enough CPU/processing power.
• You will also want to check to make sure that the multiple node instances are not competing for disk space or disk IO. Our recommendation is to give all the nodes on the machine a raid0 with lots of disks underneath, or a dedicated disk per node.
• Lower processors setting accordingly. Each ES node detects the # of cores available on the machine (not aware of other nodes present). With multiple nodes on the same machine, each node can think that it has dedicated access to all cores on the machine (this can be problematic for the default thread pool sizes are derived from this). So you will want to explicitly specify the # of cores available via the processors setting so that it does not end up overallocating the thread pools. For example, roughly # of cores / # of nodes can be a good start to configure for each node.
• Keep in mind that multiple nodes also means that network connections, OS file descriptors, mmap file limits, will also be shared between the nodes so you will want to make sure that there is enough bandwidth and the limits are set high enough to accommodate the nodes.
• The more nodes you have on the machine, the more nodes will fail at once if a single server goes down. Also, you will want to make sure that you don’t end up with all copies of a shard on the same machine. You can prevent this by settingcluster.routing.allocation.same_shard.host to true. See here for details.

• To ensure cluster stability, each dedicated master node instance should be on its own machine (certainly can be a much smaller machine, eg. 4Gb of RAM to start with) -Keep in mind that multiple nodes on a machine means additional complexity in management (eg. keeping track of different ports, config files, etc..). A good way to manage the configuration for multiple instances is to create a separate elasticsearch.yml file per instance, eg. you can pass in the -Des.config parameter to specify the yml file for each instance on startup:

  $ bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.1.yml
  $ bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.2.yml
  

• Each yml will point to the same cluster name.
• It will be helpful to specify meaningful node names
• Use explicit port numbers for each node so that they are predictable (eg. http.port and transport.tcp.port).
• Each node should have its own path.* directories (eg. path.data, path.log, path.work, path.plugins) so the nodes will not end up having conflicting folder locations for data, plugins, logs, etc..

As mentioned in another answer, you don't want to use more than 32GB per instance, and you also don't want to use all your RAM for Java heap. Rather, it's better to leave at least 50% of it to the OS for filesystem caching.

There's a really good explanation for why this is the case in this blog article from Elastic.

Answer 2

2 and 3 are the best options because docs suggest a 32GB of maximum assigned memory for each node.