ElasticSearch enforce master shard allocation & Improve cross DC write speed

Question

I currently run three Elasticsearch nodes. Each one in a different data center (EU, US, AP) as an AWS EC2 instance.

There are 2 replicas per each shard for each index. Writes go only to the EU node.

When you write to the EU, your write might take:
~ 400ms (if replication is sync and master shard is in the US or AP)
Your write gets redirected to the node with master shard, then synced synchronously,

~ 200ms (if replication is async and maste shard is in the US or AP)
Your write gets redirected to the with master shard, then it's synced asynchronously,

~ x ms (if replication is async and master shard is in the EU)
You write isn't redirected and your app does not wait for sync, Replications it's asynchronous.

I want always the latter to happen. I want to have all primary shards in the EU.

I didn't find a way to enforce all primary shards to be in the EU. I've thought that making AP and US just data nodes would solve the problem. As a result of that I've set up the cluster of 3 master+data nodes in the EU, then AP and US became just data nodes. I can still see that they're master for some of the shards of a healthy index.

What's the solution? Can I enforce a node or zone to be responsible for master shards only? I've found this: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/index-modules-allocation.html

I've though I could use exclude.tag to stop allowing shards allocation on the US and AP nodes, but my understanding is that my replica shards wouldn't be allocated either. That's not the point. I want to have replicas on these nodes.

How can I achieve it?

score 0 · Accepted Answer · answered Jun 20 '16 at 06:48

I didn't find any better way than just enforcing manually on which nodes we would like to cancel master shards. This is the command.

curl -s -POST 'http://localhost:9200/_cluster/reroute' -d '{
"commands": [
{
  "cancel": {
    "index": "your_index_name",
    "shard": 0,
    "node": "your_node_host",
    "allow_primary": true    
  }
}]}'

I had to run that command multiple times, on each node I didn't want to have a master shard, for each index and for each index shard.

Your node still should have the replica shard.

ElasticSearch enforce master shard allocation & Improve cross DC write speed

1 Answers1