Restart Elasticsearch cluster when adding new node

Question

I have an ELK stack for our logging needs. The stack is 1 Logstash server, 3 Elasticsearch, and 1 Kibana server. All have playbooks in Ansible to redeploy when needed.

When I go to redeploy the Elasticsearch I'm having a minor issue of needing to run sudo service elasticsearch restart to get it working again, and I would like for my playbook to run it start to finish. When I run the playbook it creates a temporary instance, configures that instance, saves the AMI, creates the launch config and attaches that to the scaling group. It then scales up from 1 to 3 instances, but I think the problem is the first instance isnt aware of the other two.

Is there a way to have Elasticsearch automatically restart when each Ubuntu instance boots up? That should fix the problem.

Use [UserData](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/deploying.applications.html) — Tim, Jul 18 '18 at 19:19
UserData is an EC2 feature, CloudFormation simply automates it. Try [this link](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html). — Tim, Jul 18 '18 at 22:23
Oh I see now, thanks! That still doesn't fix the issue I'm having with Elasticsearch unfortunately but good to know in the future. — Ryan Grush, Jul 19 '18 at 03:44

Andy Shinn · Answer 1 · 2018-07-23T03:06:55.700

Sounds like there are two issues here (automatically starting Elasticsearch and discovering other instances). I'll try to answer both.

Starting automatically

Assuming your build is based on a recent Linux distribution that has systemd available, you can install a system unit file to automatically start Elasticsearch. According to the docs at https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html#systemd it looks like the official RPM and APT packages include one. Assuming you installed from a package the system unit may actually already be there.

It is likely disabled and you can check:

andy@search-logs2:~$ sudo systemctl is-enabled elasticsearch
disabled

If this is the case, during your AMI build you would just need to run:

andy@search-logs2:~$ sudo systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

If you installed Elasticsearch another way and don't have a unit file, the official Ansible Elasticsearch role installs one. You could just copy theirs as an example: https://github.com/elastic/ansible-elasticsearch/blob/master/templates/systemd/elasticsearch.j2. You would then use Ansible or systemctl to enable the unit to start on boot.

Discovering other isntances

There are plugins for discovering instances in cloud providers like AWS. Check out the EC2 discovery plugin at https://www.elastic.co/guide/en/elasticsearch/plugins/6.3/discovery-ec2.html. By default, it will try and discover any other instances in the account. Limit by setting groups or host_type settings: https://www.elastic.co/guide/en/elasticsearch/plugins/6.3/_settings.html.

I have a similar setup but in Google Cloud and when adding an instance the new node starts automatically using the system unit file and joins the cluster a minute later using the GCP discovery plugin.

I had the discovery-ec2 plugin installed actually. Elasticsearch was running, I could goto http://my_address:9200 and see the json but nothing would work until I ssh'd into on of the instances and did a restart manually. I ended up just taking them off the ASG as I had seen recommended elsewhere and its working now. — Ryan Grush, Jul 24 '18 at 16:30
The logs should output what is going on. Have you taken a look at the log? I'm happy to amend the answer based on more information if you can provide log information. My guess next would be authentication failing because the instances don't have enough permissions to access the EC2 API to gather the information on other instances. — Andy Shinn, Jul 24 '18 at 16:34

score 0 · Answer 2 · answered Jul 18 '18 at 20:10

0

Create two instances as part of your ansible script and have them discover each other and then create the ami. This way the instance will think its part of a cluster and then when you load it up from the ami, it will look for other instances.

answered Jul 18 '18 at 20:10

stevo999999

127
1
3

score 0 · Answer 3 · answered Jul 19 '18 at 10:30

0

Maybe you can try to restart the elasticsearch service with a task.It would look something like that:

- name: Restart service elasticsearch
  service:
    name: "{{inventory_hostname}}_elasticsearch"
    state: restarted

hth

answered Jul 19 '18 at 10:30

AHT

166
1
7

1

the problem is once I save it as an AMI and launch config it is out of Ansible's control. I could make a standalone playbook to restart Elasticsearch but thats not really what I'm looking to do. – Ryan Grush Jul 19 '18 at 14:14
Sorry for not reading carefully, I think you can do that using monit – AHT Jul 19 '18 at 14:43

Restart Elasticsearch cluster when adding new node

3 Answers3

Starting automatically

Discovering other isntances