How can I reboot server when HTTP stop responding on AWS EC2

Question

I have a server on Amazon EC2 and I would like to reboot it whenever it stops responding for HTTP requests. It is a single micro instance.

I was thinking about using AWS Lambda but I could not find any scripts (preferably in Python). I also tried to use Route 53 healthcheck but it is impossible to link it to alarm with reboot EC2 (because EC2 actions are not available on healthchecks alarms).

Thanks

Rather than automating the reboot you should probably fix your application. My linux server is only ever rebooted to get the latest kernel, which is actually fairly regularly, but other than that I've never needed to reboot it. My instances have been stable for years. — Tim, Oct 23 '18 at 18:14

score 2 · Answer 1 · answered Oct 23 '18 at 10:21

2

If the instance stops responding to HTTP it will probably stop being "healthy" and will show up as such in CloudWatch -> Metrics -> EC2 -> Per Instance Metrics -> i-1234abcd...

Then find StatusCheckFailed and StatusCheckFailed_Instance and StatusCheckFailed_System and see if they show when the instance stops responding. One of them should. Alternatively find some other usable metric, maybe in Route53 namespace.

Once you find a suitable metric create an Alarm by clicking the Graphed Metrics and then the little "bell" on the right.

In the next dialog click +EC2 Action and select Reboot Instance. You may need to tune some other parameters, that will probably take a couple of iterations.

Done :)

Hope that helps!

answered Oct 23 '18 at 10:21

MLu

23,798
5
54
81

thanks, but unfortunately status check was just ok, but VPN and Apache service stopped responding... I tried to find a way how to do a custom status check but it seems impossible. For Route53 metrics there is unfortunately no possibility to create alarm which will reboot machine. – Koss645 Oct 23 '18 at 13:46
If this happens often enough that someone needs to automate it I doubt it's an instance problem I wonder if a custom health check could make this approach work. – Tim Oct 23 '18 at 18:13
I do not automate it because it happens often but rather because response time is shorter and for a sake of good in case I will be on vacation or have a day off (it is my personal server where there is no one else than me watching it). – Koss645 Oct 24 '18 at 03:36

score 1 · Accepted Answer · answered Oct 23 '18 at 14:48

I solved it myself, I wrote an lambda function in Python and run it every hour by event scheduler in AWS CloudWatch

import json
from botocore.vendored import requests
import boto3
import time

region = 'xx-xxxx-x'
instances = ['x-xxxxxxxxxxxx']
website = 'https://website.com/'
webstring = 'SearchText'

def lambda_handler(event, context):
    for i in range(0,3):
        if check_website():
            return 'Website OK'
        time.sleep(60)
    reboot_instance()
    return 'Restarted instances'


def check_website():
    r = requests.get(website)
    if webstring in r.text:
        return True
    else:
        return False

def reboot_instance():
    ec2 = boto3.client('ec2', region_name=region)
    ec2.reboot_instances(InstanceIds=instances)

I have published tutorial - https://www.enigma14.eu/martin/blog/2018/10/24/aws-lambda-reboot-instance-when-http-unreachable-for-free/ — Koss645, Oct 24 '18 at 07:26

score 0 · Answer 3 · answered Oct 23 '18 at 09:35

You need to use the AWS API. One way could be to use the boto

Or you could use something more high-level like the Ansible EC_instance module

You could link this to monitoring events in many ways, from a simple cron job to something event-based like a Node-Red instance or something in between like an IFTT trigger.

How can I reboot server when HTTP stop responding on AWS EC2

3 Answers3