I have an existing AWS infrastructure which is integrated with DataDog, which is responsible for monitoring various metrics, e.g. SQS queues, ELB, etc.
I'd like to set up a health check for some web sites and APIs. As far as I know, it's possible to do via AWS ELB health checks. However, I'd like to emulate the end-user experience so that the request is sent from the outside world and proceeds via ELB and to the application. Also, not all of the applications currently have ELBs. I've decided to use DataDog's HTTP checks. The question is, should I have a separate EC2 instance just to install the agent on? I certainly don't want to install the agent on the same machine as the Web Site and ping it since it would miss various network issues.
I've also considered Route 53 health checks which would be monitored by DataDog but I don't think it will be fast enough since the communication between DataDog and AWS is usually delayed in comparison to DataDog's agent reporting.
Update: for now I've decided to go with Route 53 Health Checks and CloudWatch alarms. DataDog is responsible for sending notifications when a certain alarm is triggered. As expected, there is some delay between the alarm and DataDog's reaction, but it turned out to be acceptable.
For deeper and more serious analysis I've also considered New Relic and Application Insights. Both of them seem to provide the needed health checks, though New Relic is quite expensive and Application Insights integrate better with Azue.