Sometimes my NodeBalancer takes a node out of rotation and I would like to find out about it so I can reboot it. How do I do this?
This is the background:
I have two websites of the same type, which both run CPU intensive applications. One of the websites runs off a single VPS, which has a load of approx 3000 executions a day (each of which takes anywhere from 5-50 seconds) and for the other website I've installed a NodeBalancer with 9 nodes underneath. The each execute approx 40-60% of what the single VPS does.
This all works fine actually with hardly any disruption, but once a week or so, a node stops responding to the NodeBalancer and is taken out of rotation. This usually happens in combination with very high CPU usage. Now - this never happens at the single VPS (which has run without disruption or reboot for a year now).
So like I said - on the load balanced nodes I do have disruptions (although I run the same scripts and software for 99%) and I would like to find out when the node is taken out of rotation, so I can reboot it and get it up and running again.
Currently my workaround is to act on the mails I get from Linode, which alert me to high CPU usage. In some cases I then do a manual reboot if the node indeed became inactive.