Referencing these two guides:
http://awsadvent.tumblr.com/post/105141640154/aws-advent-2014-high-availability-in-aws-with
https://blog.logentries.com/2014/12/keepalived-and-haproxy-in-aws-an-exploratory-guide/
Trying to get keepalived virtual_ipaddress working at a very basic level on a pair of AWS instances. They are deployed in the exact same zone in a VPC, and VRRP appears to be flowing between them, but they don't seem to be very ready to drop to BACKUP state!
Server A ("MASTER"):
Config:
vrrp_instance VI_1 {
debug 2
interface eth0
state MASTER
virtual_router_id 51
priority 101
unicast_src_ip 172.31.13.67
unicast_peer {
172.31.14.193
}
virtual_ipaddress {
172.31.100.100 dev eth0
}
}
Relevant /var/log/messages on Server A:
Aug 26 23:32:41 Server A systemd: Started LVS and VRRP High Availability Monitor.
Aug 26 23:32:41 Server A Keepalived_healthcheckers[1041]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 26 23:32:41 Server A Keepalived_healthcheckers[1041]: Configuration is using : 5139 Bytes
Aug 26 23:32:41 Server A Keepalived_vrrp[1042]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 26 23:32:41 Server A Keepalived_vrrp[1042]: Configuration is using : 60780 Bytes
Aug 26 23:32:41 Server A Keepalived_vrrp[1042]: Using LinkWatch kernel netlink reflector...
Aug 26 23:32:41 Server A Keepalived_vrrp[1042]: VRRP sockpool: [ifindex(2), proto(112), unicast(1), fd(10,11)]
Aug 26 23:32:41 Server A Keepalived_healthcheckers[1041]: Using LinkWatch kernel netlink reflector...
Aug 26 23:32:42 Server A Keepalived_vrrp[1042]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 26 23:32:43 Server A Keepalived_vrrp[1042]: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 26 23:32:43 Server A Keepalived_vrrp[1042]: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 26 23:32:43 Server A Keepalived_vrrp[1042]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.100.100
Aug 26 23:32:43 Server A Keepalived_healthcheckers[1041]: Netlink reflector reports IP 172.31.100.100 added
Aug 26 23:32:48 Server A Keepalived_vrrp[1042]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.100.100
tcpdump "ip proto 112":
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:36:48.823269 IP ip-172-31-13-67.us-west-2.compute.internal > ip-172-31-14-193.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20
23:36:49.824407 IP ip-172-31-13-67.us-west-2.compute.internal > ip-172-31-14-193.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20
23:36:50.825530 IP ip-172-31-13-67.us-west-2.compute.internal > ip-172-31-14-193.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20
23:36:51.826660 IP ip-172-31-13-67.us-west-2.compute.internal > ip-172-31-14-193.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20
23:36:52.827798 IP ip-172-31-13-67.us-west-2.compute.internal > ip-172-31-14-193.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20
Server B ("BACKUP"):
Config:
vrrp_instance VI_1 {
debug 2
interface eth0
state BACKUP
virtual_router_id 51
priority 100
unicast_src_ip 172.31.14.193
unicast_peer {
172.31.13.67
}
virtual_ipaddress {
172.31.100.100 dev eth0
}
}
Relevant /var/log/messages/ on Server B:
Aug 26 23:13:20 Server B systemd: Started LVS and VRRP High Availability Monitor.
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Netlink reflector reports IP 172.31.14.193 added
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Netlink reflector reports IP fe80::839:ffff:fe8f:db75 added
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Registering Kernel netlink reflector
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Registering Kernel netlink command channel
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Registering gratuitous ARP shared channel
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Netlink reflector reports IP 172.31.14.193 added
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Netlink reflector reports IP fe80::839:ffff:fe8f:db75 added
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Registering Kernel netlink reflector
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Registering Kernel netlink command channel
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Configuration is using : 60782 Bytes
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: Using LinkWatch kernel netlink reflector...
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 26 23:13:20 Server B Keepalived_vrrp[11498]: VRRP sockpool: [ifindex(2), proto(112), unicast(1), fd(10,11)]
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Configuration is using : 5141 Bytes
Aug 26 23:13:20 Server B Keepalived_healthcheckers[11497]: Using LinkWatch kernel netlink reflector...
Aug 26 23:13:23 Server B Keepalived_vrrp[11498]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 26 23:13:24 Server B Keepalived_vrrp[11498]: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 26 23:13:24 Server B Keepalived_vrrp[11498]: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 26 23:13:24 Server B Keepalived_vrrp[11498]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.100.100
Aug 26 23:13:24 Server B Keepalived_healthcheckers[11497]: Netlink reflector reports IP 172.31.100.100 added
Aug 26 23:13:29 Server B Keepalived_vrrp[11498]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.100.100
Aug 26 23:15:30 Server B kernel: device eth0 entered promiscuous mode
Aug 26 23:15:35 Server B kernel: device eth0 left promiscuous mode
tcpdump "ip proto 112":
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:37:03.060867 IP ip-172-31-14-193.us-west-2.compute.internal > ip-172-31-13-67.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
23:37:04.062044 IP ip-172-31-14-193.us-west-2.compute.internal > ip-172-31-13-67.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
23:37:05.063169 IP ip-172-31-14-193.us-west-2.compute.internal > ip-172-31-13-67.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
23:37:06.064375 IP ip-172-31-14-193.us-west-2.compute.internal > ip-172-31-13-67.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
23:37:07.065562 IP ip-172-31-14-193.us-west-2.compute.internal > ip-172-31-13-67.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
23:37:08.066790 IP ip-172-31-14-193.us-west-2.compute.internal > ip-172-31-13-67.us-west-2.compute.internal: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
I have a feeling I'm probably doing something very basic very wrong, but I've been through about a dozen iterations of config files and I'm now down to trying to do everything at the absolute most 'dumb' level. I want to restart the "MASTER" and have the "BACKUP" pick up on the change, but instead it enters the "FAULT" state, or randomly does absolutely nothing at all...
