0

I'm trying to build up two redundant forward-facing load balancers (using either IPVS or NGINX), but first I'm trying to get the floating VRRP virtual-ip/mac to work properly before moving forward through the process.

I've got a stock Ubuntu 16.04 VM on VMware vSphere 6 with the latest updates. VM is on a DvSwitch portgroup with promiscuous mode, mac address change, and forged transmits enabled. VM is using VMXNET3 for NIC. I'm using the standard keepalived in the repo.

keepalived/xenial-updates,now 1:1.2.19-1ubuntu0.1 amd64 [installed]

I'm attempting to use the following configuration for keepalived...

root@lb1:~# cat /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 150
    priority 150
    use_vmac vrrp150
        vmac_xmit_base
    advert_int 1
    virtual_ipaddress {
        10.0.4.55
    }
}

When initially set up, the pings toward the macvlan interface (vrrp150) are ARP replied with the parent interface MAC (which is bad for several reasons). I've attempted to use multitudes of different combinations of net.ipv4.conf settings laid out online, all of them seem to break ARP completely for the macvlan interface. iptables and ufw are completely disabled to my knowledge, and tcpdump shows the following...

root@lb1:~# tcpdump -s0 -i vrrp150
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vrrp150, link-type EN10MB (Ethernet), capture size 262144 bytes
10:05:19.271979 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46
10:05:20.215301 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46
10:05:21.215474 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46
10:05:22.219300 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46
10:05:23.215514 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46
10:05:24.223971 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46
10:05:25.224262 ARP, Request who-has 10.0.4.55 tell 10.0.4.31, length 46

So the ARP requests are indeed getting to the correct interface. I've attempted to use promiscuous mode on the physical interface, but it makes no difference (and the ARP requests get there regardless).

How can I get the keepalived virtual macvlan interface to respond to ARPs properly with the virtual MAC address so that traffic can be forwarded? I am trying to get the same virtual IP/MAC combination to follow between MASTER > BACKUP during transitions so that ARP tables are not affected on any upstream firewalls (GARP is not reliable/honored, and failover must be quick and as seamless as possible).

Jason Keller
  • 31
  • 1
  • 6

1 Answers1

3

I have solved my own issue...for the curious, here's the blow-by-blow.

First, make sure to carefully check your sysctls, as Ubuntu has some enabled by default you would not expect to be enabled by default, namely RP checks.

sysctl -a | grep net.ipv4.conf.*

You might be surprised at what you find in here.

On Ubuntu 16.04, set the following in /etc/sysctl.conf...

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.ens192.rp_filter = 0
net.ipv4.conf.vrrp150.rp_filter = 0

Due to the RP defaults being set in sysctl, you'll need to do it on the 'all' level as well as each interface (as shown here).

Perform the following to activate it live (or just reboot).

sysctl -p

Second thing is to make sure your DvSwitch settings are correct (again, as above, the port group needs to have promiscuous mode, mac address change, and forged transmits enabled). It need not be the same port group as the rest of your VMs, even if the VLAN they reside in is the same. This helps you keep maximum security on your other VMs, while only exposing additional traffic to VMs that absolutely need it.

If they're on the same host, even if they're in different VM portgroups on the DvSwitch, that traffic will be switched locally on the vSwitch within the host, never egressing an uplink port.

For the truly paranoid, you can spin up a completely different DvSwitch with different uplinks (if you have many to burn) and a separate VLAN for them (pruned on the physical switch). This ensures they are utterly containerized and will see no other traffic except the traffic they are meant to see.

If you are using default multicasting with keepalived and have multiple uplink ports on the DvSwitch they are assigned to (which is best practice), make sure to set...

Advanced Settings > Net > Net.ReversePathFwdCheckPromisc to 1 on each host

to prevent multicast traffic looping back to the host (similar to https://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting).

This can be omitted if using unicast_peer in your keepalived configuration.

Jason Keller
  • 31
  • 1
  • 6