keepalived - Virtual is not routing my requests to real servers

Question

I'm playing around with keepalived and running that setup on my master node:

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass XXXXXXXXXXXXXXXXXXXXXXXXX
    }
    virtual_ipaddress {
        192.168.178.200/32 dev eth0
        192.168.178.201/32 dev eth0
    }
}

virtual_server 192.168.178.201 8443 {
    delay_loop 10
    protocol TCP
    lb_algo rr
    lb_kind DR

    real_server 192.168.178.210 8443 {
        weight 1
        TCP_CHECK {
          connect_timeout 5
        }
    }

    real_server 192.168.178.211 8443 {
        weight 1
        TCP_CHECK {
          connect_timeout 5
        }
    }

    real_server 192.168.178.212 8443 {
        weight 1
        TCP_CHECK {
          connect_timeout 5
        }
    }
}

On the slave nodes I have the same setup with lower priority and state BACKUP.

For test reasons 2 of 3 nodes are offline and I would expect that the VIP (.201) balances the traffic to 192.168.178.211:8443. (.212 and 213 is offline, .211 is online) I'm faced to the behaviour that 192.168.178.201:8443 is only available if the MASTER node is the same machine as the available backend machine.

In my case that means, that 192.168.178.211:8443 is only reachable if 192.168.178.211 is the MASTER node. If i.e. 192.168.178.212 is the MASTER node the traffic is not routed to 192.168.178.211:8443.

sysctl.conf:

#keepalived
net.ipv4.ip_nonlocal_bind=1
net.ipv4.ip_forward=1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.rp_filter = 2

Who can help me on that?

Have you set up the real server correctly for direct routing? I.e. the real servers need the VIP address configured, but ignore ARP requests for that VIP (unless the server is actually the load balancer). See e.g. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/virtual_server_administration/s1-lvs-direct-vsa . Alternative that might work: on each real server: `iptables -t nat -A PREROUTING -p tcp -d 192.168.178.201 -m multiport --dport 8443 -j REDIRECT` — wurtel, Jul 20 '18 at 08:06
Indeed the loadbalancer nodes are the same nodes as the real server nodes. `iptables -t nat -A PREROUTING -p tcp -d 192.168.178.201 -m multiport --dport 8443 -j REDIRECT` doesn't change the behaviour. I added my `sysctl.conf` in the opening post. I also tried it with a dummy interface like this: http://gayangunarathne.blogspot.com/2015/06/lvs-setup-in-oracle-virtualbox.html — Tom, Jul 20 '18 at 15:12
I also tried that http://gcharriere.com/blog/?p=339&cpage=1 without any luck :( — Tom, Jul 21 '18 at 17:44

score 1 · Answer 1 · answered Feb 24 '20 at 16:15

I understand from you question and the comments that the load balancers and the real servers are the same machines.

According to the Redhat documentation:

Accessing the virtual IP from the load balancers or one of the real servers is not supported. Likewise, configuring a load balancer on the same machines as a real server is not supported.

However, this is still possible according to the LVS Knowledge Base, but requires a bit more configuration effort.

Based on your example, let's take a 3 nodes setup without track script with one virtual IP address for keepalived-2.0.19 on CentOS 7:

virtual IP address: 192.168.178.201
node 1: 192.168.178.210 with priority 150
node 2: 192.168.178.211 with priority 100
node 3: 192.168.178.212 with priority 50

Then a possible configuration for keepalived is:

On node 1

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    virtual_ipaddress {
        192.168.178.201/32
    }
}

virtual_server 192.168.178.201 {
    lvs_sched rr
    lvs_method DR
    protocol TCP
    persistence_timeout 50
    delay_loop 10
    real_server 192.168.178.210 {
        TCP_CHECK {
          connect_timeout 5
          connect_port 8443
        }
    }
    real_server 192.168.178.211 {
        TCP_CHECK {
          connect_timeout 5
          connect_port 8443
        }
    }
    real_server 192.168.178.212 {
        TCP_CHECK {
          connect_timeout 5
          connect_port 8443
        }
    }
}

On node 2

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    virtual_ipaddress {
        192.168.178.201/32
    }
}

virtual_server 192.168.178.201 {
    lvs_sched rr
    lvs_method DR
    protocol TCP
    persistence_timeout 50
    delay_loop 10
    real_server 192.168.178.211 {
        TCP_CHECK {
          connect_timeout 5
          connect_port 8443
        }
    }
    real_server 192.168.178.212 {
        TCP_CHECK {
          connect_timeout 5
          connect_port 8443
        }
    }
}

On node 3

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 50
    advert_int 1
    virtual_ipaddress {
        192.168.178.201/32
    }
}

virtual_server 192.168.178.201 {
    lvs_sched rr
    lvs_method DR
    protocol TCP
    persistence_timeout 50
    delay_loop 10
    real_server 192.168.178.212 {
        TCP_CHECK {
          connect_timeout 5
          connect_port 8443
        }
    }
}

NB: each node has LVS configured according to its priority (less and less real_server). If you configure LVS symmetrically, packets are going to be sent back and forth infinitely between the nodes and never answered.

You need to add the virtual IP address as a loopback address on all nodes. Otherwise, the BACKUP nodes receive the TCP messages from the load balancer but do not know what to do with it.

On all nodes in /etc/sysconfig/network-scripts/ifcfg-lo:0

DEVICE=lo:0
IPADDR=192.168.178.201
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback

On all nodes in /etc/sysconfig/network, add the line

GATEWAYDEV=eth0

On all nodes, configure kernel parameters

net.ipv4.conf.ens192.arp_ignore = 1
net.ipv4.conf.ens192.arp_announce = 2
net.ipv4.ip_forward = 1

References

http://kb.linuxvirtualserver.org/wiki/Building_Two-Node_Directors/Real_Servers_using_LVS_and_Keepalived

keepalived - Virtual is not routing my requests to real servers

1 Answers1

References