0

I'm rather new to this, so please be patient.

My setup is essentially thre devices using mqtt, connecting through two brokers both running emqx. Both emqx brokers are also supposed to be set up as haproxy load balancers, with keepalived in case one goes down.

broker 1's IP is 192.168.1.201

broker 2's IP address is 192.168.1.202

keepalived's virtual IP is 192.168.1.200

haproxy.cfg is 

frontend emqx_tcp
    bind *:1883
    option tcplog
    mode tcp
    default_backend emqx_tcp_back

backend emqx_tcp_back
    balance roundrobin
    server emqx_node_1 192.168.1.201:1883 check
    server emqx_node_2 192.168.1.202:1883 check

keepalived.conf is 

global_defs {
        lvs_id haproxy01
}

vrrp_sync_group SyncGroup01 {
        group {
                VI_1
        }
}

vrrp_script chkhaproxy {
        script "/usr/bin/killall -0 haproxy"
        script "/usr/sbin/service haproxy start"
        interval 9
        timeout 3
        weight 20
        rise 2
        fall 4
}

vrrp_instance VI_1 {
        interface eth0                # interface to monitor
        state MASTER
        virtual_router_id 51          # Assign one ID for this route
        priority 101                  # 101 on MASTER, 100 on BACKUP
        advert_int 5
        authentication {
                auth_type PASS
                auth_pass password
        }
        virtual_ipaddress {
                192.168.1.200         # the virtual IP
        }
        track_script {
                chkhaproxy
        }
}

Problem is, I can't seem to get either haproxy or keepalived to work as intended. When I have broker 1 only running haproxy (keepalived is not running), and connect directly to broker 1's IP address from the client, it seems to only forward connections to broker 2. It doesn't seem to be able to connect to broker 1 (and yes, I checked that they were both up. If only broker 2 is up, the client just doesn't connect). After some digging, I thought that it might be because haproxy and the broker use the same IP. So I tried using a virtual IP from keepalived. But after trying that, I found that I couldn't even connect to the virtual IP. The client, upon trying to connect to it, would give me the error "OSError: [Errno 113] No route to host".

what am I missing?

Edit: upon tailing the logs with tailf /var/log/syslog this is the result

Feb 7 14:56:19 pi01 Keepalived_healthcheckers[10453]: Opening file '/etc/keepalived/keepalived.conf'.

Feb 7 14:56:19 pi01 Keepalived_healthcheckers[10453]: Unknown keyword 'lvs_id'

Feb 7 14:56:19 pi01 Keepalived_vrrp[10454]: Default interface eth0 does not exist and no interface specified. Skipping static address 192.168.1.200.

Feb 7 14:56:19 pi01 Keepalived_vrrp[10454]: Unable to load ipset library - libipset.so.3: cannot open shared object file: No such file or directory

Feb 7 14:56:19 pi01 Keepalived_vrrp[10454]: VRRP_Instance(VI_1) Unknown interface ! Feb 7 14:56:19 pi01 Keepalived_healthcheckers[10453]: Using LinkWatch kernel netlink reflector...

Feb 7 14:56:20 pi01 Keepalived_vrrp[10454]: Stopped

Feb 7 14:56:20 pi01 Keepalived[10450]: Keepalived_vrrp exited with permanent error CONFIG. Terminating

Feb 7 14:56:20 pi01 Keepalived[10450]: Stopping

Feb 7 14:56:20 pi01 Keepalived_healthcheckers[10453]: Stopped

Feb 7 14:56:25 pi01 Keepalived[10450]: Stopped Keepalived v1.3.2 (12/25,2016)

It seems to consistently fail upon the line Using "LinkWatch kernel netlink reflector"

Brandon
  • 1
  • 3

1 Answers1

0

First problem: HAProxy is configured to listen on all interfaces on the same port used by emqx. This should cause you problems.

Either set up a specific NIC for the keepalived/haproxy listener if you want them to use the same port, or make HAProxy and emqx listen on different ports.

Once you have that working, it’s time to look at Keepalived. My best tip there is to tail your logs. Make sure the router ID is identical between the master and backup, and that the backup gets a lower prioriy than the master. I frankly don’t remember from the top of my head if you must allow mac spoofing or similar in the switch for VRRP (the protocol used by keepalived) to work.

Mikael H
  • 4,868
  • 2
  • 8
  • 15
  • 1
    Multicast traffic must also be enabled on the switch to let keepalive vrrp do its thing. – wurtel Feb 07 '19 at 15:05
  • I was able to get haproxy working through using different ports, but I'm still a bit confused on how to set up the vrrp. My router (I don't have access to a switch) is a Netgear Nighthawk AC1750, and it doesn't seem to have an option to enable multicast (to my knowledge). Is there any way around this? – Brandon Feb 07 '19 at 19:12
  • It’s not definitive that it won’t work on what gear you have, so let’s fix the obvious problems with Keepalived first: 1) You’ve probably cut-and-pasted the config from somewhere, and you’ve got the wrong name for your NIC, and 2) you’re missing some relevant library. For 1, run `ifconfig`or `ip addr show`, and see what your interface is really called. For 2, search the web for the name of the missing library (`libipset.so.3`) and see what package satisfies the dependency. – Mikael H Feb 07 '19 at 21:18
  • When it comes to switch settings: Are you using virtual machines for this setup or physical ones? If you’re running VMs on a single host, the capabilities of your physical switch (or in your case router) are irrelevant for whether keepalived will work. – Mikael H Feb 07 '19 at 21:21
  • I'm using physical ones. – Brandon Feb 08 '19 at 00:34
  • OK, we may be able to get VRRP working anyway, but I'd say the first part of the proof of concept remains to ensure that HAProxy can forward traffic to both emqx instances - which you seem to have done, and to ensure Keepalived on at least one of the hosts presents a virtual IP through which you can reach HAProxy - which was still an issue. As I said you've most likely told Keepalived to listen on network interfaces that don't exist in your computer(s). As I wrote in my last comment, try running `ifconfig` or `ip addr show`. Ifconfig will output a number of text blocks starting with...-> – Mikael H Feb 08 '19 at 07:29
  • ..the name of the network interface. Example: `ens192: flags=4163 mtu 1500...`. Likewise, ip addr show outputs numbered blocks, each starting with the name of the interface, e.g. `2: ens192: mtu 1500 qdisc fq_codel state UP group default qlen 1000`. In my specific case, the network interface is called `ens192`, so in my version of keepalived.conf, I would need to replace the line `interface eth0` with `interface ens192`. Ignore the missing libip.so.3 in your logs for now. It may be a red herring. – Mikael H Feb 08 '19 at 07:35
  • alright, I changed interface eth0 to what I found from ip addr show (wlan0) and now both the master and the backup of keepalived are working. Thanks! – Brandon Feb 08 '19 at 14:58