Nasty conundrum here.
Some background: I have a Sophos UTM ASG220 serving as gateway device for a number of networks, with a Cisco 2960 network switch, and a raft of Red Hat 6.6 servers running KVM and hosting multiple guests, with the guests being on different network subnets.
The UTM has its LAN interface populated with multiple virtual interfaces (its really a stripped down, optimized RHEL-type Linux machine under the hood) as gateways for all the network subnets except for the primary network it was created with during installation.
I have VLANs defined on the switch, and the KVM hosts are having bonded interfaces (mode 1, based on RHN support advice), VLAN sub interfaces and bridges configured for each network, and each guest is attached to its appropriate bridge and 8021q is setup. Without involving the UTM, VLAN traffic transverses beautifully, between swich, KVM hosts and guests, I have no issues there.
That said, this is what is happening:
I am successful in creating new VLAN interfaces on the Sophos UTM (but with a different IP address) to replace the existing virtual IP address serving as LAN gateway addresses. (for instance, for test network, virtual interface gateway address is 10.11.0.253, and the new VLAN interface expected to replace it is 10.11.0.254).
At first instance the guests and the kvm host are able to ping the switch, the new VLAN gateway interface and the old virtual gateway interface, after the VLAN is in place. But if I try to remove/delete the gateway's virtual interface (eg 10.11.0.253), then networking starts acting weird.
The switch's VLAN address (say 10.11.0.7) is unable to ping or reach the guests (say one guest is 10.11.0.36) on the VLAN, but it can reach the kvm host vlan bridge (say 10.11.0.4) address, and it can reach the Sophos gateway (10.11.0.254, new VLAN address).
Even after bringing the gateway virtual interface (10.11.0.253) back up the situation remains for a while. The guests can reach each other on the same VLAN, but cannot ping the switch VLAN interface address, and cannot ping their VLAN gateway address, or route traffic to other external networks except (oddly) the LAN DNS servers, which are on a different subnet entirely (192.168.2.0)! And the DNS servers are the only IP addresses in the 192.168.2.0 subnet the 10.11.0.0 guests can see!
arping responds perfectly to and from all network machines/devices while all this is going on.
This situation continues for a while even after rebooting the switch, and bringing up and down the gateway network interfaces.
Then things start working again (but with the gateway virtual and VLAN interfaces both up, I take down the virtual interface, bring it back up, then do the same to the VLAN interface.)
I'd love some insight to what's happening and how I can fix this. Thanks for reading this extremely long post!
