openvswitch: connecting two virtual private networks on two different machines with gre/ipsec

Question

Here is the setup: We have a hardware (server) that presents a service to the customer (or "outside world", if you so will). Internally, the services and functions are distributed over several virtual machines (of different roles). We use kvm for virtualisation and libvirt for the management of it. The hardware has two NICs which are both forwarded to one of the virtual machines (called gateway-vm) by the means of "Single Root I/O Virtualization" (Intel VT-D / AMD IOMMU).

We use openvswitch in the underlying host operating system to create a internal network with private addresses between all VMs and the host. This because we had problems binding a bridge to a virtual interface. (As opposed to binding it to a physical interface, the operation/command was accepted, but network communication not possible anymore)
That works fine. A proxy server in the gateway-vm offers all the internal services to the outside world. Fine, that works.

We even have several of these machines.

For redundancy, we would like one of the internal virtual machines (which is not the gateway-vm, of course, let's call it the database-vm) to talk to the database-vm of another hardware. Therefore we make sure that the internal private addresses do not collide, e.g. 10.10.1.5 and 10.10.2.5 (each /24).
Our approach is to set up gre tunnel with ipsec between the two gateway-machines, which does already work. 10.10.1.1 can ping 10.10.2.1 and vice-versa.

Now we are stuck at getting the database-vm of the one hardware to talk/ping to the other gateway-vm of the other hardware, vice-versa and ultimately letting the two database-vms talk to each other.

What would be the proper way to achieve this ? We are not sure whether we need ip_forwarding in either of the gateways, and how a route should look like. Or should all be one big subnet without routing ?

We are currently using ovs v1.7.1, kernel 3.2.6, libvirt v0.10.2, QEMU 1.1.1

Answer 1

Because you're running a proxy server on the gateway-vm(s), presumably they are not doing ip-forwarding on them.

You could enable ip-forwarding on your gateway-vm(s) and route the traffic like: database-vm1 <-> gateway-vm1 <-ipsec/gre-> gateway-vm2 <-> database-vm2

You would want to be careful that you don't enable traffic forwarding to bypass your proxy servers for outgoing traffic...unlikely, probably, but something to be aware of as you're setting this up.

Getting the routing right could be a bit tricky. If you're really using /16s on the network that the database servers are on, then you may need to adjust some things as 10.10.1.5 and 10.10.2.5 are in the same /16 and that will make the routing setup difficult. Perhaps if you just shift your netmasks to /24's so that they're in different IP networks it'll make the routes easier to figure out?

I'm a little bit of an odd-ball in the networking world as I like pushing routing decisions farther out to the edge, even out into hosts at times, so I'll mention that you might want to look at something like quagga running OSPF to perhaps simplify some of this. Maybe you won't find it simpler, I'm not sure.

The general idea is that gateway-vm1 needs to have a route for the gateway-vm2<->database-vm2 link in its routing table with a next-hop to gateway-vm2 via the IPSec/GRE tunnel. Similarly, gateway-vm2 needs to have a route for the gateway-vm1<->database-vm1 link in its routing table with a next-hop to gateway-vm1 via the IPSec/GRE tunnel.

If you're using IPSec/GRE interconnects...one big subnet isn't strictly off the table (you could theoretically bridge in the gateway-vms between the virtual switch and the GRE link), but I certainly wouldn't want to be setting it up that way.

If I were in your shoes, I might re-think my allocation of the physical NICs and use one for an interconnect between the hardware so you don't have to use IPSec/GRE for the crossover...with that, you could run an ovs instance on each hardware, connect the crossover NIC to it and use it as a private interconnect and the database-vms could then talk to each other "directly" on this private interconnect.