I have two datacenters, and in each datacenter I have two pfSense 2.1.5 servers running in CARP failover mode.
I've established a OpenVPN, shared-key tunnel between the two datacenters, pointing at the remote CARP public IP address. Each site is running both a client and a server, so they are connecting bi-directionally.
Internally, I make the remote gateway point to an INTERNAL CARP address on the firewall, so if one firewall goes down, servers behind it can get out through either.
Everything came up, and I'm able talk between the two sites fine ... for about 30 seconds. On a regular interval, the connection between the two will drop FOR EVERYTHING EXCEPT PING. i.e. mapped drives between the two will timeout, database connections will drop, etc however I get a steady successful ping the whole time.
This is driving me nuts. I'm not sure if OpenVPN, CARP or some combination of the two is causing the problem. I'm open to switching over to IPSEC if it is better supported with this config.
Network Configuration:
DATACENTER 1:
10.1.1.x => INTERNAL NETWORK
10.1.1.2 => pfSense 1
55.55.55.66 => pfSense 1 WAN address
10.1.1.3 => pfSense 2
55.55.55.67 => pfSense 2 WAN address
55.55.55.68 => PUBLIC, CARP address
10.1.1.4 => Internal CARP address (gateway for all servers behind the firewall)
10.1.253.x => CARP private subnet (for heartbeats)
DATACENTER 2:
192.168.1.x => INTERNAL NETWORK
192.168.1.2 => pfSense 1
88.88.88.66 => pfSense 1 WAN address
192.168.1.3 => pfSense 2
88.88.88.67 => pfSense 2 WAN address
88.88.88.68 => PUBLIC, CARP address
192.168.1.4 => Internal CARP address (gateway for all servers behind the firewall)
168.168.253.x => CARP private subnet (for heartbeats)
DATACENTER 1 OpenVPN config:
SERVER
Server Mode: Peer to Peer (Shared Key)
Protocol: UDP
Interface: 55.55.55.68 (this is the public CARP address)
IPv4 Tunnel Network: 10.0.8.1/30
IPv4 Local Network: 10.1.1.0/24
IPv4 Remote Network: 192.168.1.0/24
Concurrent Connections: 24
CLIENT
Protocol: UDP
Interface: WAN
Server Host: 88.88.88.68 (public CARP address at other datacenter)
Port 1194
IPv4 Tunnel Network: 10.0.9.1/30
IPv4 Remote Network: 192.168.1.0/24
DATACENTER 2
SERVER
Server Mode: Peer to Peer (Shared Key)
Protocol: UDP
Interface: 88.88.88.68 (this is the public CARP address)
IPv4 Tunnel Network: 10.0.9.1/30
IPv4 Local Network: 192.168.1.0.0/24
IPv4 Remote Network: 10.1.1.0/24
Concurrent Connections: 32
CLIENT
Protocol: UDP
Interface: WAN
Server Host: 55.55.55.68 (public CARP address at other datacenter)
Port 1194
IPv4 Tunnel Network: 10.0.8.1/30
IPv4 Remote Network: 10.1.1.0/24
