Change the default route without affecting existing TCP connections

Question

Let's say I have two public network addresses on my server: one NAT through an ISP (192.168.99.0/24), and a VPN through a different ISP (192.168.1.0/24), already configured with a per-host route to the VPN server through my ISP.

Here is my initial routing table. I am currently routing through my ISP on subnet 192.168.99.0/24.

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.99.1    0.0.0.0         UG    0      0        0 eth1
55.66.77.88     192.168.99.1    255.255.255.255 UGH   0      0        0 eth1
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 tap0

Now, I want new TCP connections to switch to my 192.168.1.0/24 so I type the following:

$ route add -net 0.0.0.0 gw 192.168.1.1 dev tap0

When I do this, it causes some long-standing TCP connections to hang. Is there a way to I safely change the default interface for new connections, while allowing existing TCP connections to use the old route (i.e. do I need enable some sort of stateful routing table)?

I am okay with a solution that only works with established TCP connections, and I don't care how hacky it is. For example, if there is a way to add temporary iptables rules for existing connections to force them over the old route. But there has to be some way to do this.

EDIT: Just a note about a simple "route add -host ... " for existing connections: this solution would work if I am fine with leaving a subset of IPs on the old interface. However, in my application, this actually doesn't solve my problem because I want to allow new connections to come on the new interface even if they have the same source IP. I'm now looking at using the "ip route" command to set source-based routing rules.

Answer 1

The following command will examine open tcp connections, and list the destinations outside of the local network:

$ netstat -4ntu | tail -n +3 | cut -c45-65 | cut -d: -f1 | grep -v '192.168.99' | grep -v '192.168.1'
214.236.220.101
63.171.235.16
174.63.61.132
159.47.216.178
62.59.197.29
139.47.218.150
56.4.17.22
204.63.49.5
155.9.79.196
144.31.207.66

So we just need to iterate through this list, and put specific routes toward the main router before changing the default route:

for i in $(netstat -4nt | tail -n +3 | cut -c45-65 | cut -d: -f1 |grep -v '192\.168\.99' | grep -v '192\.168\.1') ; do echo route add -host $i gw 192.168.1.99; done

(I have left the "echo" command in there so you can see what it does, you can remove this when you want to actually do it.

Then change your default route:

$ route add -net 0.0.0.0 gw 192.168.1.1 dev tap0

Because the routes added are more specific, they will be taken in preference to the default route. Obviously you can put this in a script to make it easier.