IP forwarding on Linux - anything important to make sure to do or know?

Question

In moving a website with a dedicated IP from one server to another, to minimize downtime due to DNS propagation delays, there's the approach of using IP forwarding so that all traffic to the original IP gets forwarded to the new IP.

Is there anything important to know when doing this? Here are the steps I plan to use. Is there anything from a security perspective or otherwise that I'm missing?

1. echo "1" > /proc/sys/net/ipv4/ip_forward (or set it permanently)
2. iptables -t nat -A PREROUTING -d original.ip.goes.here -p tcp --dport 80 -j DNAT --to-destination new.ip.goes.here
3. iptables -t nat -A POSTROUTING -p tcp -d new.ip.goes.here --dport 80 -j MASQUERADE
4. Repeat #2 and #3 but for port 443 instead of 80 if the site has SSL

I understand downtime can be reduced without resorting to this by lowering the TTL of the DNS records far enough in advance of the change, but that's still not quite as good at this at minimizing downtime since supposedly some DNS servers (and perhaps clients) will cache records for longer than the TTL says if it's short.

EDIT:

Part of what got me wondering if there's something I'm missing is the question of why ip_forward isn't always set to 1 and instead defaults to 0 - like is there some security risk or undesired behavior if having it set to 1 in certain situations.

Answer 1

Part of what got me wondering if there's something I'm missing is the question of why ip_forward isn't always set to 1 and instead defaults to 0 - like is there some security risk or undesired behavior if having it set to 1 in certain situations.

If your system (as with many others) does not need to be a router there is no reason to enable routing.

Regarding port 80. As you already have a webserver listening on example.com it is fairly easy to configure a reverse proxy to the new webserver. There are plenty of examples on Server Fault on how to do this but briefly

<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyPass / http://example.com/
    ProxyPassReverse / http://example.com/

    ServerName example.com
</VirtualHost>

You can do exactly the same for https on port 443

<VirtualHost *:443>
    ProxyPreserveHost On
    ProxyPass / https://example.com/
    ProxyPassReverse / https://example.com/

    ServerName example.com
</VirtualHost>

The only other thing you need to do is configure a local DNS resolver with an entry for example.com so that it is picked up in preference to the global DNS. Something like dnsmasq should do this easily.

---

In your particular case you an prepare your new vhosts for example.com ahead of time, install dnsmasq and add example.com to your local hosts file. Then when you are ready, enable the dnsmasq service and restart apache and away you go.

Answer 2

There is no inherent insecurity with IP forwarding itself, other then how your firewall is configured, if they are the same machine. On the contrary, it can provide some sort of security by hiding the real server ip.

By enabling ip_forwarding one can turn a linux box into a router (that can do packet forwarding between networks) which is not always needed or expected and that's why it is disabled by default.

The follwoing article from RedHat explains all this very well.

7.4. FORWARD and NAT Rules

It is not quite clear where you are adding the rules, as you will need to add this at the edge firewall/router/gateway that can intercept and route packet to the destination desired. Otherwise it is not going to work. And as long as this rules are applied at the edge, there is no additional security issues involved, as your internal network remains secured as before. But this depends on your network structure.

I also guess, this will be a temporary measure and the rules will be removed later on. May be you should also do all possible tests beforehand and make sure that, it is going to work, the way you want it.

Answer 3

ip_forwarding: ip_forwarding could be dangerous in situations where public ip addresses are used. A newly installed Linux machine could then be used as a router for networks that are not supposed to be routed this way.

iptables: The main problem with your iptables setup is probably the routing on the new machine. That machine has to use the old machine as a router to send back the mangled packets to and that way you end up in an routing challenge. It is probably a lot safer / easier to use a proxy like varnish to forward the http traffic. If you use apache or nginx for hosting you can even setup these as proxy servers for your new webserver.

Answer 4

You can use haproxy for this. The config can be found below:

global
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    option                  abortonclose
    no option               checkcache
    option                  redispatch
    retries                 3
    timeout http-request    30s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout server          1m
    timeout http-keep-alive 5s
    timeout check           10s
    maxconn                 4000

# x.x.x.x = new IP
# y.y.y.y = old IP

listen l.x.x.x.x
       option forwardfor header X-Real-IP
       option http-server-close
       source y.y.y.y
       bind y.y.y.y:80
       server newserver x.x.x.x:80 id 1

listen l.x.x.x.x.ssl
       source y.y.y.y
       mode tcp
       bind y.y.y.y:443
       server newserver x.x.x.x:443 id 1

Also, you can use dnsmasq to forward the DNS requests - a useful answer for this can be found right here on serverfault How to enforce dnsmasq to use dns for some hosts?

Answer 5

I have moved data centers a few times, with a full class C block changing along with the move. It is wise to use conntrack in iptables as well as snat.

Here is a handy little script I used a few times. Simple and works like a charm. Add additional ports as needed. Once DNS has refreshed and you have no more connections, remove the iptables rules.

#!/bin/sh
IPTABLES="/sbin/iptables"

# modify to suit
EXTERNAL="eth0" 
OLDSERVER="10.10.10.1"
NEWSERVER="10.10.20.2"

$IPTABLES -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

$IPTABLES -A FORWARD -i $EXTERNAL -o $EXTERNAL -p tcp --dport 80 -d $OLDSERVER -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -t nat -A PREROUTING -i $EXTERNAL -p tcp --dport 80 -d $OLDSERVER -j DNAT --to-destination $NEWSERVER
$IPTABLES -t nat -A POSTROUTING -p tcp --dport 80 -d $NEWSERVER -j SNAT --to $OLDSERVER

echo 1 > /proc/sys/net/ipv4/ip_forward

This forwarding should be implemented inside a firewall, and not left to public access.

As for why forwarding is not set on by default, may other responses say it best: If the device is not routing packets, it should not be on. Is it a security risk? That all depends on the server's role and config.

Hope this helps.

Answer 6

You also have to make sure that -A Forwarding is not set to ACCEPT as your Server would become an Open Router which could (and most likely will) be used for malicious activity. With that said just add a rule to Forward only the traffic you need and you should be good to go.

Forwarding is probably disabled for 2 reasons: Number 1 being that every module loaded and active uses computing power. When it's disabled, you save computing power. Its as simple as that. Number 2 is a little more complicated: When someone who is new to Linux or Iptables just uses the default Ruleset there will be the default FORWARD accept policy which leads to an open router (again). As nobody wants that there's a second "security measure" taken before your server starts routing packets.

Answer 7

Your iptables NAT redirects will work but be aware that this method is dependent on the conntrack module. If your server has too many simultaneous requests, the conntrack table will become full and you will experience downtime. You can of course increase the size of the conntrack hash-table and how the hash-lookups are done, but this may impact performance. So I advise you investigate this thoroughly if your server is servicing a lot of web traffic.

I would also not use -j MASQUERADE in your rules here; only SNAT and DNAT. I have done this before; my rules are as follows:

iptables -t nat -A PREROUTING  -p tcp -s TEST_IP -d ORIGINAL_IP --dport 80 -j DNAT --to NEW_IP
iptables -t nat -A POSTROUTING -p tcp -s TEST_IP -d NEW_IP --dport 80 -j SNAT --to ORIGINAL_IP

This is handy because it restricts the forwarding down to a single source (test) host/IP address and you will then be able to use this host to test that the forwarding works. Once you're happy, you can then re-apply these rules with the -s TEST_IP removed.

Because of my first point regarding conntrack overloading, I would still lower DNS TTL as you have described - this should minimize the total amount of traffic reaching your old server, so any rogue requests are dealt with via the iptables NAT port redirect, where the volume of requests should be low enough so as not to risk filling your conntrack table.

Answer 8

You don't need iptables, haproxyetc for a such simple task. Just install rinetd, its configuration file is most simple.

<your IP/FQDN> <your port> <where to forward IP/FQDN> <where to forward port>

i.e.

old.webserver.com 80 new.webserver.com 80

Answer 9

The best example I have seen of IP forwarding is by the town Taiji, Japan website. As a popular target for hactivists they keep a dummy outdated Apache webserver running at their dedicated forwarding destination IP address of 58.94.160.100. This serves as a lightening rod for their other three webservers which are MS 2012R models, assigned to 10.0.0.0 but physically located at parts unknown. They utilize a dozen different hosting companies to stabilize their website while using persistent IP forwarding, with almost no downtime whatsoever. Unless your company is a popular target for denial of service attacks, forwarding your signal to the dedicated and forwarded IP address should be no problem at all, especially with your planning so well in advance. My VPN uses a NYC network, which is registered with opendns.com, and with att.net as my ISP, this gives me the nameservers at 208.67.220.220 and 208.67.222.222 which are close to the att.net nameservers at 209.xxx.xxx.xxx. My loop is set for 127.0.0.1 which works well using Tor and dnschef in TCP and ipv6 modes. My first network adapter is NAT, then I designate two more as nat network, with my fourth and final adapter as host only. The result is that I always occupy either 10.0.2.2, 127.0.0.1 or 192.156.0.100 and this is ideal for anonymity with the added benefit of being able to resolve website addresses. In summation, I thought IP forwarding was probably intended to be utilized for security purposes, thus defaulting to 0 may be a way to protect the anonymity of the specific subnet geographics, even with registered nameservers, as there should be a complete separation of the working network in the event of a breakdown at the forwarded IP. People searching for your IP will be unlikely to discover any dns leakage, as the system will need to be actively maintained. I've heard that there are as many as 50,000 public nameservers available worldwide, and someone with your clear networking expertise could easily work some of those into your system, on a temporary basis while moving, and avoid any downtime.