2

We have a server with both a /31 and a /24 in different /8s statically routed to it. We have a gateway in the same /24 as the /31.

Incoming connections to IPs in the /24 work as expected, but we are unable to initiate outgoing connections. Our "gold-standard" test is (using an IP to avoid dns issues at this point):

curl --interface SOME_IP_IN_THE_24 -v -H 'Host: httpbin.org' 34.230.136.58/ip

We're using Ubuntu 16.04, and we've configured eno1 with one of the /31, eno1:1 with the other /31, and eno1:2 with one of the IPs from the /24. When we run the command above, we still get the IP for eno1.

SEK
  • 123
  • 4
  • Provide the outputs of commands `ip -4 a` and `ip -4 r`, and `iptables-save -c`, if you can. – Anton Danilov Jul 04 '19 at 14:34
  • @AntonDanilov iptables only has rules on the INPUT chain, and a MASQUERADE rule on the POSTROUTING chain of the nat table, but otherwise empty. This is the result of `ip route` using RFC 5737 IPs in place of real ones: ``` default via 192.0.2.1 dev eno1 onlink 203.0.113.0/24 dev eno1 proto kernel scope link src 203.0.113.198 192.0.2.0/24 dev eno1 proto kernel scope link src 192.0.2.198 ``` We're able to get incoming traffic, but outgoing it's getting mangled to use the wrong IP. – SEK Jul 04 '19 at 15:44

1 Answers1

4

Troubleshooting

  • Check the routing with ip route get <dst> from <SOME_IP_IN_THE_24> command. Fix it if the command has returned unexpected route.
  • Check the firewall with iptables-save -c. The source address can be overridden by NAT rules.

The answer

Your issue caused by the MASQUERADE rule. Any source address is rewritten to an ip address, which you see in the source attribute of route, that has been got by the ip route get <dst> command. You can sure in that with using of the tcpdump and conntrack -E commands. Likely also you see increment of this rule counters.

Simple way to avoid this behavior is just insert ACCEPT rules at the top of the nat/PREROUTING chain. Better way to change the firewall rule set is usage of iptables-save/iptables-apply commands. Your nat/PREROUTING chain should look like (iptables-save -t nat):

# Generated by iptables-save v1.6.2 on Thu Jul  4 19:05:27 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING --src <LOCAL-IP-1> -j ACCEPT
-A POSTROUTING --src <LOCAL-IP-2> -j ACCEPT
...
-A POSTROUTING --src <LOCAL-IP-N> -j ACCEPT
-A POSTROUTING -j MASQUERADE
COMMIT
# Completed on Thu Jul  4 19:05:27 2019
Anton Danilov
  • 4,874
  • 2
  • 11
  • 20
  • we brought in a networking expert who fixed the issue for us. The solution was to add a new routing table for the /24 and configure all the IPs for the same interface (not using virtual interfaces). The masquerade rule was also causing issues (it was added automatically by docker) and removing it worked perfectly, but broke outgoing connections from docker, so we added rules as you suggested. Thank you! – SEK Jul 21 '19 at 11:21