How to configure on a multi-homed host so Docker containers on a custom network use a specific external IP for outbound traffic?

Question

I have a server (Ubuntu) with two public IPs: the first one is dynamic and the second static.

Software run on the server connects to a public internet endpoint which is behind a firewall for whitelisting IPs, hence it needs to use the static IP for outbound traffic there for things to work.

The server is configured so the static IP is on a loopback interface and set up to

$ sudo vim /etc/network/interfaces.d/eip.cfg

auto lo:1
iface lo:1 inet static
    address <static_ip>/32

post-up ip route replace default via <dynamic_ip> src <static_ip>

After network restart, it's possible to see the static IP like so:

$ curl ipinfo.io/ip
<static_ip>

Hence, all works well and it's possible to curl the other endpoint. All good.

PROBLEM

The software uses Docker and thus sits inside a container that even uses a custom network.

I have already tested to use --network=host and that works well because this puts the container on the host's network (where it's working).

The issue is that the container must be on the custom network, because it connects to other internal services on that network.

I understand that Docker uses MASQUERADE and that it could be possible to switch that off and configure iptables manually as an option. However, I wonder what other options are available and what is the option that could be recommended.

I'm open for suggestions - maybe someone has done this already! Thanks!

Answer 1

The packets generated in the container are being forwarded via the Docker hosts interface rather than being generated by the host. The src hint added to the default route does not take effect as the packets already have a source address. I believe iptables masq only chooses the source IP address based on the outbound interface, so I think you are stuck with inserting a new rule before the default MASQ rule (like the serverfault answer).

iptables -t nat -I POSTROUTING -p all -s <docker_network>/24 \ 
  -j SNAT --to-source <static_ip>

Unless you are familiar with Dockers use of iptables, then switching it off completely may cause more issues.

Answer 2

The easiest solution is to use the --outgoing-ip command option with docker run commands. This example represents a container with an outgoing IPv4 address. For IPv6, use --outgoing-ip6 instead.

An alternative method is define this in your network definition and add some iptables rules, you’ll have something like this:

# docker network create <NETWORK NAME> --subnet=192.168.10.0/24 \ --gateway=192.168.10.1

# iptables -t nat -I POSTROUTING -s \ 192.168.10.0/24 -j SNAT --to-source OUTGOING_IP 

Docker will have POSTROUTING rules.

Then connect your containers to this custom network created. All traffic for these containers will go through the specified IP Address.

# docker network connect <NETWORK-NAME> <CONTAINER>