Route traffic through physical interface

Question

I have two ethernet interface on a linux machine. The both have the following IP addresses: Interface A: 10.0.1.5/24 Interface B: 10.0.2.5/24

Both interfaces are connected to a router that routes between the two networks.

Now I want to start an iperf3 measurment where I bind the server to Interface A and the client to Interface B.

The problem is that the traffic does not leave the physical interface but instead only flows through the kernel resulting in a speed that's not even possible on the link.

What can I do to force the traffic through the physical interface? ip_forward is disabled.

Thanks.

Answer 1

It would be very difficult (but not impossible) to have a Linux system, using a single routing stack, perform multi-homed routing of packets from itself to itself without using the lo interface.

But it's very easy on Linux to create additional network stacks to simulate multiple systems within one system: by using network namespaces.

Here one can relinquish one of the two NICs to a new network namespace which will be a peer to the initial (host) network namespace. They won't communicate directly, but only through the external router. Let's assume interfaces are really named A and B and that the router uses addresses 10.0.1.1/24 + 10.0.2.1/24.

• create a new network namespace with additional facilitated management when using iproute2 tools (under the hood, namespace pseudo-files are mounted to keep the resource around without a process, etc.).

  ip netns add sideB
  

• move interface B to the new network namespace:

  ip link set dev B netns sideB
  

  note: wireless interfaces require using the iw command instead.

• configure the new network namespace:

  All of its network settings are lost when an interface changes namespace (both on the host where the interface disappears triggering addresses disappearing triggering routes disappearing, and on the new network namespace):

  ip -n sideB link set dev B up
  ip -n sideB address add 10.0.2.5/24 dev B
  

  Not needed but in case iperf3 gets confused, have a functional loopback interface:

  ip -n sideB link set lo up
  

• configure routes between the two sides (initial/host namespace might have it already through the default route, but let's be explicit)

  ip route add 10.0.2.0/24 via 10.0.1.1 dev A
  
  ip -n sideB route add 10.0.1.0/24 via 10.0.2.1 dev B
  

Now one can run (in two terminals):

iperf3 -s

ip netns exec sideB iperf3 -c 10.0.1.5

with the client running in the new network namespace.

Once measuring is over, deleting the network namespace will return the NIC to the host namespace. But if any process is left by mistake using this network namespace (something like ip netns exec side B setsid sleep 9999) the ip netns delete sideB below will only remove the namespace from the view of the iproute2 tools, but not actually remove the namespace until the process ends, leaving the NIC difficult to recover back. So better move it back first:

ip -n sideB link set dev B netns 1
ip netns delete sideB

where 1 means PID 1's network namespace: the initial/host namespace. Additionally this allows to perform the experiment inside a container having been given physical NICs (because there 1's network namespace is the container's namespace, instead of the actual host's namespace) or the physical NIC would reappear on the host and be lost forever to the container if there's no access to the host.

If there's no tool (like udev + ifupdown's allow-hotplug or NetworkManager) detecting the apparition of the NIC, it will again have to be configured again on its return.