TCP Anycast Multipath Routing: Router always prefers one nexthop over another

Question

I am tring to use TCP Anycast without additional protocols to connect one client to one of multiple hosts with the same anycast IP.
My topology looks as follows:

The server in the middle is an Ubuntu machine, providing two routes for the anycast address.
My multipath routing configuration (at router) is:

`10.11.12.13  proto static
    nexthop via 192.168.2.1  dev enp0s9 weight 1
    nexthop via 192.168.3.1  dev enp0s10 weight 1`  

The anycast address is 10.11.12.13 and is assinged to the local loopback of the two routers at the right hand side. On the left hand side is a client with the router in the middle as its default gateway.

Scenario 1: The router in the middle starts a tcp session with the anycast address as destination.

Scenario 2: The client at the left hand side starts a tcp session with the anycast address as destination.

Observation:

When starting a tcp session from the router (ubuntu linux), everything works as desired. The SYN Package gets forwarded over one of each hops with an equal probability. When a session is established, all packages of this session go through the same hop, keeping the connection alive (successful flow-based multipath routing).

The problem occurs in scenario 2, when I try to establish a tcp session from the client at 192.168.1.1/24. The router always forwards the packages to only one of the hosts (the one with the IP suffix 3.1). If I take the connection eth2 to this host down, the router forwards the packages to the second host. But when putting eth2 back up again, the connection switches back to the host at 3.1 leading to an disconnect between the client and the host at suffix 2.1.

I can only post one video link, so here is a video showing how it looks like from the client's perspective: https://youtu.be/kthYK9uEwvM
Notice how the tcp SYN from client gets always forwarded to the host with the 3.1 suffix (bottom right corner).

Terminals:
Router: top left, client: bottom left, host 2.1: top right, host 3.1: bottom right.

My kernel version is the 4.4 Linux Kernel. The tcp session is established using netcat (server) and telnet(client).

Changing the weight of the nexthops also only affects routing from router (192.168.1.254 > 10.11.12.13).

Why does the router forward the SYN package diffrent, when the package is send from client than when it is send from the router itself?

Thanks for your help!

Answer 1

It seems like this is a problem of ipv4.
Using ipv6 and a linux kernel 4.4 ubuntu machine with nearly the same setup worked as expected. Link to image of topology
Although ICPM ping messages always get routet through the first nexthop, tcp messages are equaly distributed along the nexthops. Established tcp sessions seem to be stable (thus the kernel identifies flows correctly).