I have a IP block from RIR.
I'm using two providers for make anycast to "some" IPs. I will call this as A and B.
I want forward to other provider when that IP not at that location. I'm using two Ubuntu servers to BGP announce and forwarding packets, and a pfSense machine between Provider A and unicasted endpoint.
The anycasted IP works great. However, If client near at Provider B that packet doesn't reach endpoint.
Here is how this configured:
Client --> Provider B -(GRE, Static Route)-> Provider A -(GRE, Static Route)-> pfSense --> Endpoint
^ Here is problem
The exact problem is packet not forwarded from Provider A to pfSense. I can see ICMP packets at Provider A Server tcpdump, But not at pfSense tcpdump. I allowed firewall and I can't see any blocked logs. All of endpoint traffic passing GRE tunnel that located between pfSense and Provider A.
If I do ping using Provider B Server, that not works too. However, If I do ping with GRE Interface IP between Provider A, That works even there is no NAT.
Of course, I can ping to Endpoint when I do ping near Provider A.
What works (itself means server at provider):
Client(or itself) --> Provider A or B -(GRE, Static Route)-> pfSense -(GRE, Static Route)-> Anycast IP
Client(or itself) --> Provider A -(GRE, Static Route)-> pfSense -(GRE, Static Route)-> Unicast IP at near Provider A
Provider B -(GRE, Static Route, GRE IP or same subnet IP excluding Anycasted)-> pfSense -(GRE, Static Route)-> Unicast IP at near Provider A
Unicast IP at near Provider A --> pfSense -(GRE, Static Route)-> Provider A
Provider B --> Server near Provider B
What not works:
Client(or itself) --> Provider B -(GRE, Static Route)-> Provider A -(GRE, Static Route, Traffic stops here)-> pfSense -(GRE, Static Route)-> Unicast IP at near Provider A
Unicast IP at near Provider A --> pfSense -(GRE, Static Route)-> Provider B --> Server near Provider B --> Provider B --> Provider A -(Traffic stops here)-> pfSense --> Endpoint
Here is sysctl -p results:
Provider A
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.accept_source_route = 1
net.ipv6.conf.all.accept_source_route = 1
net.ipv6.conf.all.accept_ra = 2
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
Provider B
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.accept_source_route = 1
net.ipv6.conf.all.accept_source_route = 1
If more information needed, Please let me know. Thanks.
