I have set up GCE-instances (google cloud) and EC2-instances (aws), and I want to set up routing like this:
AWS-DUB <-A-> GCE-LON <-B-> GCE-TOK <-C-> AWS-TOK
Connection A is wireguard VPN. Connection B is google cloud internal network. Connection C is wireguard VPN.
Now, I have been able to set up connectivity like I wanted, but bandwidth performance is poor.
Here are the examples from bandwidth testing with iperf:
TEST 1. GCE-TOK -> AWS-TOK = +200Mbits/s (over wireguard) TEST 2. GCE-LON -> GCE-TOK = +95Mbits/s (over google internal network) TEST 3. GCE-LON -> AWS-TOK = +20Mbits/s
The routing setup is done partially with Google Cloud Platform, VPC Network routing and also with routing on the instances. Iptables are not used to do any routing. Ip forwarding is enabled properly.
As we can see, bandwidth is poor in test 3.
The most noticeable difference is that TEST 1 and 2 both have test server and client on same subnet, while in test 3 we have one extra hop.
Traffic is routed like this from GCE-LON (instance-X.c.clp-perf.internal (10.158.0.99)) to AWS-TOK (172.54.0.229):
On ens4 interface using internal google cloud routing with a route set up for 172.54.0.0/16 with next hop as 10.188.0.4 (TOKYO GCE).
On 10.188.0.4 traffic arrives on ens4, then the route 172.54.0.0/16 dev wg0 forwards the traffic over wireguard ( (Tokyo GCE) 192.168.2.1 -> (AWS Tokyo) 192.168.2.2
(AWS Tokyo) Trafic is routed on 192.168.2.1 to 172.54.0.229 (eth0).
Latency:
TEST A. GCE-TOK -> AWS-TOK = 4ms
TEST B. GCE-LON <-> GCE-TOK = 225ms
TEST C. GCE-LON -> AWS-TOK = 229ms (Adds up correctly)
Individually, the routes in the chain in test 1 and test 2 is very fast. However when they are combined as in test 3, they become slow. I have experimented with MTU, TCP windows size etc, but no success yet.
I am aware that more details could be given here, but I would think from this description that someone with the experience would be able to chime in and give me some pointers as to why I am seeing such an intense hit in bandwidth performance.
All instances running Ubuntu 16.
Thank you
