I have VPC A with CIDR 10.A.0.0/16 and VPC B with CIDR 10.B.0.0/16. I have VPC A and B peered and updated the route tables and from a server in 10.B.0.0/16 can ping a server in 10.A.0.0/16 and vice versa.
The applications on VPC A also use some IPs in the 192.168.0.0/16 range. Not something I can easily change, but I need to be able to reach 192.168.0.0/16 on VPC A from VPC B. VPC A is used for an older kubernetes cluster that uses project-calico. Worker nodes (ec2 instances) get IPs in the VPC CIDR block 10.A.0.0/16 but the calico networking is set with the cluster CIDR setting 192.168.0.0/16 and pod IPs on those worker nodes get assigned in that range. The newer cluster is an EKS cluster and pod IPs are assigned from the VPC's CIDR range, 10.B.0.0/16. During the transition period I've got the two clusters' VPCs peered together. There's a distributed Elixir application running and the pods form an Erlang cluster by reaching each other via their pod IP address. With my current peering cluster A pods can reach both A and B pods, but cluster B pods can only reach B (due to the 192.168.0.0/16 IPs not being reachable.
I've tried adding 192.168.0.0/16 to the route table used for VPC B and setting the target of the peered connection. That does not work, I believe because 192.168.0.0/16 is not in the CIDR block for VPC A.
I'm unable to add 192.168.0.0/16 as a secondary CIDR in VPC A because it is restricted. See CIDR block association restrictions and related question. I understand it is restricted, but why is it restricted? RFC1918 doesn't seem to say anything against using more than one of the private address spaces.
I've also tried making a Transit Gateway, attaching both VPCs, and adding a static route to the Transit Gateway Route Table for 192.168.0.0/16 that targets the VPC A attachment. But still cannot reach that range from within VPC B.
Route Table
The route table for the private subnet for VPC A
10.A.0.0/16 local
10.B.0.0/16 pcx-[VPC A - VPC B peering connection]
0.0.0.0/0 nat-[gateway for cluster A]
Route table for the private subnet for VPC B
10.B.0.0/16 local
10.A.0.0/16 pcx-[VPC A - VPC B peering connection]
192.168.0.0/16 pcx-[VPC A - VPC B peering connection]
0.0.0.0/0 nat-[gateway for cluster B]
This does not work, of course, because 192.168.0.0/16 is not in VPC A's CIDR block, nor can it be added.
If I get a shell going on a Node A I can ping a 192.168... pod and I can ping a 10.B.0.0 pod. But from a shell on Node B I can only ping a 10.B.0.0 pod.
Is there another way to peer to both 10.0.0.0/8 and 192.168.0.0/16 CIDR blocks on the same VPC?
