What am I doing wrong with private GKEs peering and Cloud NAT?

Question

I am trying to mount the following platform on Google Cloud:

2 private (native) GKE clusters, in 2 different VPCs and to give them access to the internet, each vpc has a Cloud Nat configurated.

What I need is that the 2 GKE clusters interact, but peering the VPCs, I only get communication between PODs, not between POD -> Service or POD -> internal load balancer.

Clusters:

NAME     LOCATION      MASTER_VERSION  MASTER_IP        MACHINE_TYPE  NODE_VERSION   NUM_NODES  STATUS
Shrek01  asia-east1-a  1.16.8-gke.15   <none>           g1-small      1.16.8-gke.15  3          RUNNING
Shrek02  asia-east2-a  1.15.9-gke.24   <none>           g1-small      1.15.9-gke.24  3          RUNNING

vpcs:

NAME         SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
Shrek01      CUSTOM       REGIONAL
Shrek02      CUSTOM       REGIONAL

subnets:

NAME                REGION        NETWORK      RANGE
Shrek01             asia-east1    Shrek01      192.168.13.0/24
Shrek02             asia-east2    Shrek02      192.168.14.0/24

peerings:

NAME                                     NETWORK      PEER_PROJECT                  PEER_NETWORK                            AUTO_CREATE_ROUTES  STATE   STATE_DETAILS
Shrek01-Shrek01-peering                  Shrek01      pocprod2-2019001              Shrek02                                 True                ACTIVE  [2020-05-16T14:29:57.864-07:00]: Connected.
Shrek02-Shrek01-peering                  Shrek02      pocprod2-2019001              Shrek01                                 True                ACTIVE  [2020-05-16T14:29:57.864-07:00]: Connected.

firewalls rules:

"Shrek01-peering-ingress"

{
  "allowed": [
    {
      "IPProtocol": "all"
    }
  ],
  "creationTimestamp": "2020-05-16T16:05:14.829-07:00",
  "description": "",
  "direction": "INGRESS",
  "disabled": false,
  "id": "6807007164648771397",
  "kind": "compute#firewall",
  "logConfig": {
    "enable": false
  },
  "name": "peering-ingress",
  "network": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/networks/Shrek01",
  "priority": 1000,
  "selfLink": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/firewalls/peering-ingress",
  "sourceRanges": [
    "192.168.14.0/24",
    "10.113.64.0/19",
    "10.213.64.0/19"
  ]
}

"Shrek02-peering-ingress"

{
  "allowed": [
    {
      "IPProtocol": "all"
    }
  ],
  "creationTimestamp": "2020-05-16T16:24:28.545-07:00",
  "description": "",
  "direction": "INGRESS",
  "disabled": false,
  "id": "7130188648920500419",
  "kind": "compute#firewall",
  "logConfig": {
    "enable": false
  },
  "name": "Shrek02-peering-ingress",
  "network": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/networks/Shrek02",
  "priority": 1000,
  "selfLink": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/firewalls/Shrek02-peering-ingress",
  "sourceRanges": [
    "192.168.13.0/24",
    "10.113.32.0/19",
    "10.213.32.0/19"
  ]
}

k8s Shrek01 cluster:

kubectl get svc -o wide

NAME         TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE   SELECTOR
kubernetes   ClusterIP      10.213.32.1    <none>         443/TCP        85m   <none>
nginx        LoadBalancer   10.213.60.14   192.168.13.7   80:32612/TCP   92s   app=nginx
nginx-cip    ClusterIP      10.213.34.24   <none>         80/TCP         93s   app=nginx
nginx-np     NodePort       10.213.35.31   <none>         80:30444/TCP   92s   app=nginx

kubectl get pod -o wide

NAME                     READY   STATUS    RESTARTS   AGE     IP             NODE                                     NOMINATED NODE   READINESS GATES
nginx-64b4f9bb85-9sjcp   1/1     Running   0          3m34s   10.113.34.11   gke-Shrek01-default-pool-f9ecbfcc-dz9z   <none>           <none>
nginx-64b4f9bb85-l2bzd   1/1     Running   0          3m34s   10.113.32.5    gke-Shrek01-default-pool-f9ecbfcc-pdll   <none>           <none>
nginx-64b4f9bb85-xd7kw   1/1     Running   0          3m34s   10.113.33.9    gke-Shrek01-default-pool-f9ecbfcc-v67d   <none>           <none>

kubectl get nodes -o wide

NAME                                     STATUS   ROLES    AGE   VERSION          INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-Shrek01-default-pool-f9ecbfcc-dz9z   Ready    <none>   89m   v1.16.8-gke.15   192.168.13.4                 Container-Optimized OS from Google   4.19.109+        docker://19.3.1
gke-Shrek01-default-pool-f9ecbfcc-pdll   Ready    <none>   89m   v1.16.8-gke.15   192.168.13.2                 Container-Optimized OS from Google   4.19.109+        docker://19.3.1
gke-Shrek01-default-pool-f9ecbfcc-v67d   Ready    <none>   89m   v1.16.8-gke.15   192.168.13.3                 Container-Optimized OS from Google   4.19.109+        docker://19.3.1

Checks from Shrek02's pod:

root@nginx-5c66c56f55-8jwv2:/# echo ${MY_POD_IP} 
10.113.66.9

# internal load balancer
root@nginx-5c66c56f55-8jwv2:/# nc -vz 192.168.13.7 80
192.168.13.7: inverse host lookup failed: Unknown host
(UNKNOWN) [192.168.13.7] 80 (?) : Connection timed out

# intarnal load balancer's Cluster IP
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.213.60.14 80
10.213.60.14: inverse host lookup failed: Unknown host
(UNKNOWN) [10.213.60.14] 80 (?) : Connection timed out

# ClusterIP
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.213.34.24 80
10.213.34.24: inverse host lookup failed: Unknown host
(UNKNOWN) [10.213.34.24] 80 (?) : Connection timed out

# NodePort 
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.213.35.31 80
10.213.35.31: inverse host lookup failed: Unknown host
(UNKNOWN) [10.213.35.31] 80 (?) : Connection timed out

# Pod IP
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.113.34.11 80
10.113.34.11: inverse host lookup failed: Unknown host
(UNKNOWN) [10.113.34.11] 80 (?) open

root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.113.32.5 80
10.113.32.5: inverse host lookup failed: Unknown host
(UNKNOWN) [10.113.32.5] 80 (?) open

root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.113.33.9 80
10.113.33.9: inverse host lookup failed: Unknown host
(UNKNOWN) [10.113.33.9] 80 (?) open

Did I forget any step ? I don't find the mistake.

I'm trying to reproduce your issue, just to clear out: In the beggining of your question you say "What I need is that the 3 GKE clusters interact". Was that a typo or do you have a third cluster in the mix? — Will R.O.F., May 18 '20 at 17:50
I created two VPC, enabled VPC peering and instantaneously they are communicating (they exchanged the routes automatically). Does your cluster have "VPC-native (alias IP) Enabled" ? if you SSH into one node, type `toolbox` and try to ping a node in the other VPC, does it pings? This way we can first exclude the VPC part and then focus on the Clusters communications. — Will R.O.F., May 19 '20 at 19:22
Yep. In my lab, the comunication between the nodes it's ok, but the problem it's that I don't get comunication between pods from cluster A and cluster B services. The two cluster are VPC-native with both alias IP ( pods and services ). My first PoC with VPC-peering was with two instances in diferents VPC and it was right. — figarcia, May 21 '20 at 01:05
good, thanks for the info.. I'll analyze the communication between then, it's a very interesting subject. — Will R.O.F., May 21 '20 at 01:30

Will R.O.F. · Accepted Answer · 2020-05-22T17:04:20.327

I was able to connect pods to pods, pods to node-port and port to external loadbalancer with a similar environment, here's the caveat:

Internal load balancer is only allowed on VPC Peering under these circunstances:

Client virtual machine (VM) instances in the peer network are located in the same region as your internal load balancer
You configure global access. With global access configured, client VM instances from any region of the peered VPC network can access your internal TCP/UDP load balancer. Global access isn't supported for Internal HTTP(S) Load Balancing.

In order to use Internal Global Access with VPC Peering from different regions, you have two options:

If you know the internal load balancer name, you can use the following command:

$ gcloud compute forwarding-rules update <LB_NAME> \
   --region=<REGION> \
   --allow-global-access

And check it with:

gcloud compute forwarding-rules describe <LB_NAME> \
   --region=us-west1 \
   --format="get(name,region,allowGlobalAccess)"

Other easy way to change it if you have few internal LB in your region, is through GCP Load Balancing Page Click Here to Access:
- In the Name column, click your internal TCP/UDP load balancer of the cluster region (after clicking it you will see the subnet name as in my example below):

Then Click EDIT.
Click Frontend configuration
Click the Pencil to edit
Under Global Access, select Enable.
Click Done
Open and click Update to update the rule.
- Wait until the rule finish applying.
- After changing that, I was able from Shrek02 to target the Internal LB in Shrek01. (I'll give the example bellow).

NOTE:
ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. So you will not be routed to external access.

Reproduction:

I created two VPCs in following your parameters:

$ gcloud container clusters list
NAME         LOCATION        MASTER_VERSION  MASTER_IP        MACHINE_TYPE   NODE_VERSION       NUM_NODES  STATUS
shrek01      europe-west1-b  1.16.8-gke.15   XX.XXX.XX.XXX    g1-small       1.16.8-gke.15      3          RUNNING
shrek02      europe-west2-b  1.15.9-gke.24   XXX.XXX.XX.XXX   g1-small       1.15.9-gke.24      3          RUNNING

$ gcloud compute networks subnets list
NAME     REGION                   NETWORK  RANGE
shrek01  europe-west1             shrek01  192.168.13.0/24
shrek02  europe-west2             shrek02  192.168.14.0/24

$ gcloud compute networks peerings list-routes sh1-sh2 --network=shrek01 --region europe-west1 --direction=INCOMING
DEST_RANGE       TYPE                  NEXT_HOP_REGION  PRIORITY  STATUS
192.168.14.0/24  SUBNET_PEERING_ROUTE  europe-west2     1000      accepted
10.229.0.0/20    SUBNET_PEERING_ROUTE  europe-west2     1000      accepted
10.36.0.0/14     SUBNET_PEERING_ROUTE  europe-west2     1000      accepted

$ gcloud compute networks peerings list-routes sh2-sh1 --network=shrek02 --region europe-west2 --direction=INCOMING
DEST_RANGE       TYPE                  NEXT_HOP_REGION  PRIORITY  STATUS
192.168.13.0/24  SUBNET_PEERING_ROUTE  europe-west1     1000      accepted
10.154.0.0/20    SUBNET_PEERING_ROUTE  europe-west1     1000      accepted
10.24.0.0/14     SUBNET_PEERING_ROUTE  europe-west1     1000      accepted

After assuring my nodes can Ping between VPCs, I'll test the ingress and connections with this yamls

hello-1.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-1
spec:
 replicas: 3
 selector:
   matchLabels:
     app: hello-1
 template:
   metadata:
     labels:
       app: hello-1
   spec:
     containers:
     - name: hello-1
       image: gcr.io/google-samples/hello-app:1.0
       ports:
       - name: http
         containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
 name: hello-1-svc
spec:
 type: NodePort
 selector:
   app: hello-1
 ports:
   - protocol: TCP
     port: 80
     targetPort: 8080

hello-2.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-2
spec:
 replicas: 3
 selector:
   matchLabels:
     app: hello-2
 template:
   metadata:
     labels:
       app: hello-2
   spec:
     containers:
     - name: hello-2
       image: gcr.io/google-samples/hello-app:2.0
       ports:
       - name: http
         containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
 name: hello-2-svc
spec:
 type: NodePort
 selector:
   app: hello-2
 ports:
   - protocol: TCP
     port: 80
     targetPort: 8080

hello-ingress.yaml:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host:
      http:
        paths:
          - path: /
            backend:
              serviceName: hello-1-svc
              servicePort: 80
          - path: /v2
            backend:
              serviceName: hello-2-svc
              servicePort: 80

Take a look at the pods names and IP for pods, and nodes IP and loadbalancer/nodeport Ports.

o$ kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE    IP           NODE                                     NOMINATED NODE   READINESS GATES
hello-1-84d5994678-dx8dv   1/1     Running   0          140m   10.24.0.9    gke-shrek01-default-pool-5ffc38d7-bz35   <none>           <none>
hello-1-84d5994678-t74mn   1/1     Running   0          14m    10.24.1.3    gke-shrek01-default-pool-5ffc38d7-70sk   <none>           <none>
hello-1-84d5994678-zq7t2   1/1     Running   0          14m    10.24.2.9    gke-shrek01-default-pool-5ffc38d7-zfj6   <none>           <none>
hello-2-5c4f554ccc-b8j6f   1/1     Running   0          140m   10.24.0.10   gke-shrek01-default-pool-5ffc38d7-bz35   <none>           <none>
hello-2-5c4f554ccc-km4ph   1/1     Running   0          13m    10.24.1.4    gke-shrek01-default-pool-5ffc38d7-70sk   <none>           <none>
hello-2-5c4f554ccc-z4f6n   1/1     Running   0          13m    10.24.2.10   gke-shrek01-default-pool-5ffc38d7-zfj6   <none>           <none>

$ ubectl get services
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
hello-1-svc   NodePort    10.154.13.186   <none>        80:32186/TCP   140m
hello-2-svc   NodePort    10.154.4.214    <none>        80:32450/TCP   140m

$ kubectl get svc ingress-nginx-controller -n ingress-nginx
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.154.10.104   192.168.13.5   80:30112/TCP,443:32156/TCP   4h20m

$ kubectl get ingress
NAME            HOSTS   ADDRESS        PORTS   AGE
hello-ingress   *       192.168.13.5   80      98m

$ kubectl get nodes -o wide
NAME                                     STATUS   ROLES    AGE     VERSION          INTERNAL-IP    EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-shrek01-default-pool-5ffc38d7-70sk   Ready    <none>   2d19h   v1.16.8-gke.15   192.168.13.3   XX.XXX.XX.XXX    Container-Optimized OS from Google   4.19.109+        docker://19.3.1
gke-shrek01-default-pool-5ffc38d7-bz35   Ready    <none>   2d19h   v1.16.8-gke.15   192.168.13.2   XXX.XXX.XX.XXX   Container-Optimized OS from Google   4.19.109+        docker://19.3.1
gke-shrek01-default-pool-5ffc38d7-zfj6   Ready    <none>   2d19h   v1.16.8-gke.15   192.168.13.4   XX.XXX.X.XXX     Container-Optimized OS from Google   4.19.109+        docker://19.3.1

Now I'll connect to shrek02 cluster, create a pod and install curl:

project@cloudshell:~$ kubectl run ubuntu --image=ubuntu -it -- /bin/bash
root@ubuntu:/# apt update
root@ubuntu:/# apt install curl
root@ubuntu:/# exit

project@cloudshell:~$ kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE     IP          NODE                                     NOMINATED NODE   READINESS GATES
ubuntu   1/1     Running   1          2m51s   10.36.1.6   gke-shrek02-default-pool-a7a08ac8-0lrz   <none>           <none>

You can see we are in shrek02, now let's test the connection to shrek01 resources. Remember that kube-dns are only available inside cluster, so we will connect using the IPs:

project@cloudshell:~$ kubectl exec -it ubuntu -- /bin/bash
###Hello-1 POD:
root@ubuntu:/# curl 10.24.0.9:8080
Hello, world!
Version: 1.0.0
Hostname: hello-1-84d5994678-dx8dv
###Hello-2 POD:
root@ubuntu:/# curl 10.24.1.4:8080
Hello, world!
Version: 2.0.0
Hostname: hello-2-5c4f554ccc-km4ph

### HELLO-1-SVC USING NODE IP + NODEPORT:
root@ubuntu:/# curl 192.168.13.3:32186
Hello, world!
Version: 1.0.0
Hostname: hello-1-84d5994678-t74mn
### HELLO-2-SVC USING ANOTHER NODE IP + NODEPORT:
root@ubuntu:/# curl 192.168.13.2:32450
Hello, world!
Version: 2.0.0
Hostname: hello-2-5c4f554ccc-km4ph

### NOW LET'S TEST OUR INGRESS which routes "/" to hello-1 and "/v2" to hello-2:
root@ubuntu:/# curl 192.168.13.5/
Hello, world!
Version: 1.0.0
Hostname: hello-1-84d5994678-dx8dv

root@ubuntu:/# curl 192.168.13.5/v2 
Hello, world!
Version: 2.0.0
Hostname: hello-2-5c4f554ccc-b8j6f

I hope it help you troubleshoot your environment, and If you have any question let me know in the comments.

Thanks, I'll try it right now. – figarcia May 23 '20 at 13:45 — figarcia, May 23 '20 at 13:45

What am I doing wrong with private GKEs peering and Cloud NAT?

1 Answers1