We have troubles with the configuration of two GCP components (the GCP Cloud Composer, the GCP Functions) under VPC Service Controls for a single project. We can configure them separately with VPC Service Controls, but as soon as we add the second component we experience troubles.
Once we have configured the Firewall rules and DNS settings for the Cloud Function, the Composer starts to fail. The Composer monitoring tool (for webserver, SQL server, etc.) shows that the composer is unhealthy.
What are we doing wrong?
The docs we've used:
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls
https://cloud.google.com/composer/docs/configuring-vpc-sc
For the Composer we use these settings: 3 node standard machine type image version: composer-1.16.3-airflow-1.10.15 python: v3 private IP Composer
The APIs what we have enabled for VPC Service Controls: Restricted Services: Google Cloud Dataproc API Cloud Functions API Google Cloud Pub/Sub API Cloud SQL API Google Cloud Storage API Google Compute Engine API Google Kubernetes Engine API Google Container Registry API Cloud Monitoring API Cloud Composer API Artifact Registry API
The FireWall settings (based on the documentation):
Name | Type | Targets | Filters | Protocols / Ports | Action | Priority |
---|---|---|---|---|---|---|
composer-egress-gke-cluster-all-port | Egress | Apply to all | IP ranges: 10.10.0.0/16 | tcp, udp | Allow | 1000 |
composer-egress-healthcheck | Egress | Apply to all | IP ranges: 130.211.0.0/22, 35.191.0.0/16 | tcp:80,443 | Allow | 1000 |
composer-egress-port53 | Egress | Apply to all | IP ranges: 0.0.0.0/0 | tcp:53, udp:53 | Allow | 1000 |
composer-egress-webserver | Egress | Apply to all | IP ranges: 172.31.251.0/24 | tcp:3306,3307 | Allow | 1000 |
restricted-google-apis | Egress | Apply to all | IP ranges: 199.36.153.4/30 | tcp:443 | Allow | 1000 |
composer-deny-all | Egress | Apply to all | IP ranges: 0.0.0.0/0 | all | Deny | 2000 |
composer-ingress-helthcheck | Ingress | Apply to all | IP ranges: 130.211.0.0/22, 35.191.0.0/16 | tcp:80,443 | Allow | 1000 |
ingress-iap | Ingress | Apply to all | IP ranges: 35.235.240.0/20 | tcp | Allow | 1000 |
gke-europe-west1-composer-dns-xxx-xxx-all | Ingress | gke-europe-west1-composer-dns-xxx-xxx-node | IP ranges: 10.124.0.0/14 | tcp;udp;esp;ah;sctp;icmp | Allow | 1000 |
gke-europe-west1-composer-dns-xxx-xxx-master | Ingress | gke-europe-west1-composer-dns-xxx-xxx-node | IP ranges: 172.16.8.0/28 | tcp:10250,443 | Allow | 1000 |
gke-europe-west1-composer-dns-xxx-xxx-vms | Ingress | gke-europe-west1-composer-dns-xxx-xxx-node | IP ranges: 10.10.0.0/16 | tcp:1-65535;udp:1-65535;icmp | Allow | 1000 |
The DNS settings (based on the documentation):
DNS name: googleapis.com. Type: Private
DNS name | Type | TTl(Seconds) | Data |
---|---|---|---|
*.googleapis.com. | CNAME | 300 | restricted.googleapis.com. |
googleapis.com. | SOA | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
googleapis.com. | NS | 21600 | ns-gcp-private.googledomains.com. |
DNS name: cloudfunctions.net. Type: Private
DNS name | Type | TTl(Seconds) | Data |
---|---|---|---|
*.cloudfunctions.net. | A | 300 | 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7 |
cloudfunctions.net | NS | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
cloudfunctions.net. | SOA | 21600 | ns-gcp-private.googledomains.com. |
DNS name: pkg.dev. Type: Private
DNS name | Type | TTl(Seconds) | Data |
---|---|---|---|
*.pkg.dev. | CNAME | 300 | pkg.dev. |
pkg.dev. | SOA | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
pkg.dev. | NS | 21600 | ns-gcp-private.googledomains.com. |
pkg.dev. | A | 300 | 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7 |
DNS name: gcr.io. Type: Private
DNS name | Type | TTl(Seconds) | Data |
---|---|---|---|
*.gcr.io. | CNAME | 300 | gcr.io. |
gcr.io. | SOA | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
gcr.io. | NS | 21600 | ns-gcp-private.googledomains.com. |
gcr.io. | A | 300 | 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7 |
Thank you for any idea or help what you can give.