I have a not-so-small cluster hosted on Google Kubernetes Engine (2 e2-standard nodes) with a couple of web services talking to each other. The architecture is composed by:
- 2 Cloud SQL instances, hosted on GCP
- 1 deployment and 1 statefulset, 1 replica each
- an Nginx Ingress Controller installation that exposes both apps (the deploy and the sts) over HTTPS
- a few internal tools such CertManager and others
The two apps talk to each other over the internal k8s DNS system, they connect to the relative Cloud SQL instance over the external IP and they send emails and push notifications to APNS and Firebase. They expose RESTful APIs for my mobile apps to consume.
In the bill breakdown I can see 1.5+ TB/month of egress traffic generated by Nginx ingress controller which I cannot explain. I expect the egress traffic to consist only in SQL connections and in the email messages and push notifications sent by my pod, since this is the only traffic actually generated in the cluster. I can also see the egress traffic consumed by the 2 apps but the sum of these two values is much smaller then the one consumed by Nginx.
Now I wonder if Google is charging me also the HTTP responses my pod generate towards the mobile apps consuming the APIs, because I cannot see any other reason why my cluster is so expensive.