I have an application that is ~40 docker containers varying from NoSQL, RDBMS, C applications, Go apps, Python and so on, orchestrated using Kubernetes
, Its all running on GCP
. With a GLB(Load Balancer) at the frontend.
Now if I create a lot of replicas and give a lot of resources to these applications then everything runs properly. But if I give just enough resources then the frontend sometimes loads very slowly, the web application becomes unresponsive for sometime and then mysteriously comes back up again.
All this happens with no pod evictions or restarts.
When this happens I can see that the CPU/Memory are at 50%, so resources are not exhausted.
How to a go about debugging what is the reason for slowness? How to I calibrate which application requires how mush of resources?