Our product lives in a Kubernetes cluster on our server. It is not in production yet, so there are multiple instances running in the cluster for different purposes, each in its own namespace. I need to run some load tests on one of the namespaces and I need to monitor CPU usage meanwhile. We have Prometheus and Grafana for monitoring.
One of the objectives of these tests is to learn what load drives CPU usage to its maximum.
So I'm looking for a way to query the CPU usage of a namespace as a percentage.
Here is what I put together based on examples:
sum (rate (container_cpu_usage_seconds_total{namespace="$Namespace"}[1m])) / sum(kube_pod_container_resource_limits{resource="cpu", unit="core", namespace="$Namespace"}) * 100
However, something must be wrong with this solution because occasionally values over 100% show up on the dashboard. Thinking the units must be different, I tried to look up the exact specification of these metrics but I didn't succeed.
(Sadly, I don't even know much about how CPU usage is calculated and what a 100% actually means.)
I searched for metrics that could be used for this problem through a few exporters: cAdvisor, Node, kube-state-metrics and more. Even in this seemingly exhaustive article, which was brought to my attention, it is stated that the metric I'm looking for is an important one but no way is provided to query it.
Any help would be appreciated, thank you.