1

I want to use a kubernetes(cluster-0)with multiple kube-state-metrics to monitor multiple other kubernetes cluster(cluster-1,2,3,4)

In the (cluster-0), I split into multiple namespaces like this:

namespace: monitor

(here I will run prometheus to scrape metrics from other kube-state-metrics pod)

namespaces: cluster-0
(here I will run kube-state-metrics pod to monitor cluster-0
....
namespaces: cluster-4
(here I will run kube-state-metrics pod to monitor cluster-4)
...

And so on...

The problems here is how would I setup prometheus to auto scrape new kube-state-metrics since I will expand the kubernetes cluster very fast, or shrink it. And how to tell the different metrics between them, like the having a header ...

I have try using config but just locate single - targets: ['serviceIP:8080'] is not very good in my situation.

Lê Minh Quân
  • 191
  • 12
  • Have you considered using [Prometheus Self Discovery on Kubernetes](https://developer.sh/posts/prometheus-self-discovery)? Does it fit your use case? – Wytrzymały Wiktor Oct 16 '20 at 12:45

1 Answers1

1

As I already mentioned in my comment. What you need is the Prometheus Self Discovery on Kubernetes:

Kubernetes Self Discovery configurations allow retrieving scrape targets automatically, as and when new targets come up. The scraping is based on Kubernetes service names, so even if the IP address change (and they will), Prometheus can seamlessly scrape the targets.

Prometheus self discovery is based on Kubernetes labels and annotations explained here. This allows a great deal of granularity on choosing the applications to be scraped. It is also important here to understand the role field, since it defines the behavior of a scraping job. role defines the type of Kubernetes resource you want Prometheus to look for. Currently it can be endpoints, service, pod, node, or ingress. For example, if role is set to pod, Prometheus will discover a target for each pod and exposes their containers as targets.

With it you will be able to scrap metrics from new nodes when they come up. More details and examples can be found in the linked docs.

Wytrzymały Wiktor
  • 731
  • 3
  • 10