What does it mean to have more than one instance of Prometheus in Kubernetes

Question

Suppose I'm using a volume to persist my Prometheus data, I wonder if I can have more than one instance of it running to have high availability.

I believe only one instance of Prometheus must be in charge of writing to the tsdb series, and having more than one instance can cause race conditions and corrupt the data (and I think the reason its tsdb has a lockfile must be that).

So, the question is, can I have multiple instances of Prometheus running, and both point to a same data and use it as a high availability option? Or is there any other way to keep Prometheus always available during a rolling restart caused by configuration changes?

score 3 · Answer 1 · answered Sep 05 '20 at 21:10

3

You can also run two instances/replicas of Prometheus in parallel, both scraping targets and saving the data independently, then have another service like Thanos Query in front of them to query both and de-duplicate the results.

This setup involves more moving parts, but gives you more flexibility to achieve your data-availability and service reliability goals.

answered Sep 05 '20 at 21:10

Ali Sattari

179
4

Yes, Use Thanos is better then use federation :) – c4f4t0r Sep 05 '20 at 21:36
Would this mean that the data is transferred and stored twice? Even if Thanos Query de-duplicates it at the user's end there could be significant costs increase because of this, right? – peetasan Dec 14 '20 at 14:18
@peetasanthat it s true, data is scraped (almost) twice and storage will be double. But that is the usual cost of highly available systems (when it comes to data stores). You want to have duplicate things for the time that one of the two (or some of the many) fail and then the data is not duplicate for some time (until you recover the broken components). – Ali Sattari Dec 15 '20 at 15:33
Would I be better off then to use Cortex perhaps which offers deduplication by only reading from one instance of the HA Prometheus? – peetasan Dec 17 '20 at 10:19

Malgorzata · Answer 2 · 2020-05-27T14:37:43.337

Answering on your question having multiple prometheus instances which point to a same data and use it is not a high availability option. During pod/node restart data is not available for others instances which is useless. If you are considering have multiple Prometheus instances and make it HA. I advice you to use Prometheus federation. It is also a solution for simple scaling, where one master server collects metrics from different servers in different datacenters.

Setting this architecture up is quite simple. A master server is deployed with “targets” that contain a list of slave Prometheus server URLs, like this:

scrape_configs:
      - job_name: federated_prometheus
        honor_labels: true
        metrics_path: /federate
        params:
          match[]:
          - '{job="the-prom-job"}'
        static_configs:
          - targets:
            - prometheus-slave1:9090
            - prometheus-slave2:9090

The match[] param in the configuration instructs Prometheus to accumulate and store all the slave metrics for a specific job. You can also set it as a regex: {__name__=~”^job:.*”}. This will collect metrics from several different jobs that match the expression definition.

The Prometheus slaves should have the following configuration:

global:
  external_labels:
    slave: 1
  relabel_configs:
  - source_labels: [_prometheus_slave]
    action: keep
    regex: slave1

When data size is continuously growing, the storage drivers should be capable of handling growth in order to supply reliable historical data. The simple solution for this is using cloud elastic storage services such as AWS S3 or Google Storage as storage backends. These services provide endless capacity for Prometheus servers that need to maintain large amounts of data, such as servers connected to a large Kubernetes cluster or several clusters with one monitoring endpoint.

Prometheus itself provides a solution for advanced storage management - storage scalability with snapshots. Taking snapshots of Prometheus data and deleting the data using the storage retention configuration, users can have data older than X days or months, or larger than a specific size, available in real-time. They can also then store old data on a separate disk and have it available on demand.

Please take a look: promethus-federating, kubernetes-scaling-prometheus.

What does it mean to have more than one instance of Prometheus in Kubernetes

2 Answers2