Answering on your question having multiple prometheus instances which point to a same data and use it is not a high availability option. During pod/node restart data is not available for others instances which is useless. If you are considering have multiple Prometheus instances and make it HA. I advice you to use Prometheus federation. It is also a solution for simple scaling, where one master server collects metrics from different servers in different datacenters.
Setting this architecture up is quite simple. A master server is deployed with “targets” that contain a list of slave Prometheus server URLs, like this:
scrape_configs:
- job_name: federated_prometheus
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{job="the-prom-job"}'
static_configs:
- targets:
- prometheus-slave1:9090
- prometheus-slave2:9090
The match[] param in the configuration instructs Prometheus to accumulate and store all the slave metrics for a specific job. You can also set it as a regex: {__name__=~”^job:.*”}
. This will collect metrics from several different jobs that match the expression definition.
The Prometheus slaves should have the following configuration:
global:
external_labels:
slave: 1
relabel_configs:
- source_labels: [_prometheus_slave]
action: keep
regex: slave1
When data size is continuously growing, the storage drivers should be capable of handling growth in order to supply reliable historical data. The simple solution for this is using cloud elastic storage services such as AWS S3 or Google Storage as storage backends. These services provide endless capacity for Prometheus servers that need to maintain large amounts of data, such as servers connected to a large Kubernetes cluster or several clusters with one monitoring endpoint.
Prometheus itself provides a solution for advanced storage management - storage scalability with snapshots. Taking snapshots of Prometheus data and deleting the data using the storage retention configuration, users can have data older than X days or months, or larger than a specific size, available in real-time. They can also then store old data on a separate disk and have it available on demand.
Please take a look: promethus-federating, kubernetes-scaling-prometheus.