1

I am using Prometheus for our monitoring and I have a lot of configs (our prometheus.yml main config file is 8000+ lines long).

I would like to divide this out into logical groupings so that it becomes much readable. I came to know that Prometheus doesn't support this and we can use configuration management systems like Ansible.

Has anyone done this with their Prometheus config file? If so, how did you do it?

PRS
  • 11
  • 2

2 Answers2

0

Ansible doesn't have a built in module for modifying YAML files. But you can use something like kwoodson.yedit to make any desired changes to the configuration file.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • I want to breakdown a single file of prometheus.yml into multiple like depending on projects and then using include(or similar) in main Prometheus config file (I can't seem to find any documentation like include in Prometheus). So, how can we do that? – PRS Jun 18 '20 at 11:29
  • @PRS You can't do that, because Prometheus doesn't support it! You could hire developers to add this functionality to Prometheus. But you can't do something that it doesn't support. – Michael Hampton Jun 18 '20 at 14:47
  • Okay, so switching static_config part for service discovery to file_sd to move the target lists to their own file is the only option to break down my prometheus.yml file? – PRS Jun 19 '20 at 07:31
0

Prometheus can work with only one main config file. There is file service discovery feature, but it is only for adding targets. If you want to divide your monitored targets into logical groups you can have multiple instances of Prometheus, one for each each group. This way each group will have its own Prometheus server and thus its own config file. There are a few other advantages, ie.:

  • each group is monitored independently, so ie. if you do some maintance on Prometheus server that monitors one group or it goes down for some reason, others are unaffected;
  • each Prometheus instance has its own TSDB, so it helps to keep the databases small which should improve the query performance;
  • you can choose an optimal location for each Prometheus instance, ie. AZ or VPC;
  • you can still have a central console if you set up federation or build dashboard in Grafana to display metrics from multiple Prometheus servers.
mac13k
  • 133
  • 6