13

Right now we're using Prometheus for our monitoring and we have a lot of config (our prometheus.yml main config file is 1400+ lines long).

I would like to split this out into logical groupings (maybe DEV/TEST/PROD?) but I can't seem to find any documentation as to how to use "includes" (or similar) in the Prometheus config file syntax.

Has anyone done this with their Prometheus config file? If so, how did you do it?

srkiNZ84
  • 531
  • 1
  • 6
  • 10
  • What about a script joining multiple files into one? – gxx Oct 05 '17 at 23:35
  • Yeah, I think that's what I'm going to have to do. But at best this is a "workaround". I wanted to be able to create a small config file, defining a "job_name" to test out config ("development" of scraping configs I suppose) and then just call "reload" to try it out. – srkiNZ84 Oct 06 '17 at 04:07

1 Answers1

9

The Prometheus config file (and other config files in the ecosystem) explicitly do not support any form of templating. Instead this is left up to your configuration management system to handle.

Additionally it sounds a bit unusual that you have dev/test/prod sections in your config file. Usually a) you'd have a Prometheus per environment and b) the main difference between these Prometheus servers would be a different value for the env label in your external_labels.

brian-brazil
  • 3,904
  • 1
  • 20
  • 15
  • Doesn't this violate the "single pane of glass" idea? How would we be abele to compare DEV to PROD metrics if we had separate instances per environment? Should we be using federated Prometheus for this use case? – srkiNZ84 Oct 20 '17 at 02:40
  • The use case is we have separate DEV/TEST/PROD Kubernetes clusters. For each cluster we make use of "service discovery" to get all the metrics from the Service and Pod (container) objects. – srkiNZ84 Oct 20 '17 at 02:43
  • 1
    Prometheus doesn't have a single pane of glass idea, that doesn't scale well for anything beyond the tiniest of systems. Even the metrics of Prometheus itself is too big for a single pane of glass, it's more like 4-5. The usual approach would be to use datasource templating in Grafana, and you can compare dashboards side by side. – brian-brazil Oct 20 '17 at 07:12
  • I had a similar problem had to resort to script file and templating. I use ECS and my prometheus docker container will bootstrap container by generating the prometheus.yml file before starting the prometheus server. – Subtubes Jun 25 '21 at 15:30