How to setup alert in helm chart values file?

Question

I am trying to setup grafana alerts in helm chart.

For that, using the existing manually created alert from another grafana instance, which we want to keep in new grafana setup by default with helm chart.

As not able to find any export/import option, trying to edit manually. In this sample: helm-charts/values.yaml at main · grafana/helm-charts (github.com)

Found the alerts option in the values file link/ But not able to set below attributes

But, not able to find below option in the values file to set.

Please help. And also where to get datasourceUid and orgId values to set in alerts?

sample rule in chart:

alerting:
  rules.yaml:
    apiVersion: 1
    groups:
      - orgId: 1
        folder: "Performance Metrics"
        name: qtm
        interval: 60s
        rules:
          - uid: my_id_1
            condition: B
            title: qtm
            data:
              - refId: A
                queryType: ""
                relativeTimeRange:
                  from: 600
                  to: 0
                datasourceUid: "-100"
                model:
                  editorMode: code
                  expr: '100 * (sum(rate(n_bolt_api_usage{app="data-cruncher-read-replica", err="internal"}[60m])) or vector(0)) / (vector(1) or  (sum(rate(n_bolt_api_usage{app="data-cruncher-read-replica"}[60m]))))'
                  hide: false
                  intervalMs: 1000
                  legendFormat: __auto
                  maxDataPoints: 43200
                  range: true
                  refId: A
              - refId: B
                queryType: ""
                relativeTimeRange:
                  from: 0
                  to: 0
                datasourceUid: "-100"
                model:
                  conditions:
                    - evaluator:
                        params:
                          - 10
                        type: gt
                      operator:
                        type: and
                      query:
                        params:
                          - A
                      reducer:
                        params: []
                        type: last
                      type: query
                  datasource:
                    type: __expr__
                    uid: "-100"
                  expression: A
                  hide: false
                  intervalMs: 1000
                  maxDataPoints: 43200
                  refId: B
                  type: classic_conditions
            updated: "2022-09-22T18:27:51Z"
            noDataState: NoData
            execErrState: Alerting
            for: 1h
            annotations:
              summary: P1 QTM  High Internal Errors in API Usage

Getting error as:

logger=ngalert uid=my_id_1 org=1 version=4 attempt=0 now=2022-09-23T15:52:00Z t=2022-09-23T15:52:00.001235843Z level=error msg="failed to evaluate alert rule" results="[{Instance: State:Error Error:failed to execute conditions: invalid expression command type in 'A' EvaluatedAt:2022-09-23 15:52:00 +0000 UTC EvaluationDuration:1.232689ms EvaluationString: Values:map[]}]" duration=92.684µs

You appear to be missing a `|` from the end of the end of the `rules.yaml: ` line. It should be `rules.yaml: |` (I have edited the answer, the example I took it from was also missing it) — hardillb, Sep 23 '22 at 21:28

hardillb · Answer 1 · 2022-09-23T21:39:52.087

0

This a new feature in one of the latest release (9.1.x iirc)

And is discussed in the docs here https://grafana.com/docs/grafana/latest/administration/provisioning/#alerting

Which can be included in the gafana values file as follows:

e.g.

alerting:
  rules.yaml: |
    apiVersion: 1
    groups:
    ....
  contactpoints.yaml: |
    apiVersion: 1
    contactPoints:
    ....

See PR https://github.com/grafana/helm-charts/pull/1720 for details.

The orgId and datasourceUid you should already have from when you provisioned them earlier in your values.yaml (https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources)

As for how to get hold of the Alert query section, I think it's implied that you have to manually build it in the Web UI then extract it via the API (https://grafana.com/docs/grafana/latest/developers/http_api/alerting_provisioning/#route-get-alert-rule)

        condition: A
        # <list, required> list of query objects that should be executed on each
        #                  evaluation - should be obtained trough the API
       data:
       ...

edited Sep 23 '22 at 21:39

answered Sep 22 '22 at 11:04

hardillb

1,275
1
9
19

how to set the query above in the values file, can you show that attribute related to that? – Uday Kiran Reddy Sep 22 '22 at 11:27
It's laid out in this PR https://github.com/grafana/helm-charts/pull/1720 and there should be a commented out example in the new values.yaml file – hardillb Sep 22 '22 at 12:10
I am able to setup contact points and some configuration to alerts as mentioned already, if you see the first screenshot shared, where I highlighted metrics browser part, can you tell how to set that value atlease – Uday Kiran Reddy Sep 22 '22 at 12:24
Your edit should have been a comment. New alert rules created in the UI will not be written to the filesystem, as I mentioned, you will have to use the HTTP API (docs link in answer) to get the `data` section to go in the values file. – hardillb Sep 22 '22 at 15:53
I created an api key with admin role and ran like ( curl -H "Authorization: Bearer tokencreated" http://34.70.85.144/api/v1/provisioning/alert-rules ). But getting error as {"message": "Not Found"} Any suggestion, how to export the alert rule – Uday Kiran Reddy Sep 22 '22 at 16:30
You need to include a Alert UID on the end of that URL (you can't just list them all by the look of the doc). You can find the UID in the web UI URL when editing the Alert – hardillb Sep 22 '22 at 20:10
Able to export the json file of alert using this. curl -H "Authorization: Bearer tokenvalue" http://34.70.85.144/api/v1/provisioning/alert-rules/GIW56LnVz But, can you suggest how to import this? – Uday Kiran Reddy Sep 23 '22 at 14:33
JSON is valid YAML... but I'd probably re-type it as YAML so it's easier to read. – hardillb Sep 23 '22 at 14:34
I tried using json to yaml online converters and kept as part of helm chart values file, that is not loading properly. To simiplify, thought to use the POST /api/v1/provisioning/alert-rules, but not creating, it would be helpful if you provide such sampel for that – Uday Kiran Reddy Sep 23 '22 at 15:19
I added the sample chart and error coming with that from the data taken from exported curl command – Uday Kiran Reddy Sep 23 '22 at 15:56

How to setup alert in helm chart values file?

1 Answers1