I'm struggling with Microsoft Teams/Prometheus integration on K8s cluster. I used helm to start all components. I have correctly working Prometheus and Alertmanager. It seems that all works fine. Prometheus communicate with Alertmanager. Then prometheus-msteams receives POST alert from Alert Manager and it should send it to a Microsoft Teams Channel but it's not.
2022/03/01 06:49:38 [DEBUG] POST https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
2022/03/01 06:50:08 [ERR] POST https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx request failed: Post https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx: dial tcp 42.12.12.542:443: i/o timeout
30s and timeout. I thought that it may be a proxy issue. So I added extraEnv parameter to config map, restarted pod but nothing changed. So my configuration looks like:
apiVersion: v1
data:
connectors.yaml: |
connectors:
- alertmanager-warning: https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
- alertmanager-critical: https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
extraEnvs:
HTTPS_PROXY: http://my-proxy.com:911
kind: ConfigMap
metadata:
I also loged into container to check if /etc/config/connectors.yaml is OK. I'm afraid that this extraEnvs doesn't work somehow. From K8s worker node I tried manually (with curl) post some test json to MS Chanel and without proxy it hanged. When I exported HTTPS_PROXY var the message was sucessfully created in MS Teams Chanel.
export HTTPS_PROXY=http://my-proxy.com:911
curl -X POST -d @test.json https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
Do you have any idea what can cause the problem? Is this HTTPS_PROXY env should be listed when I type printvenv in prometheus-msteams container?
