Alertmanager 实现钉钉告警

Alertmanager 实现钉钉告警

打开电脑版钉钉创建机器人

WebHook 接收器

{ "receiver": "webhook", "status": "firing", "alerts": [ { "status": "firing", "labels": { "alertname": "NodeMemoryUsage", "beta_kubernetes_io_arch": "amd64", "beta_kubernetes_io_os": "linux", "instance": "node1", "job": "nodes", "kubernetes_io_arch": "amd64", "kubernetes_io_hostname": "node1", "kubernetes_io_os": "linux", "team": "node" }, "annotations": { "description": "node1: Memory usage is above 30% (current value is: 42.097619438581596)", "summary": "node1: High Memory usage detected" }, "startsAt": "2022-03-02T02:13:19.69Z", "endsAt": "0001-01-01T00:00:00Z", "generatorURL": " "fingerprint": "8cc4749f998d64dd" } ], "groupLabels": { "instance": "node1" }, "commonLabels": { "alertname": "NodeMemoryUsage", "beta_kubernetes_io_arch": "amd64", "beta_kubernetes_io_os": "linux", "instance": "node1", "job": "nodes", "kubernetes_io_arch": "amd64", "kubernetes_io_hostname": "node1", "kubernetes_io_os": "linux", "team": "node" }, "commonAnnotations": { "description": "node1: Memory usage is above 30% (current value is: 42.097619438581596)", "summary": "node1: High Memory usage detected" }, "externalURL": " "version": "4", "groupKey": "{}/{team=\"node\"}:{instance=\"node1\"}", "truncatedAlerts": 0}

我这里实现了一个简单的 webhook 程序，代码仓库地址：​​webhook 的地址，该地址会带一个 acess_token 的参数，该参数下面也会使用：

接下来我们需要将 webhook 服务部署到集群中，对应的资源清单如下：

配置完成后，直接创建上面的资源对象即可：

☸ ➜ kubectl apply -f promoter.yaml☸ ➜ kubectl get pods -n kube-mon -l app=promoterNAME READY STATUS RESTARTS AGEpromoter-67c5795c4c-7mlvq 1/1 Running 3 (34m ago) 3d16h

部署成功后，现在我们就可以给 AlertManager 配置一个 webhook 了，在上面的配置中增加一个路由接收器。

routes: - receiver: webhook group_wait: 10s group_by: ['instance'] match: team: nodereceivers:- name: 'webhook' webhook_configs: - url: ' send_resolved: true

我们这里配置了一个名为 webhook 的接收器，地址为：​​webhook 的接收程序的 Service 地址。

然后我们可以更新 AlertManager 和 Prometheus 的 ConfigMap 资源对象，更新完成后，隔一会儿执行 reload 操作是更新生效，如果有报警触发的话，隔一会儿关于这个节点文件系统的报警就会被触发了，由于这个报警信息包含一个​​team=node​​ 的 label 标签，所以会被路由到 webhook 这个接收器中，也就是上面我们自定义的这个 webhook，触发后可以观察这个 Pod 的日志：

☸ ➜ kubectl logs -f promoter-5dbd47798c-bnjqm -n kube-monts=2022-03-07T01:38:08.001Z caller=main.go:58 level=info msg="Staring Promoter" version="(version=0.2.3, branch=HEAD, revision=0a9cf8fc9bd55d1d2d47d181867135914927c2fc)"ts=2022-03-07T01:38:08.001Z caller=main.go:59 level=info build_context="(go=go1.17.8, user=root@91adc4eacff7, date=20220305-05:40:54)"ts=2022-03-07T01:38:08.001Z caller=main.go:127 level=info component=configuration msg="Loading configuration file" file=/etc/promoter/config.yamlts=2022-03-07T01:38:08.002Z caller=main.go:138 level=info component=configuration msg="Completed loading of configuration file" file=/etc/promoter/config.yamlts=2022-03-07T01:38:08.003Z caller=main.go:88 level=info msg=Listening address=:808

可以看到 POST 请求已经成功了，同时这个时候正常来说就可以收到一条钉钉消息了：

1.创建钉钉机器人

打开电脑版钉钉，创建一个群，创建自定义机器人，按如下步骤创建

​​怎么添加自定义机器人？-钉钉帮助中心您好，群主/群成员可以在电脑端通过如下的路径添加自定义机器人：步骤一：【电脑钉钉 】-【群聊】-【群设置】-【智能群助手】-【添加更多】-【添加机器人】-【自定义】-【添加】，编辑机器人名称和选择添加的群组。完成必要的安全设置（至少选择一种zxvf prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz

prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz压缩包所在的百度网盘地址如下：

链接：​​​​

提取码：d59o

cd prometheus-webhook-dingtalk-0.3.0.linux-amd64

启动钉钉报警插件(关键词不要忘记了，这是上面已经创建好了的 profile="cluster1)

nohup ./prometheus-webhook-dingtalk --web.listen-address="0.0.0.0:8060" --ding.profile="cluster1=&

[root@master prometheus-webhook-dingtalk-0.3.0.linux-amd64]# tail -f nohup.out level=info ts=2021-11-23T11:47:41.638047928Z caller=main.go:37 msg="Starting prometheus-webhook-dingtalk" version="(version=0.3.0, branch=HEAD, revision=4a7dee0be14073aba1ea2eed80acbb515564f664)"level=info ts=2021-11-23T11:47:41.63812183Z caller=main.go:57 msg="Using default template"level=info ts=2021-11-23T11:47:41.638150328Z caller=main.go:62 msg="Using following dingtalk profiles: map[cluster1:ts=2021-11-23T11:47:41.638256519Z caller=main.go:83 msg="Listening on address" address=0.0.0.0:8060

对原来的文件做备份 cp alertmanager-cm.yaml alertmanager-cm.yaml.bak

3.alertManager对接webhook插件

[root@master ~]# kubectl get svc -n monitorNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEprometheus-webhook-dingtalk NodePort 10.233.6.147 8060:30481/TCP 3m57s[root@master prometheus]# vim alertmanager-dingding-configmap.yaml receivers: - name: default-receiver email_configs: - to: "xxx@163.com" - name: dingding webhook_configs: - url: send_resolved: true[root@master prometheus]# kubectl exec -it dns-test sh/ # nslookup prometheus-webhook-dingtalk.monitorServer: 169.254.25.10Address 1: 169.254.25.10Name: prometheus-webhook-dingtalk.monitorAddress 1: 10.233.6.147 prometheus-webhook-dingtalk.monitor.svc.cluster.local

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。