ELK钉钉机器人告警
原创
©著作权归作者所有:来自51CTO博客作者阚小看的原创作品,请联系作者获取转载授权,否则将追究法律责任
elk服务器安装钉钉插件
创建个文件夹存放插件
mkdir -p /data/prometheus/dingtalk
下载prometheus-webhook-dingtalk
wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.4.0/prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz
解压
tar -vxzf prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz
移动dingtalk下
mv prometheus-webhook-dingtalk-1.4.0.linux-amd64 /usr/local/prometheus-webhook-dingtalk
配置服务
进入system下
cd /usr/lib/systemd/system
创建服务文件
vim prometheus-webhook-dingtalk.service
添加下面内容
[Unit]
Description=https://github.com/timonwong/prometheus-webhook-dingtalk/releases/
After=network-online.target
[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --config.file=/usr/local/prometheus-webhook-dingtalk/config.example.yml
[Install]
WantedBy=multi-user.target
重新加载服务配置
查看服务状态
systemctl status prometheus-webhook-dingtalk
设置开机自启
systemctl enable prometheus-webhook-dingtalk
钉钉添加机器人
取个名字,选择安全设置,这里我选择IP地址
不知道填什么IP地址?先随便填个保存,会生成个Webhook地址,复制下来
回到 ELK 服务器输入下面测试命令
把后面的地址换成上面钉钉机器人生成的地址
curl -H "Content-Type: application/json" -d '{"msgtype":"text","text":{"content":"hello,我是告警机器人!!!"}}' https://oapi.dingtalk.com/robot/send?access_token=ff488d860cc615bd8d56aa967bb5eabf64564c3d76451819e78c7387e0198917
会看见报错了,然后将提示的ip添加到钉钉机器人里面
再执行上面的测试命令,可以看见机器人成功了
ELK Watcher告警设置
创建个Watcher
Watch JSON配置如下
每分钟执行一次
size:1,保存一条查到的数据
查询过去一分钟里日志带ERROR的记录大于等于1条,就钉钉机器人告警
并输出报错日志的_index、source、traceId、spanId、tpye字段的信息
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"easyspeed-cloud-logs-*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 1,
"query": {
"bool": {
"must": [
{
"match": {
"level": "ERROR"
}
},
{
"range": {
"@timestamp": {
"gte": "now-1m",
"lte": "now"
}
}
}
]
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 1
}
}
},
"actions": {
"Watcher的ID": {
"webhook": {
"scheme": "https",
"host": "oapi.dingtalk.com",
"port": 443,
"method": "post",
"path": "/robot/send",
"params": {
"access_token": "钉钉生成的"
},
"headers": {},
"body": """{"msgtype":"text","text":{"content":"TEST发现ERROR,错误信息如下:
_index:{{ctx.payload.hits.hits.0._index}},
source:{{ctx.payload.hits.hits.0._source.source}},
traceId:{{ctx.payload.hits.hits.0._source.traceId}} ,
spanId:{{ctx.payload.hits.hits.0._source.spanId}},
type:{{ctx.payload.hits.hits.0._source.type}}
详情信息请登录查看:http://logs.hkeasyspeed.com/ "}}"""
}
}
}
}