配置邮件示例
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- 192.168.19.55:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "rules/node_rules.yml"
# - "second_rules.yml"
prometheus.yml
groups:
- name: node-rules
rules:
- alert: node-up
expr: up == 0
for: 15s
labels:
severity: 1
team: node
annotations:
summary: "{{$labels.instance}}Instance has been down for more than 5 minutes"
node_rules.yml
global:
resolve_timeout: 5m
smtp_smarthost: 'smtp.163.com:25'
smtp_from: 'cfgitlab_admin@163.com'
smtp_auth_username: 'cfgitlab_admin@163.com'
smtp_auth_password: '1111111'
smtp_require_tls: false
templates:
- '/root/prom/alertmanager-0.26.0.linux-amd64/email.tmpl'
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: '12727@qq.com'
html: '{{ template "email.to.html" . }}'
send_resolved: true
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname']
alertmanager.yml
{{ define "email.to.html" }}
{{ if gt (len .Alerts.Firing) 0 }}{{ range .Alerts }}
@告警: <br>
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} 级<br>
告警类型: {{ .Labels.alertname }} <br>
故障主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ .StartsAt }} <br>
{{ end }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}{{ range .Alerts }}
@恢复: <br>
告警主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
恢复时间: {{ .EndsAt }} <br>
{{ end }}
{{ end }}
{{ end }}
email.tmpl
up指标用来监控主机是否宕机下线
告警规则的rule的数据 就是这里的查询语句查询出来的结果
内网代理发送邮件
#!/usr/bin/python3
import smtplib
import requests
from email.mime.text import MIMEText
def send_email():
#构建邮件内容
msg = MIMEText('主机要爆炸了 主机要爆炸了 主机要爆炸了 主机要爆炸了 主机要爆炸了 请速速撤离 ')
msg['Subject'] = 'you are so bad'
msg['From'] = 'cfgitlab_admin@163.com'
msg['To'] = '龙腾四海 运河就笑'
server = None
try:
#创建SMTP对象并配置代理服务器地址及端口号
server = smtplib.SMTP(host="smtp.163.com",port=25)
#开始TLS加密传输
server.starttls()
#登录到SMTP服务器(若有)
username = 'cfgitlab_admin@163.com'
password = '11111111'
server.login(username, password)
#发送邮件
server.sendmail('cfgitlab_admin@163.com', ['1227@qq.com','10372@qq.com','13652@qq.com','88888@qq.com'], msg.as_string())
print("邮件已成功发送")
except Exception as e:
print("邮件发送失败:", str(e))
finally:
# 关闭与SMTP服务器的连接
if server != None:
server.quit()
if __name__ == "__main__":
send_email()
python无代理发送邮件
import socks
import socket
import smtplib
from email.mime.text import MIMEText
socks.set_default_proxy(socks.SOCKS5, "10.30.90.15", 18)
socket.socket = socks.socksocket
smtp_server = "smtp.163.com"
smtp_port = "25"
sender_email = "cfgitlab_admin@163.com"
sender_password = "222222"
server = smtplib.SMTP(smtp_server, smtp_port)
server.login(sender_email, sender_password)
msg = MIMEText('这是测试邮件', _charset='utf-8')
msg['Subject'] = '测试邮件主题'
msg['From'] = sender_email
msg['To'] = '124627@qq.com'
server.sendmail(sender_email, msg['To'], msg.as_string())
print("邮件已成功发送!")
python代理代码发送
Alertmanager集成python自研告警组件
配置钉钉消息告警
global:
resolve_timeout: 5m
smtp_smarthost: 'smtp.163.com:25'
smtp_from: 'cfgitlab_admin@163.com'
smtp_auth_username: 'cfgitlab_admin@163.com'
smtp_auth_password: 'TCTJGF'
smtp_require_tls: false
templates:
- '/root/prom/alertmanager-0.26.0.linux-amd64/email.tmpl'
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'dingding.webhook1'
routes:
- receiver: 'dingding.webhook1'
continue: true #发送完钉钉消息后继续发送邮件
- receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: '12727@qq.com,1037715202@qq.com,423096959@qq.com,2841184943@qq.com'
html: '{{ template "email.to.html" . }}'
send_resolved: true
- name: 'dingding.webhook1'
webhook_configs:
- url: 'http://192.168.19.55:8060/dingtalk/webhook1/send'
send_resolved: true
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname']
alertmanager.yml
## Request timeout
# timeout: 5s
## Uncomment following line in order to write template from scratch (be careful!)
#no_builtin_template: true
## Customizable templates path
templates:
- contrib/templates/legacy/template.tmpl
## You can also override default template using `default_message`
## The following example to use the 'legacy' template from v0.3.0
#default_message:
# title: '{{ template "legacy.title" . }}'
# text: '{{ template "legacy.content" . }}'
## Targets, previously was known as "profiles"
targets:
webhook1:
url: https://oapi.dingtalk.com/robot/send?access_token=7f7fe25
# secret for signature
secret: 4ee16602f4108eb793475803cfa26a1be5210ccf1
# webhook2:
# url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
# webhook_legacy:
# url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
# Customize template content
# message:
# Use legacy template
# title: '{{ template "legacy.title" . }}'
# text: '{{ template "legacy.content" . }}'
# webhook_mention_all:
# url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
# mention:
# all: true
# webhook_mention_users:
# url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
# mention:
# mobiles: ['156xxxx8827', '189xxxx8325']
prometheus-webhook-dingtalk.yml
需单独安装 prometheus-webhook-dingtalk-2.1.0.linux-amd64组件
内网代理钉钉消息
内网中只有一台机器可以联网
监控的主机无法连接外部的机器
给prometheus-webhook-dingtalk进程设置代理地址环境变量 不能设置成系统变量,因为这样会影响系统上所有的https请求
添加进程环境变量
//在程序启动的时候注入多个环境变量
export https_proxy=http://ct:kXoNx@10.30.90.15:3128 && export http_proxy=http://cta:kgX3Nx@10.30.90.15:3128 && ./prometheus-webhook-dingtalk --config.file=config.yml
1.把代理地址做为环境变量注入到指定的进程中
//在程序启动的时候注入一个环境变量
export https_proxy=http://aaaaa:22222@10.30.90.1:3128 && ./prometheus-webhook-dingtalk --config.file=config.yml
//在程序启动的时候注入多个环境变量
export https_proxy=http://ct:kXoNx@10.30.90.15:3128 && export http_proxy=http://cta:k8TgX3Nx@10.30.90.15:3128 && ./prometheus-webhook-dingtalk --config.file=config.yml
2.查看进程的环境变量
ps -ef | grep webhook
cat /proc/5663/environ | tr '\0' '\n'
cat /proc/5663/comm
作者:不懂123