prometheus-operator 之企业邮箱告警

1.1.1.5 内网服务器 k8s master(修改默认告警系统)
1.1.1.1 外部机器(搭建邮件转发nginx)

1. nginx搭建

cat nginx.conf

stream{

    server{
        listen 4650;
        proxy_pass exmail.xxx.com:587;

          }
      }

2. 告警系统

此配置只适用企业邮箱，qq网易邮箱得自行网上寻找配置

2.1 alertmanager.yaml

mkdir -p /home/kubernetes/k8s/app/alertmanager_alert
cd /home/kubernetes/k8s/app/alertmanager_alert
touch alertmanager.yaml

#cat alertmanager.yaml
global:
  resolve_timeout: 5m
  smtp_smarthost: '1.1.1.1:4650'
  smtp_from: '123@xxx.com'
  smtp_auth_username: '123@xxx.com'
  smtp_auth_password: 'password'
  smtp_hello: 'xxx.com'
  smtp_require_tls: true

templates:
  - "./*.tmpl"

receivers:
- name: "email"
  email_configs:
  - to: '456@xxx.com'
    html: '{{ template "email.html" . }}'
    send_resolved: true
    tls_config:
      insecure_skip_verify: true

route:
  group_by:
  - job
  group_interval: 30s
  group_wait: 5s
  receiver: "email"
  repeat_interval: 12h

2.2 新增告警模板

cd /home/kubernetes/k8s/app/alertmanager_alert
mkdir template
touch template/email.tmpl

#cat touch template/email.tmpl
#抄的 http://www.manongjc.com/detail/26-ysorykovlufnizt.html(http://www.manongjc.com/detail/26-ysorykovlufnizt.html)


{{ define "email.html" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}

========= <span style=color:red;font-size:36px;font-weight:bold;> 监控告警 </span>=========<br>

<span style=font-size:20px;font-weight:bold;> 告警程序:</span>     Alertmanager <br>
<span style=font-size:20px;font-weight:bold;> 告警类型:</span>    {{ $alert.Labels.alertname }} <br>
<span style=font-size:20px;font-weight:bold;> 告警级别:</span>    {{ $alert.Labels.severity }} 级 <br>
<span style=font-size:20px;font-weight:bold;> 告警状态:</span>    {{ .Status }} <br>
<span style=font-size:20px;font-weight:bold;> 故障主机:</span>    {{ $alert.Labels.instance }} {{ $alert.Labels.device }} <br>
<span style=font-size:20px;font-weight:bold;> 告警主题:</span>    {{ .Annotations.summary }} <br>
<span style=font-size:20px;font-weight:bold;> 告警详情:</span>    {{ $alert.Annotations.message }}{{ $alert.Annotations.description}} <br>
<span style=font-size:20px;font-weight:bold;> 主机标签:</span>    {{ range .Labels.SortedPairs  }} <br> [{{ .Name }}: {{ .Value  | html }} ]{{ end }}<br>
<span style=font-size:20px;font-weight:bold;> 故障时间:</span>    {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>

========= = end =  =========<br>
<br>
<br>
<br>
<br>

<div>
    <div style=margin:40px>
        <p style=font-size:20px>xxxx</p>
        <p style=color:red;font-size:14px>
        (这是一封自动发送的邮件，请勿回复。)
        </p>
    </div>


    <div align=right style="margin:40px;border-top:solid 1px gray" id=bottomTime>
        <p style=margin-right:20px>
            xxx
        </p>
        <label style=margin-right:20px>
            {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 " }}<br>
        </label>
    </div>
</div>


{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
========= <span style=color:#00FF00;font-size:24px;font-weight:bold;> 告警恢复 </span>=========<br>
<span style=font-size:20px;font-weight:bold;> 告警程序:</span>    Alertmanager <br>
<span style=font-size:20px;font-weight:bold;> 告警主题:</span>    {{ $alert.Annotations.summary }}<br>
<span style=font-size:20px;font-weight:bold;> 告警主机:</span>    {{ .Labels.instance }} <br>
<span style=font-size:20px;font-weight:bold;> 告警类型:</span>    {{ .Labels.alertname }}<br>
<span style=font-size:20px;font-weight:bold;> 告警级别:</span>    {{ $alert.Labels.severity }} 级 <br>
<span style=font-size:20px;font-weight:bold;> 告警状态:</span>    {{   .Status }}<br>
<span style=font-size:20px;font-weight:bold;> 告警详情:</span>    {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}<br>
<span style=font-size:20px;font-weight:bold;> 故障时间:</span>    {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
<span style=font-size:20px;font-weight:bold;> 恢复时间:</span>    {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
========= = end =  =========
<br>
<br>
<br>
<br>

<div>
    <div style=margin:40px>
        <p style=font-size:20px>xxxx</p>
        <p style=color:red;font-size:14px>
        (这是一封自动发送的邮件，请勿回复。)
        </p>
    </div>


    <div align=right style="margin:40px;border-top:solid 1px gray" id=bottomTime>
        <p style=margin-right:20px>
            xxx
        </p>
        <label style=margin-right:20px>
            {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 " }}<br>
        </label>
    </div>
</div>
{{- end }}
{{- end }}
{{- end }}

2.3 删除默认配置，换上新配置

cd /home/kubernetes/k8s/app/alertmanager_alert
kubectl delete -n monitoring secrets alertmanager-main
kubectl  create   secret generic  alertmanager-main --from-file=alertmanager.yaml --from-file=./template/email.tmpl -n monitoring