mimir搭建记录

准备

需要有k8s集群
本机安装有helm,kubectl,docker

git clone https://github.com/wwq-2020/observability.git
cd observability/metrics/mimir
kubectl create ns mimir

安装

安装mimir

用于存储监控数据,执行告警规则检查和处理告警

helm install mimir-distributed ./mimir-distributed -n mimir

安装 grafana

helm install grafana ./grafana -n mimir

安装kube-prometheus

用于采集监控数据

helm install kube-prometheus-stack ./kube-prometheus-stack -n mimir

配置采集

kubectl apply -f servicemonitor.yaml

安装demo应用

cd demo
docker build -t  yourregistry/metricsdemo:v1 .
docker push yourregistry/metricsdemo:v1 

修改deployment.yaml中的image为yourregistry/metricsdemo:v1 ,然后执行

kubectl apply -f deployment.yaml

配置告警

新建文件rule,内容如下

{
        "name": "demo",
        "interval": "10s",
        "rules": [{
                "alert": "demo_to_slow",
                "expr": "histogram_quantile(0.95, sum(rate(demo_duration_bucket[5m])) by (le)) >0",
                "for": "1s",
                "labels": {
                        "serverity": "warning"
                },
                "annotations": {
                        "summary": "demo too slow"
                }
        }]
}

调用mimir ruler服务创建

 curl -XPOST k8sip:30091/prometheus/config/v1/rules/mimir -d@rule

创建alertmanagerconfig

curl -v -XPOST  k8sip:30092/api/v1/alerts -d '{"alertmanager_config":"route:\n  receiver: default-receiver\n  continue: false\n  group_wait: 10s\n  group_interval: 5m\n  repeat_interval: 1h\nreceivers:\n- name: default-receiver\ntemplates: []\n"}'

配置grafana

访问k8sip:30083
账号admin
密码通过如下获取

kubectl get secrets -n loki loki-grafana -ojsonpath='{.data.admin-password}'|base64 -d

依次点击齿轮,data sources,add data source

image.png

添加url值http://mimir-distributed-nginx.mimir.svc.cluster.local/prometheus
然后点击save & test

点击explore输入histogram_quantile(0.95, sum(rate(demo_duration_bucket[5m])) by (le)),点击run query,查到如下

image.png

查看告警

访问k8sip:30092/alertmanager/#/alerts