prometheus + grafana指标监控系统安装与配置

整体架构

1. promethueus: 一个开源的服务监控系统和时间序列数据库。附带丰富的数据采集器，支持告警处理。
2. grafana： 一款开源的数据可视化工具，可以非常轻松的将数据转成图表的展现形式来做到数据监控以及数据统计。支持promethueus、influxDB等多种数据源。

prometheus-architecture.png

基础安装

本文采用的主要安装方式为docker-compose

prometheus

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.37.6
    ports:
      - 9090:9090
    volumes:
      - ./config:/etc/prometheus
      - ./prom-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus/data'
      - '--web.console.libraries=/prometheus/console_libraries'
      - '--web.console.templates=/prometheus/consoles'
    user: '0'

grafana

1. docker-compose启动

   version: '3.8'
   
   services:
     grafana:
       image: grafana/grafana-oss:9.4.3
       ports:
         - 3000:3000
       volumes:
         - ./gfdata:/var/lib/grafana
         - TZ=Asia/Shanghai
       environment:
         - GF_SERVER_ROOT_URL=http://100.127.9.84:3000/
       user: '0'
   

2. 浏览器访问http://{host}:3000,使用admin/admin登录；

3. 根据提示修改密码；

4. Config -> Data Sources, 添加Prometheus数据源：

image-20230310120537493.png

监控配置

服务器

1. 每台服务器分别安装node-exporter

   wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
   tar -xvf node_exporter-1.5.0.linux-amd64.tar.gz & mv node_exporter-1.5.0.linux-amd64 node_exporter
   cd node_exporter
   nohup ./node_exporter >/dev/null 2>&1 &
   

2. prometheus.yml配置job，并重启prometheus

   scrape_configs:
     - job_name: "node"
       static_configs:
         - targets: ['192.168.0.184:9100','192.168.0.59:9100','192.168.0.182:9100','192.168.1.10:9100','192.168.3.220:9100']
   

3. grafana导入dashboard模板
   node-exporter-full_rev30.json

4. 效果展示

image-20230310132748412.png

mysql

1. 创建用户并配置权限

   CREATE USER 'exporter'@'%' IDENTIFIED BY 'xxxxxx' WITH MAX_USER_CONNECTIONS 3;
   GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
   

2. 安装mysqld-exporter

   version: '3.8'
   
   services:
     mysqld-exporter:
       image: prom/mysqld-exporter:v0.14.0
       ports:
         - "9104:9104"
       environment:
         - DATA_SOURCE_NAME=exporter:xxxxxx@(host:port)/
   

3. prometheus.yml配置job，并重启prometheus

   scrape_configs:
     - job_name: "mysql"
       static_configs:
         - targets:
           # All mysql hostnames to monitor.
           - 192.168.0.184:2206
       relabel_configs:
         - source_labels: [__address__]
           target_label: __param_target
         - source_labels: [__param_target]
           target_label: instance
         - target_label: __address__
           # The mysqld_exporter host:port
           replacement: 192.168.3.39:9104
   

4. grafana导入dashboard模板
   mysql-overview.json

5. 效果展示

image-20230310133226180.png

mongodb

1. 创建用户并配置权限

   db.getSiblingDB("admin").createUser({
       user: "exporter",
       pwd: "xxxxxx",
       roles: [
           { role: "clusterMonitor", db: "admin" },
           { role: "read", db: "local" }
       ]
   })
   

2. 下载mongodb-exporter二进制文件

   wget https://github.com/percona/mongodb_exporter/releases/download/v0.11.0/mongodb_exporter-0.11.0.linux-amd64.tar.gz
   tar -xvf mongodb_exporter-0.11.0.linux-amd64.tar.gz
   

3. 编写Dockerfile

   FROM alpine:3.4
   EXPOSE 9216
   
   RUN apk add --update ca-certificates
   COPY mongodb_exporter /bin/mongodb_exporter
   RUN chmod 755 /bin/mongodb_exporter
   
   CMD [ "mongodb_exporter" ]
   

4. 安装mongodb-exporter

   version: '3.8'
   
   services:
     mongodb-exporter:
       build:
         context: ./
         dockerfile: Dockerfile
       image: percona/mongodb_exporter:0.11.0
       ports:
         - 9216:9216
       environment:
         - MONGODB_URI=mongodb://exporter:xxxxxx@host:port/?authSource=admin
   

5. prometheus.yml配置job，并重启prometheus

   scrape_configs:
     - job_name: "mongodb"
       static_configs:
         - targets: ["192.168.3.39:9216"]
   

6. grafana导入dashboard模板
   mongodb-overview_rev1.json

7. 效果展示

image-20230310134222718.png

redis

1. 安装redis-exporter

   version: '3.8'
   
   services:
     redis-exporter:
       image: oliver006/redis_exporter:v1.48.0
       ports:
         - "9121:9121"
       environment:
         - REDIS_ADDR=redis://host:port # 连接集群中一个节点
         - REDIS_PASSWORD=xxxxxxx
         - REDIS_EXPORTER_INCL_SYSTEM_METRICS=true
   

2. prometheus.yml配置job，并重启prometheus

   scrape_configs:
     - job_name: 'redis_exporter_targets'
       static_configs:
         - targets:
           - redis://192.168.0.182:8001
           - redis://192.168.0.182:8002
           - redis://192.168.0.59:8001
           - redis://192.168.0.59:8002
           - redis://192.168.0.184:8001
           - redis://192.168.0.184:8002
       metrics_path: /scrape
       relabel_configs:
         - source_labels: [__address__]
           target_label: __param_target
         - source_labels: [__param_target]
           target_label: instance
         - target_label: __address__
           replacement: 192.168.3.39:9121
   
     ## config for scraping the exporter itself
     - job_name: 'redis_exporter'
       static_configs:
         - targets:
           - 192.168.3.39:9121
   

3. grafana导入dashboard模板
   redis-dashboard.json

4. 效果展示

image-20230310134847117.png

rocketmq

1. 构建rocketmq-exporter

   curl -O https://github.com/apache/rocketmq-exporter/archive/refs/heads/master.zip
   unzip master.zip
   cd rocketmq-exporter-master
   mvn package
   cd target
   

   或者下载我构建好的制品:

   rocketmq-exporter-0.0.2-SNAPSHOT-exec.jar

2. 编写Dockerfile

   FROM openjdk:11
   EXPOSE 5557
   
   COPY rocketmq-exporter-0.0.2-SNAPSHOT-exec.jar /usr/local/rocketmq-exporter.jar
   
   CMD [ "java", "-jar", "/usr/local/rocketmq-exporter.jar" ]
   

3. 安装rocketmq-exporter

   version: '3.8'
   
   services:
     rockermq-exporter:
       build:
         context: ./
         dockerfile: Dockerfile
       image: apache/rocketmq_exporter:0.0.2
       ports:
         - 5557:5557
       environment:
         - ROCKETMQ_CONFIG_NAMESRVADDR=host:port
         - ROCKETMQ_CONFIG_ROCKETMQ_VERSION=V4_4_0
   

4. prometheus.yml配置job，并重启prometheus

   scrape_configs:
     - job_name: 'rockermq_exporter'
       static_configs:
         - targets:
           - 192.168.3.39:5557
   

5. grafana导入dashboard模板
   rocketmq-dashboard.json

6. 查看效果

c530acf5ba82fa9196afdab371abc36.png

minIO

Prometheus + Grafana Minio监控

docker容器

cAdvisor + Prometheus + Grafana docker容器监控

elasticsearch

1. 安装elasticsearch-exporter

   version: '3.8'
   
   services:
     elasticsearch_exporter:
       image: prometheuscommunity/elasticsearch-exporter:v1.2.1
       command:
         - '--es.uri=http://user1:password1@192.168.0.59:9200' # 连接集群中的一个节点
         - '--es.all'
         - '--es.indices'
         - '--es.clusterinfo.interval=5m'
       ports:
         - "9114:9114"
   

2. prometheus.yml配置job，并重启prometheus

   scrape_configs:
     - job_name: 'elasticsearch_exporter'
       static_configs:
         - targets:
           - 192.168.3.39:9114
   

3. grafana导入dashboard模板
   elasticsearch-dashboard.json

4. 效果展示

image-20230320105112512.png

告警配置

grafana + prometheus告警配置