编写配置文件

prometheus.yml

cat <<EOF > ./data/prometheus.yml

# 全局配置
global:
  scrape_interval:     15s   # 多久 收集 一次数据
  evaluation_interval: 30s   # 多久评估一次 规则
  scrape_timeout:      10s   # 每次 收集数据的 超时时间

scrape_configs:
  - job_name: "apisix"
    scrape_interval: 15s
    metrics_path: "/apisix/metrics"
    static_configs:
      - targets: ["10.3.23.191:9907"]
EOF

这里配置的scrape_configs是为了抓取APISIX的指标。如果你没有抓取任务，这里可以去掉

grafana.ini
这里暂时先不做配置。
grafana.ini默认有很多配置，我们等启动以后再从容器中copy出来，因为后续需要修改此配置文件。
docker cp grafana:/etc/grafana/grafana.ini ./data/grafana/grafana.ini

编写docker-compose文件

cat <<EOF > docker-compose.yml
version: '3'
services:
  prometheus:
    container_name: "prometheus"
    image: "prom/prometheus:v2.41.0"
    user: root
    restart: always
    ports:
      - 9911:9090
    volumes:
      - ./data/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./data/prometheus:/prometheus

  grafana:
    container_name: "grafana"
    image: "grafana/grafana:9.3.2"
    user: root
    restart: always
    ports:
      - 9912:3000
    restart: always
    volumes:
      - ./data/grafana/data:/var/lib/grafana
      - ./data/grafana/log:/var/log/grafana
EOF

请注意，一定要加入user: root，否则启动时prometheus和grafana都会遇到permission denied相关的错误。

grafana相关文件的路径如下，如有需要可以映射到宿主机：
https://grafana.com/docs/grafana/latest/setup-grafana/configure-docker/
GF_PATHS_CONFIG /etc/grafana/grafana.ini
GF_PATHS_DATA /var/lib/grafana
GF_PATHS_HOME /usr/share/grafana

GF_PATHS_LOGS /var/log/grafana
GF_PATHS_PLUGINS /var/lib/grafana/plugins
GF_PATHS_PROVISIONING /etc/grafana/provisioning

启动测试

docker-compose up -d

访问prometheus
http://10.3.23.191:9911/

image.png

prometheus 启动成功

访问 grafana
http://10.3.23.191:9912/

image.png

默认登录密码 admin/admin，第一次登录进去后需要立即修改密码

image.png

说明grafana启动成功。

映射Grafana配置文件

grafana容器启动以后，通过以下命令把grafana.ini文件copy出来

docker cp grafana:/etc/grafana/grafana.ini  ./data/grafana/grafana.ini

重新修改docker-compose.yml文件

cat <<EOF > docker-compose.yml
version: '3'
services:
  prometheus:
    container_name: "prometheus"
    image: "prom/prometheus:v2.41.0"
    user: root
    restart: always
    ports:
      - 9911:9090
    volumes:
      - ./data/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./data/prometheus:/prometheus

  grafana:
    container_name: "grafana"
    image: "grafana/grafana:9.3.2"
    user: root
    restart: always
    ports:
      - 9912:3000
    restart: always
    volumes:
      - ./data/grafana/data:/var/lib/grafana
      - ./data/grafana/grafana.ini:/etc/grafana/grafana.ini
      - ./data/grafana/log:/var/log/grafana
EOF

重启即可。

APISIX 代理grafana以后出现问题：

image.png

反向代理Grafana以后，由于增加了路径前缀：http://10.3.23.191:9906/grafana/

docker cp grafana:/etc/grafana/grafana.ini ./data/grafana/grafana.ini

重启grafana
docker-compose restart

常见问题

prometheus | ts=2023-01-16T01:09:43.681Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus | panic: Unable to create mmap-ed active query log

grafana | mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
grafana | GF_PATHS_DATA='/var/lib/grafana' is not writable.
grafana | You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later

https://cloud.tencent.com/developer/ask/sof/1522407

第一次登录修改密码：