1 prometheus说明

1.1 监控方案

前提：已安装kubernetes集群
监控服务：prometheus
资源采集：cadvisor+node-exporter+kube-state-metrics
告警：altermanager
展示：grafana

1.2 采集监控对象

1.2.1 cadvisor

kubernetes集群自带，主要采集node节点上的容器资源信息，接口地址：https://{api_server}:6443/api/v1/nodes/{node_name}/proxy/metrics/cadvisor

1.2.2 node-exporter

采集node节点资源信息，需安装

1.2.3 kube-state-metrics

采集kubernetes对象信息，如deployment、statefulset、daemonset等，需安装

1.2.4 自定义对象

在metadata里面添加prometheus自动发现的标识，示例如下：

metadata:
  annotations:
    # 采集的端口号
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
    # 默认路径为/metrics，如果路径正确可不填写
    promethues.io/path: "/metrics"

---

2 prometheus安装

2.1 创建获取集群接口权限的rbac角色

prometheus-rbac.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-role
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-role-bind
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-role
subjects:
- kind: ServiceAccount
  name: prometheus-sa
  namespace: kube-system

kubectl create -f prometheus-rbac.yml

2.2 prometheus配置文件

prometheus-configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-system
data:
  prometheus.yml: |
    global:
      scrape_interval:     60s
      scrape_timeout:      20s
      evaluation_interval: 30s
    
    # 容器启动后再把下面注解去掉，如果规则文件变化比较频繁，建议放在pvc盘里面，通过脚本更新
    #alerting:
    #  alertmanagers:
    #  - static_configs:
    #    - targets: ["127.0.0.1:9093"]
    
    #rule_files:
    #  - "/prometheus/conf/rules/base_rules.yml"
    
    scrape_configs:
    - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
    
    - job_name: 'kubernetes-nodes'
      kubernetes_sd_configs:
      - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
    
    - job_name: 'kubernetes-cadvisor'
      kubernetes_sd_configs:
      - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name
    
    - job_name: 'kubernetes-services'
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module: [http_2xx]
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name
    
    - job_name: 'kubernetes-ingresses'
      kubernetes_sd_configs:
      - role: ingress
      relabel_configs:
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: ${1}://${2}${3}
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        target_label: kubernetes_name
    
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

2.3 prometheus容器

注意：如果挂载了数据盘pvc，需要确认挂载的可写用户是否为nobody，因为prometheus镜像默认以nobody用户启动，nfs挂载模式问题不大，在阿里云云盘挂载上owner是root会导致没有写入权限，需要自定义prometheus的DockerFile以root或有权限写入pvc的用户身份运行

prometheus-statefulset.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: prometheus-statefulset
  name: prometheus
  namespace: kube-system
spec:
  replicas: 1
  serviceName: prometheus
  selector:
    matchLabels:
      app: prometheus
      usage: monitoring
  template:
    metadata:
      labels:
        app: prometheus
        usage: monitoring
    spec:
      containers:
      - image: prom/prometheus:latest
        imagePullPolicy: IfNotPresent
        name: prometheus
        securityContext:
          privileged: true
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus.yml"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention=30d"
        - "--web.enable-admin-api"
        - "--web.enable-lifecycle"
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - mountPath: "/prometheus"
          name: data
        resources:
          limits:
            cpu: "2"
            memory: 2000Mi
          requests:
            cpu: 300m
            memory: 300Mi
      serviceAccountName: prometheus-sa
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: prometheus-data

2.4 prometheus服务

prometheus-svc.yml

kind: Service
apiVersion: v1
metadata:
  labels:
    name: prometheus-svc
  name: prometheus
  namespace: kube-system
spec:
  # 若前面有ingress，可以使用ClusterIP模式+创建ingress对象访问
  type: NodePort
  ports:
  - port: 9090
    targetPort: 9090
    nodePort: 30003
  selector:
    app: prometheus
    usage: monitoring

kubectl create -f prometheus-svc.yml

---

3 prometheus告警规则文件

3.1 /prometheus/conf/rules/base_rules.yml

groups:
- name: pod.rules
  rules:
  - alert: PodCpuUsage
    expr: min(rate(container_cpu_usage_seconds_total{pod!="",container_name!="POD",container_name!=""}[3m]) * 100) by (instance,namespace,pod) > 90
    for: 1m
    labels:
        severity: CRITICAL
        object: pod
    annotations:
        summary: CPU使用率大于90%持续3分钟
        value: '{{ $value }}'
  - alert: PodMemoryUsage
    expr: max((max_over_time(container_memory_working_set_bytes{pod!="",container_name!="",container_name!="POD"}[5m]) / container_spec_memory_limit_bytes{pod!="",container_name!="",container_name!="POD"}) * 100 > 95 and container_spec_memory_limit_bytes{pod!="",container_name!="",container_name!="POD"} > 0) by (instance,namespace,pod)
    for: 1m
    labels:
        severity: CRITICAL
        object: pod
    annotations:
        summary: 内存使用率大于95%持续3分钟
        value: '{{ $value }}'
- name: node.rules
  rules:
  - alert: NodeCpuUsage
    expr: min((1 - rate(node_cpu_seconds_total{mode="idle"}[3m])) * 100) by (instance) > 90
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: CPU使用率大于90%持续3分钟
        value: '{{ $value }}'
  - alert: NodeMemoryUsage
    expr: min(100 - (max_over_time(node_memory_MemAvailable_bytes[5m]) / node_memory_MemTotal_bytes) * 100) by (instance) > 90
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 内存使用率大于90%持续3分钟
        value: '{{ $value }}'
  - alert: NodeLoadAverage
    expr: min(min_over_time(node_load5[5m])) by (instance) >= 8
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 平均负载>=8持续5分钟
        value: '{{ $value }}'
  - alert: NodeFileSystem
    expr: min(min_over_time(node_filesystem_avail_bytes{fstype!="rootfs",fstype!="tmpfs"}[5m]) / node_filesystem_size_bytes{fstype!="rootfs",fstype!="tmpfs"} * 100) by (instance,mountpoint) <= 15
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 分区使用率大于85%持续5分钟
        value: '{{ $value }}'
  - alert: NodeNfConntrack
    expr: min(min_over_time(node_nf_conntrack_entries[5m]) / node_nf_conntrack_entries_limit * 100) by (instance) > 80
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 防火墙跟踪表使用率大于80%持续5分钟
        value: '{{ $value }}'
  - alert: NodeNetworkReceive
    expr: min(rate(node_network_receive_bytes_total{device="eth0"}[5m])) by (instance) * 8 >= 966367641.6
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 网卡eth0下载流量大于900Mbps持续5分钟
        value: '{{ $value }}'
  - alert: NodeNetworkTransmit
    expr: min(rate(node_network_transmit_bytes_total{device="eth0"}[5m])) by (instance) * 8 >= 966367641.6
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 网卡eth0上传流量大于900Mbps持续5分钟
        value: '{{ $value }}'
  - alert: NodeDiskIO
    expr: min(rate(node_disk_io_time_seconds_total[5m])) by (instance,device) * 100 > 80
    for: 1m
    labels:
        severity: CRITICAL
        object: node
    annotations:
        summary: 磁盘IO使用率大于80%持续5分钟
        value: '{{ $value }}'

3.2 PromQL语法

3.2.1 时间范围查询

http_request_total{}[5m]
可选单位：
s - 秒
m - 分钟
h - 小时
d - 天
w - 周
y - 年

3.2.2 聚合操作

sum (求和)
min (最小值)
max (最大值)
avg (平均值)
stddev (标准差)
stdvar (标准差异)
count (计数)
count_values (对value进行计数)
bottomk (后n条时序)
topk (前n条时序)
quantile (分布统计)

3.2.3 内置函数

increase(v range-vector)：获取区间向量中的第一个和最后一个样本并返回其增长量
rate(v range-vector)：计算区间向量v在时间窗口内平均增长速率
irate(v range-vector)：计算区间向量v在时间窗口内平均增长速率（瞬时增长率，基于最后两个数据）
predict_linear(v range-vector, t scalar)：预测时间序列v在t秒后的值
abs(v instant-vector)： 返回输入向量的绝对值
sort(v instant-vector)： 按升序排列
sort_desc(v instant-vector)： 按降序排列

3.2.4 时间位移offset

查询5分钟前的样本数据：
http_request_total{} offset 5m
查询昨天1天内的样本数据：
http_request_total{}[1d] offset 1d

3.3 资源监控指标

3.3.1 CPU

container：

rate(container_cpu_usage_seconds_total{pod!="",container_name!="POD",container_name!=""}[3m]) * 100

node：

(1 - rate(node_cpu_seconds_total{mode="idle"}[3m])) * 100

3.3.2 内存

container：

container_memory_working_set_bytes{pod!="",container_name!="",container_name!="POD"}
container_spec_memory_limit_bytes{pod!="",container_name!="",container_name!="POD"}

node：

100 - (max_over_time(node_memory_MemAvailable_bytes[5m]) / node_memory_MemTotal_bytes) * 100

3.3.3 平均负载

container：

container_cpu_load_average_10s

node：

node_load1
node_load5
node_load15
machine_cpu_cores

3.3.4 网络IO

container：

rate(container_network_transmit_bytes_total{pod!="",container_name!="",container_name!="POD"}[5m])
rate(container_network_receive_bytes_total{pod!="",container_name!="",container_name!="POD"}[5m])

node：

rate(node_network_transmit_bytes_total{device=~"^(eth|em).*"}[5m])
rate(node_network_receive_bytes_total{device=~"^(eth|em).*"}[5m])

3.3.5 磁盘IO

node：

rate(node_disk_io_time_seconds_total[5m])

3.3.6 磁盘空间

container：

container_fs_usage_bytes
container_fs_limit_bytes

node：

node_filesystem_avail_bytes{fstype!="rootfs",fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="rootfs",fstype!="tmpfs"} * 100

3.3.7 防火墙跟踪表

node：

node_nf_conntrack_entries / node_nf_conntrack_entries_limit * 100

4 alertmanager

5 grafana