一. Kubernetes监控指标与监控方案

1.Kubernetes监控指标

(1).集群监控

• 节点资源利用率
• 节点数
• 运行Pods

(2).Pod监控

• Kubernetes指标
• 容器指标
• 应用程序

2. Kubernetes监控方案

监控方案	特点	适用
Zabbix	大量定制工作	大部分的互联网公司
open-falcon	功能模块分解比较细显得更复杂	系统和应用监控
cAdvisor+Heapster+InfluxDB+Grafana	简单	容器监控
cAdvisor/exporter+Prometheus+Grafana	扩展性好	容器，应用，主机全方面监控

二. 监控系统部署

heapster使用域名连接apiserver和influxDB, 需要内部DNS,如coredns
Heapster+InfluxDB+Grafana

Heapster+-influxDB-grafana.png

1. 部署influxDB

这里使用本地存储，如果要持久存储，可以使用PV和PVC

# cat influxdb.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-influxdb
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: influxdb
    spec:
      containers:
      - name: influxdb
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-influxdb-amd64:v1.1.1
        volumeMounts:
        - mountPath: /data
          name: influxdb-storage
      volumes:
      - name: influxdb-storage
        emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-influxdb
  name: monitoring-influxdb
  namespace: kube-system
spec:
  ports:
  - port: 8086
    targetPort: 8086
  selector:
    k8s-app: influxdb
# kubectl create -f influxdb.yaml
deployment.extensions/monitoring-influxdb created
service/monitoring-influxdb created

2. 部署heapster

heapster要连接apiserver获取每个节点的暴露监控指标,所以需要rbac授权和每个节点都开启监控指标暴露端口。

(1). 节点监控指标暴露端口

# cat /opt/kubernetes/cfg/kubelet.config
    ....
readOnlyPort: 10255
    ....

# systemctl restart kubelet.service

数据采集：
# curl 10.40.6.210:10255/metrics

(2). heapster与角色授权绑定

ServiceAccount heapster 绑定 ClusterRole角色cluster-admin，具有管理权限。

# cat sa_heapster_bind.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: heapster
    namespace: kube-system

# kubectl create -f sa_heapster_bind.yaml

(3). 部署heapster资源

# cat heapster.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-amd64:v1.4.2
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=influxdb:http://monitoring-influxdb:8086

# kubectl create -f heapster.yaml

3. 创建grafana

# cat grafana.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: grafana
    spec:
      containers:
      - name: grafana
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-grafana-amd64:v4.4.1
        ports:
          - containerPort: 3000
            protocol: TCP
        volumeMounts:
        - mountPath: /var
          name: grafana-storage
        env:
        - name: INFLUXDB_HOST
          value: monitoring-influxdb
        - name: GF_AUTH_BASIC_ENABLED
          value: "false"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: Admin
        - name: GF_SERVER_ROOT_URL
          value: /
      volumes:
      - name: grafana-storage
        emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-grafana
  name: monitoring-grafana
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port : 80
    targetPort: 3000
  selector:
    k8s-app: grafana

# kubectl create -f grafana.yaml
# kubectl get pod,svc -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
pod/coredns-56666cdc6b-27gw4                1/1     Running   0          12d
pod/coredns-56666cdc6b-g5cpw                1/1     Running   0          12d
pod/heapster-66687b8845-gdl5l               1/1     Running   0          18m
pod/kubernetes-dashboard-774f47666c-97c86   1/1     Running   1          17d
pod/monitoring-grafana-cd8b89587-fprxc      1/1     Running   0          3m24s
pod/monitoring-influxdb-864c767966-tn9fp    1/1     Running   0          63m

NAME                           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns               ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP,9153/TCP   12d
service/kubernetes-dashboard   NodePort    10.0.0.198   <none>        443:30899/TCP            17d
service/monitoring-grafana     NodePort    10.0.0.37    <none>        80:34611/TCP             3m24s
service/monitoring-influxdb    ClusterIP   10.0.0.14    <none>        8086/TCP                 63m

地址：http://10.40.6.210:34611