在单kubernetes集群中，使用单实例或多副本的prometheus即可搭建好一个比较可用的监控平台。单该方案有很多的不足，有如下的缺点
a. 不能保存永久的监控数据（数据量太多会影响prometheus性能）
b. 多集群下grafana需要配置很多个数据源，比较麻烦
c. 节点异常会导致数据丢失， 不高可用
针对这些问题，目前的方案有：prometheus多副本+HA方案，数据量大是采用分层的prometheus联邦集群方案。终极方案：thanos
thanos部署方案有两种：sidecar和receive，github和官方文档都介绍得比较详细了。
Deployment with Sidecar:
image.png
Deployment with Receive:
image.png
k8s集群通过kind安装的，硬件为mac M1芯片
1. 通过kube-prometheus部署prometheus
https://github.com/prometheus-operator/kube-prometheus
下载好代码后，直接通过yaml的方式安装
kubectl apply --server-side -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f manifests/
2. 通过docker安装minio 对象存储
在kind环境下通过kubectl krew 安装minio时需要了问题，所以这儿通过docker来直接安装
docker run -d -p 9000:9000 -p 9001:9001 -v /Users/user/minio/data/:/data -v /Users/user/minio/config:/root/.minio minio/minio server --console-address ":9001" /data
通过过localhost:9001直接登录minio dashboard页面，9000端口给api用的，9001为dashboard
默认用户名密码为：minioadmin、minioadmin
登录后可以自己手动创建bucket
image.png
3. 通过kube-thanos安装thanos
https://github.com/search?q=kube-thanos
直接下载源码，通过yaml方式来安装。
由于thanos receive store等组件都是用statefulset来部署的，所以需要volume，这儿使用local pv
创建storageclass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
创建local pv
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /home
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kind-control-plane
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv1
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /home
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kind-control-plane
创建minio secrets
apiVersion: v1
kind: Secret
metadata:
name: thanos-objectstorage
namespace: thanos
type: Opaque
stringData:
thanos.yaml: |-
type: s3
config:
bucket: thanos
endpoint: 192.168.0.102:9000 # 这个ip为上面安装的minio的ip+port，也就是我电脑的ip地址
insecure: false
access_key: minioadmin
secret_key: minioadmin
kubectl create ns thanos
kubectl create -f manifests/
安装完后后，查看部署的内容如下：
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage (default) kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 7h39m
kubectl get pv -n thanos
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv 10Gi RWO Delete Bound thanos/data-thanos-receive-ingestor-default-0 local-storage 4h22m
local-pv1 10Gi RWO Delete Bound thanos/data-thanos-store-0 local-storage 4h22m
kubectl get pvc -n thanos
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-thanos-receive-ingestor-default-0 Bound local-pv 10Gi RWO local-storage 4h23m
data-thanos-store-0 Bound local-pv6 10Gi RWO local-storage 4h23m
kubectl get pods -n thanos
NAME READY STATUS RESTARTS AGE
thanos-query-778c77b5-w6jjc 1/1 Running 0 4h21m
thanos-receive-ingestor-default-0 1/1 Running 0 4h21m
thanos-receive-router-557ddd87b9-4bb95 1/1 Running 0 4h21m
thanos-store-0 1/1 Running 0 4h21m
指定port-forward打开thanos的页面
kubectl port-forward svc/thanos-query 9090:9090
配置prometheus的remotewrite为thanos receive的地址
# kube-prometheus/manifests/prometheus-prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.32.1
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- apiVersion: v2
name: alertmanager-main
namespace: monitoring
port: web
enableFeatures: []
externalLabels: {}
image: quay.io/prometheus/prometheus:v2.32.1
nodeSelector:
kubernetes.io/os: linux
podMetadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.32.1
podMonitorNamespaceSelector: {}
podMonitorSelector: {}
probeNamespaceSelector: {}
probeSelector: {}
replicas: 1
retention: 10d
resources:
requests:
memory: 200Mi
ruleNamespaceSelector: {}
ruleSelector: {}
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: 2.32.1
remoteWrite: #加入remoteWrite地址，注意替换为自己的地址
- url: "http://thanos-receive-router-ip:19291/api/v1/receive"
打开thanos query的页面
image.png
数据中的cluster、datacenter字段为promtheus中的external_label字段，在多个集群接入thanos时，用来区分不同的集群，非常有用。filed中的tenant_id从remoteWrite的header字段中提取，支持多租户。
网友评论