OpenShift 4 使用CronJob备份etcd数据库

etcd & kubernetes.png

我们都知道 etcd 是 OpenShift/Kubernetes 集群里最为重要的一个组件，用于存储集群所有资源对象的状态。因此，对 etcd 数据进行备份同样的也非常重要。
通常对数据进行备份都是通过定时执行脚本来实现，接下来我们使用 Kubernetes 的 CronJob 来备份 OpenShift 4 的 etcd。

下边的方法只适合 OpenShift 4，OpenShift 3 及原生 Kubernetes 不能直接使用。

创建Cronjob备份etcd数据

1. 创建专门用于备份etcd数据的namespace

# oc create namespace openshift-etcd-backup
# oc project openshift-etcd-backup

2. 备份的Cronjob yaml

# cat > etcd-backup-cronjob.yaml << "EOF"
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: openshift-etcd-backup
  # 与前面创建的 namespace 一致
  namespace: openshift-etcd-backup
spec:
  # 设置备份周期
  schedule: "*/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: openshift-etcd-backup
            # 执行备份动作的image，使用 etcd-member pod的image(/etc/kubernetes/manifests/etcd-member.yaml)
            image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31a7eaddd0eb02e46663457f77bf8a327603dade31a3e92e9d7760580034f885
            # 备份的工作官网已经有现成的脚本可以实现
            command: ["/bin/sh"]
            args: ["-c", "cd /backup && /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup/$(date +%Y-%m-%d_%H:%M:%S_%Z).db"]
            volumeMounts:
            - mountPath: /usr/local/bin
              name: script-tools
              readOnly: true
            - mountPath: /etc/kubernetes
              name: kubernetes-dir
              readOnly: true
            - mountPath: /backup
              name: etcd-backup-pvc
            securityContext:
              privileged: true
            resources:
              requests:
                memory: 500Mi
                cpu: 300m
          restartPolicy: OnFailure
          nodeSelector:
            # 需要在 master 节点上执行备份任务
            node-role.kubernetes.io/master: ""
          tolerations:
          - effect: NoSchedule
            operator: Exists
          hostNetwork: true
          volumes:
          - name: script-tools
            hostPath:
              # 备份需要用到的工具，在 master 节点上
              path: /usr/local/bin
              type: DirectoryOrCreate
          - name: kubernetes-dir
            hostPath:
              # 证书以及 yaml 文件所在目录
              path: /etc/kubernetes
              type: DirectoryOrCreate
          - name: etcd-backup-pvc
            persistentVolumeClaim:
              # 备份数据存放在持久化存储
              claimName: etcd-backup-pvc
              readOnly: false
EOF

3. 存放etcd备份数据的存储：pv/pvc

# nfs-pv
# cat > etcd-backup-pv.yaml << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  finalizers:
  - kubernetes.io/pv-protection
  name: etcd-backup-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  nfs:
    path: /srv/nfs/ocp4-cluster1-etcd-backup-pv
    server: 10.72.35.249
EOF
# pvc
# cat > etcd-backup-pvc.yaml << EOF
apiVersion: v1 
kind: PersistentVolumeClaim
metadata:
  name: etcd-backup-pvc
  namespace: openshift-etcd-backup
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
EOF

4. 创建上面定义的3个资源对象即可

# oc create -f etcd-backup-pv.yaml
# oc create -f etcd-backup-pvc.yaml
# oc create -f etcd-backup-cronjob.yaml

5. 需要对openshift-etcd-backup的default这个serviceaccount增加权限

# oc adm policy add-scc-to-user privileged -z default

验证备份

• 查看Cronjob状态

# oc get cronjobs.batch 
NAME                    SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
openshift-etcd-backup   */30 * * * *   False     0        64m             81m
# oc get jobs.batch 
NAME                               COMPLETIONS   DURATION   AGE
openshift-etcd-backup-1575957420   1/1           12s        74m
openshift-etcd-backup-1575957600   1/1           12s        71m
openshift-etcd-backup-1575957900   1/1           11s        66m
# oc get pod
NAME                                     READY   STATUS      RESTARTS   AGE
openshift-etcd-backup-1575957420-wbspn   0/1     Completed   0          74m
openshift-etcd-backup-1575957600-rzvpx   0/1     Completed   0          71m
openshift-etcd-backup-1575957900-btf8z   0/1     Completed   0          66m

• 验证备份快照

# ETCDCTL_API=3 ../bin/etcdctl --write-out=table snapshot status 2019-12-10_05\:54\:02_UTC.db 
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| d4533406 |  7093192 |       5969 |     152 MB |
+----------+----------+------------+------------+

---

参考

• Backing up etcd