十三云原生存储及存储进阶

作者: 負笈在线 | 来源:发表于2022-05-18 15:05 被阅读0次

十三云原生存储及存储进阶
云存储原理及系统构架原理是什么？
小程序·云开发数据库
五个Taurus垃圾回收compactor优化方案，减少系统资源
为数据计算提供强力引擎，阿里云文件存储HDFS v1.0公测发布
MySQL进阶——存储引擎
【Filecoin周报-25】：go-filecoin将实现与l
Filecoin 项目简介
Cloud Team：上能修 DB，下能改容器的云原生信仰者｜
云原生存储和云存储有什么区别？

StorageClass：存储类，由K8s管理员创建，用于动态PV的管理，可以链接至不同的后端存储，比如Ceph、GlusterFS等。之后对存储的请求可以指向StorageClass，然后StorageClass会自动的创建、删除PV。实现方式：
--in-tree: 内置于K8s核心代码，对于存储的管理，都需要编写相应的代码。
--out-of-tree：由存储厂商提供一个驱动（CSI或Flex Volume），安装到K8s集群，然后StorageClass只需要配置该驱动即可，驱动器会代替StorageClass管理存储。
StorageClass地址：https://kubernetes.io/docs/concepts/storage/storage-classes/

(一)Rook

1.rook概念

Rook是一个自我管理的分布式存储编排系统，它本身并不是存储系统，在存储和k8s之前搭建了一个桥梁，使存储系统的搭建或者维护变得特别简单，Rook将分布式存储系统转变为自我管理、自我扩展、自我修复的存储服务。它让一些存储的操作，比如部署、配置、扩容、升级、迁移、灾难恢复、监视和资源管理变得自动化，无需人工处理。并且Rook支持CSI，可以利用CSI做一些PVC的快照、扩容、克隆等操作。
官网：https://rook.io/

2.rook安装

下载rook，目前1.6.3

# git clone --single-branch --branch v1.6.3 https://github.com/rook/rook.git

配置更改
修改Rook CSI镜像地址，原本的地址可能是gcr的镜像，但是gcr的镜像无法被国内访问，所以需要同步gcr的镜像到阿里云镜像仓库 75行左右添加，注意对其

# vim operator.yaml
ROOK_CSI_REGISTRAR_IMAGE: "registry.cn-beijing.aliyuncs.com/dotbalo/csi-node-driver-registrar:v2.0.1"
ROOK_CSI_RESIZER_IMAGE: "registry.cn-beijing.aliyuncs.com/dotbalo/csi-resizer:v1.0.1"
ROOK_CSI_PROVISIONER_IMAGE: "registry.cn-beijing.aliyuncs.com/dotbalo/csi-provisioner:v2.0.4"
ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.cn-beijing.aliyuncs.com/dotbalo/csi-snapshotter:v4.0.0"
ROOK_CSI_ATTACHER_IMAGE: "registry.cn-beijing.aliyuncs.com/dotbalo/csi-attacher:v3.0.2"

如果是其他版本，需要自行同步，同步方法可以在网上找到相关文章。可以参考https://blog.csdn.net/sinat_35543900/article/details/103290782 还是operator文件，新版本rook默认关闭了自动发现容器的部署，可以找到ROOK_ENABLE_DISCOVERY_DAEMON改成true即可：
部署

# cd cluster/examples/kubernetes/ceph
# kubectl create -f crds.yaml -f common.yaml -f operator.yaml

等待operator容器和discover容器启动

[root@k8s-master01 ceph]# kubectl -n rook-ceph get pod
NAME                                                     READY   STATUS      RESTARTS   AGE
rook-ceph-operator-7c7d8846f4-fsv9f                      1/1     Running     0          25h
rook-discover-qw2ln                                      1/1     Running     0          28h
rook-discover-wf8t7                                      1/1     Running     0          28h
rook-discover-z6dhq                                      1/1     Running     0          28h

3.用ROOK搭建ceph集群

3.1.修改配置文件

修改cluster文件，配置image（同步到私有仓库），文件路径（本地），mon（大于3），mgr(生成环境大于2)等配置，把useAllNodes: false和useAllDevices: false都写flase，

[root@k8s-master01 ceph]# vim cluster.yaml
...
  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
....
# nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
    - name: "k8s-master03"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
    - name: "k8s-node01"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
    - name: "k8s-node02"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
...

3.2.安装创建

[root@k8s-master01 ceph]# kubectl create -f cluster.yaml

[root@k8s-master01 ceph]# kubectl get pod -n rook-ceph    #查看新创建的pod,大概装了一个多小时才好（尽量把镜像先下载好放在私有仓库，下载特别慢）
NAME                                                     READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-c2g9n                                   3/3     Running     0          89m
csi-cephfsplugin-c6kt2                                   3/3     Running     0          89m
csi-cephfsplugin-dgc2k                                   3/3     Running     0          89m
csi-cephfsplugin-provisioner-846ffc6cb4-2949b            6/6     Running     2          89m
csi-cephfsplugin-provisioner-846ffc6cb4-zzwvn            6/6     Running     5          89m
csi-cephfsplugin-rxvlh                                   3/3     Running     0          89m
csi-cephfsplugin-zst4j                                   3/3     Running     0          89m
csi-rbdplugin-5f8df                                      3/3     Running     0          89m
csi-rbdplugin-5wgw4                                      3/3     Running     0          89m
csi-rbdplugin-7c4lc                                      3/3     Running     0          89m
csi-rbdplugin-c7g9w                                      3/3     Running     0          89m
csi-rbdplugin-provisioner-75fd5c779f-2264d               6/6     Running     0          89m
csi-rbdplugin-provisioner-75fd5c779f-qlhtz               6/6     Running     0          89m
csi-rbdplugin-xn4cd                                      3/3     Running     0          89m
rook-ceph-crashcollector-k8s-master03-6f8447cdff-rj7pc   1/1     Running     0          32m
rook-ceph-crashcollector-k8s-node01-747795874c-d44c8     1/1     Running     0          32m
rook-ceph-crashcollector-k8s-node02-5d4867cfb8-qrpb8     1/1     Running     0          32m
rook-ceph-mgr-a-645f487fbb-27g5v                         1/1     Running     0          32m
rook-ceph-mon-a-5f6488c96c-bgj98                         1/1     Running     0          89m
rook-ceph-mon-b-cb5c9c669-kjdwg                          1/1     Running     0          69m
rook-ceph-mon-c-59bf47bb86-vx8hj                         1/1     Running     0          57m
rook-ceph-operator-65965c66b5-2z6nf                      1/1     Running     0          120m
rook-ceph-osd-0-5fdd49695b-5srzl                         1/1     Running     0          32m
rook-ceph-osd-1-5db9d74fb5-pkthf                         1/1     Running     0          32m
rook-ceph-osd-2-756ccbbcbd-mgmrj                         1/1     Running     0          32m
rook-ceph-osd-prepare-k8s-master03-k5g89                 0/1     Completed   0          32m
rook-ceph-osd-prepare-k8s-node01-fkks9                   0/1     Completed   0          32m
rook-ceph-osd-prepare-k8s-node02-jf985                   0/1     Completed   0          32m
rook-discover-5bttf                                      1/1     Running     0          117m
rook-discover-ghxxf                                      1/1     Running     1          117m
rook-discover-kf74h                                      1/1     Running     0          117m
rook-discover-q6ss9                                      1/1     Running     0          117m
rook-discover-znjml                                      1/1     Running     1          117m

3.3.安装ceph客户端工具

[root@k8s-master01 ceph]# kubectl  create -f toolbox.yaml -n rook-ceph
deployment.apps/rook-ceph-tools created

[root@k8s-master01 ceph]# kubectl get pod -n rook-ceph | grep tool
rook-ceph-tools-fc5f9586c-g5q5d                          1/1     Running     0          3m31s

进入客户端工具pod，使用ceph命令查看ceph状态

[root@k8s-master01 ceph]# kubectl exec -it rook-ceph-tools-fc5f9586c-g5q5d -n rook-ceph -- bash 
[root@rook-ceph-tools-fc5f9586c-g5q5d /]# ceph status
  cluster:
    id:     37055851-a453-4a79-a032-4f8a94099896
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
 
  services:
    mon: 3 daemons, quorum a,c,b (age 61m)
    mgr: a(active, since 61m)
    osd: 3 osds: 3 up (since 61m), 3 in (since 61m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 57 GiB / 60 GiB avail
    pgs:     1 active+clean
 
[root@rook-ceph-tools-fc5f9586c-g5q5d /]# ceph osd status
ID  HOST           USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE      
 0  k8s-node01    1026M  18.9G      0        0       0        0   exists,up  
 1  k8s-node02    1026M  18.9G      0        0       0        0   exists,up  
 2  k8s-master03  1026M  18.9G      0        0       0        0   exists,up  

[root@rook-ceph-tools-fc5f9586c-g5q5d /]# ceph df
--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED     RAW USED  %RAW USED
hdd    60 GiB  57 GiB  6.2 MiB   3.0 GiB       5.01
TOTAL  60 GiB  57 GiB  6.2 MiB   3.0 GiB       5.01
--- POOLS ---
POOL                   ID  PGS  STORED  OBJECTS  USED  %USED  MAX AVAIL
device_health_metrics   1    1     0 B        0   0 B      0     18 GiB

3.4.Ceph dashboard

默认情况下，ceph dashboard是打开的，由于测试没有安装ingress，就创建一个nodePort类型的Service暴露服务：

[root@k8s-master01 ceph]# kubectl get svc -n rook-ceph  | grep dashboard
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mgr-dashboard    ClusterIP   10.107.189.100           7000/TCP            74m    #默认创建，可以用ingress代理访问

创建一个Service

[root@k8s-master01 ceph]# vim dashboard-np.yaml
vim dashboard-np.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rook-ceph-mgr
    ceph_daemon_id: a
    rook_cluster: rook-ceph
  name: rook-ceph-mgr-dashboard-np
  namespace: rook-ceph
spec:
  ports:
  - name: http-dashboard
    port: 7000
    protocol: TCP
    targetPort: 7000
  selector:
    app: rook-ceph-mgr
    ceph_daemon_id: a
    rook_cluster: rook-ceph
  sessionAffinity: None
  type: NodePort

[root@k8s-master01 ceph]# kubectl create -f dashboard-np.yaml 
service/rook-ceph-mgr-dashboard-np created

[root@k8s-master01 ceph]# kubectl get svc -n rook-ceph  | grep dashboard  #获得端口32377
rook-ceph-mgr-dashboard      ClusterIP   10.107.189.100           7000/TCP            79m
rook-ceph-mgr-dashboard-np   NodePort    10.109.245.0             7000:32377/TCP      71s

其中一个IP+端口即可访问 http://192.168.0.102:32377

[root@k8s-master01 ceph]# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
oa....a60

警告解决：https://docs.ceph.com/en/octopus/rados/operations/health-checks/

[root@k8s-master01 ceph]# kubectl exec -it rook-ceph-tools-fc5f9586c-g5q5d -n rook-ceph -- bash
[root@rook-ceph-tools-fc5f9586c-g5q5d /]# ceph config set mon auth_allow_insecure_global_id_reclaim false</pre>

(三) StorageClass动态存储

1.Storageclass动态储存-块存储

块存储一般用于一个Pod挂载一块存储使用，相当于一个服务器新挂了一个盘，只给一个应用使用。参考文档：https://rook.io/docs/rook/v1.6/ceph-block.html

1.1.搭建

官网：https://rook.github.io/docs/rook/v1.6/ceph-pool-crd.html

[root@k8s-master01 ceph]# pwd
/root/rook/cluster/examples/kubernetes/ceph

[root@k8s-master01 ceph]# vim csi/rbd/storageclass.yaml   #找到自带的yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool     #创建的pool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 2   #测试环境用2个，不能超过OSD的数量
    requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass   #创建的StorageClass连接到pool
metadata:
  name: rook-ceph-block   #定义块储存的StorageClass 名称
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph  #创建rook时的名称，一般和命名空间名称一致
  pool: replicapool    #连接pool池子的名称（上述）
  imageFormat: "2"    
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner    #创建的账号密码secret
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster
  csi.storage.k8s.io/fstype: ext4  #格式类型
allowVolumeExpansion: true   #是否允许扩容
reclaimPolicy: Delete    #回收策略

创建StorageClass和存储池：

[root@k8s-master01 ceph]# kubectl  create -f csi/rbd/storageclass.yaml -n rook-ceph
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block create

查看创建的cephblockpool和storageClass（StorageClass没有namespace隔离性）：

[root@k8s-master01 ceph]# kubectl get cephblockpools.ceph.rook.io -n rook-ceph  #查看块存储的pool 
NAME          AGE
replicapool   29s

[root@k8s-master01 ceph]# kubectl get sc   #查看storageclass
NAME              PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   38s

1.2.测试挂载mysql-deployment

创建一个MySQL服务，该文件有一段pvc的配置，pvc会连接刚才创建的storageClass，然后动态创建pv，然后连接到ceph创建对应的存储之后创建pvc只需要指定storageClassName为刚才创建的StorageClass名称即可连接到rook的ceph。如果是statefulset，只需要将volumeTemplateClaim里面的Claim名称改为StorageClass名称即可动态创建Pod。其中MySQL deployment的volumes配置挂载了该pvc：

[root@k8s-master01 kubernetes]# pwd
/root/rook/cluster/examples/kubernetes
apiVersion: v1
kind: Service    #创建mysql-svc
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim   #创建mysqlpvc
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  storageClassName: rook-ceph-block     #指定storageClassName为刚才创建的StorageClass名称即可连接到rook的ceph
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
    tier: mysql
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
        - image: mysql:5.6
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: changeme
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql   #挂载路径
      volumes:
        - name: mysql-persistent-storage  
          persistentVolumeClaim:
            claimName: mysql-pv-claim   #指定上述创建的PVC名称

[root@k8s-master01 kubernetes]# kubectl create -f mysql.yaml 
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created

因为MySQL的数据不能多个MySQL实例连接同一个存储，所以一般只能用块存储。相当于新加了一块盘给MySQL使用。
创建完成后可以查看创建的pvc和pv

[root@k8s-master01 kubernetes]# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-65583bf8-33c1-45f5-a99c-351c88014576   2Gi        RWO            rook-ceph-block   9m31s
[root@k8s-master01 kubernetes]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS      REASON   AGE
pvc-65583bf8-33c1-45f5-a99c-351c88014576   2Gi        RWO            Delete           Bound    default/mysql-pv-claim   rook-ceph-block            9m33s

此时在ceph dashboard上面也可以查看到对应的image

此时发现mysql的pod已经起来了，测试之后删除，然后回到ceph的web端，images已被删除

[root@k8s-master01 kubernetes]# kubectl get po 
NAME                               READY   STATUS    RESTARTS   AGE
wordpress-mysql-6965fc8cc8-m5vdk   1/1     Running   0          6m52s

[root@k8s-master01 kubernetes]# kubectl delete -f mysql.yaml   
service "wordpress-mysql" deleted
persistentvolumeclaim "mysql-pv-claim" deleted
deployment.apps "wordpress-mysql" deleted

1.3.测试挂载nginx-StatefulSet

创建StatefulSet会创建几个副本，每个pod都需要一个储存，如果按照块储存的方法，先创建PVC的方法就行不通，一个块存储PVC只能挂载一个POD,这时要用到volumeClaimTemplates独有标识，这是StatefulSet专门的标识，使用比较简单，直接连接storageClassName自动创建PVC。

[root@k8s-master01 kubernetes]# vim sts-sc.yaml
apiVersion: v1
kind: Service   #首先创建一个SVC
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: nginx 
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "rook-ceph-block"
      resources:
        requests:
          storage: 1Gi

[root@k8s-master01 kubernetes]# kubectl create -f sts-sc.yaml   #创建STS
service/nginx created
statefulset.apps/web created

[root@k8s-master01 kubernetes]# kubectl get pvc   #查看PVC，storageClass已经自动创建了带序号排序的PVC
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
www-web-0   Bound    pvc-60238e27-31b1-48b6-9e6f-5309b11a0b56   1Gi        RWO            rook-ceph-block   118s
www-web-1   Bound    pvc-cbf20374-5e9d-4bb3-8361-59002878ecdb   1Gi        RWO            rook-ceph-block   84s
www-web-2   Bound    pvc-1390c984-003d-4d38-a0fd-00ceefda01ed   1Gi        RWO            rook-ceph-block   7s

2.Storageclass动态储存-文件存储

共享文件系统一般用于多个Pod共享一个存储
官方文档：https://rook.io/docs/rook/v1.6/ceph-filesystem.html

2.1.搭建

[root@k8s-master01 ceph]# vim filesystem.yaml
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph # namespace:cluster
spec:
  metadataPool:
    replicated:
      size: 3    
      requireSafeReplicaSize: true
    parameters:
      compression_mode:
        none
  dataPools:
    - failureDomain: host
      replicated:
        size: 3
        requireSafeReplicaSize: true
      parameters:
        compression_mode:
          none
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true
    placement:
      podAntiAffinity:  #设置反亲和力，提高高可用性
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values:
                    - rook-ceph-mds
            topologyKey: kubernetes.io/hostname
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - rook-ceph-mds
              topologyKey: topology.kubernetes.io/zone
    annotations:
    labels:
    #  key: value
    resources:
    # The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory
    #  limits:
    #    cpu: "500m"
    #    memory: "1024Mi"
    #  requests:
    #    cpu: "500m"
    #    memory: "1024Mi"
    # priorityClassName: my-priority-class
  mirroring:
    enabled: false

[root@k8s-master01 ceph]# kubectl  create -f filesystem.yaml  #创建
cephfilesystem.ceph.rook.io/myfs created


[root@k8s-master01 ceph]#  kubectl -n rook-ceph get pod -l app=rook-ceph-mds   #查看
NAME                                    READY   STATUS    RESTARTS   AGE
rook-ceph-mds-myfs-a-5c8749c777-bhrdh   1/1     Running   0          51s
rook-ceph-mds-myfs-b-8547d89787-jznhf   1/1     Running   0          50s

然后创建storageclass

[root@k8s-master01 ceph]# vim csi/cephfs/storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com #driver:namespace:operator，下示例查看
parameters:
  clusterID: rook-ceph # namespace:cluster
  fsName: myfs
  pool: myfs-data0

  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner   #控制集群的密钥
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:  

[root@k8s-master01 ceph]# kubectl get csidrivers  #与上面provisioner:匹配，要存在
NAME                            ATTACHREQUIRED   PODINFOONMOUNT   MODES        AGE
rook-ceph.cephfs.csi.ceph.com   true             false            Persistent   3d23h
rook-ceph.rbd.csi.ceph.com      true             false            Persistent   3d23h

[root@k8s-master01 ceph]# kubectl create -f csi/cephfs/storageclass.yaml   #创建storageclass
storageclass.storage.k8s.io/rook-cephfs created

[root@k8s-master01 ceph]# kubectl get sc  
NAME          PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-cephfs   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   27s

2.2.挂载测试nginx

先创建一个nginx的svc，再创建PVC，创建一个nginx的yaml

[root@k8s-master01 cephfs]# vim nginx-file.yaml
apiVersion: v1
kind: Service   #创建svc
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  selector:
    app: nginx
  type: ClusterIP
---
kind: PersistentVolumeClaim  #创建pvc
apiVersion: v1
metadata:
  name: nginx-share-pvc  #自定义pvc的名称
spec:
  storageClassName: rook-cephfs  #之前创建的storageClass的名称
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment   #创建nginx-deployment挂载测试
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      containers:
      - name: nginx
        image: nginx 
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html    #设置的共享文件
      volumes:
        - name: www
          persistentVolumeClaim:
            claimName: nginx-share-pvc   #上面创建PVC的名称

创建并查看资源

persistentvolumeclaim/nginx-share-pvc created
deployment.apps/web created

[root@k8s-master01 cephfs]# kubectl get pvc 
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-share-pvc   Bound    pvc-06c21214-79ff-4bde-8b18-bd12414963b2   1Gi        RWX            rook-cephfs    6s

[root@k8s-master01 cephfs]# kubectl get pv   #自动创建的pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-06c21214-79ff-4bde-8b18-bd12414963b2   1Gi        RWX            Delete           Bound    default/nginx-share-pvc   rook-cephfs             33s

[root@k8s-master01 cephfs]# kubectl get svc 
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1                443/TCP   20d
nginx        ClusterIP   10.102.186.144           80/TCP    12s

[root@k8s-master01 cephfs]# kubectl get pod   #pod已启动完成
NAME                   READY   STATUS    RESTARTS   AGE
web-7bf54cbc8d-cr767   1/1     Running   0          112s
web-7bf54cbc8d-hjx6n   1/1     Running   0          112s
web-7bf54cbc8d-xnb2h   1/1     Running   0          112s

进入其中一个pod修改index，查看是否共享储存

[root@k8s-master01 cephfs]# kubectl exec -it web-7bf54cbc8d-cr767 -- bash
root@web-7bf54cbc8d-cr767:/usr/share/nginx/html# echo "123 mutouren" > index.html   #进入容器添加数据

[root@k8s-master01 cephfs]# curl 10.102.186.144  #多次curl pvc访问一致
123 mutouren
[root@k8s-master01 cephfs]# curl 10.102.186.144
123 mutouren
[root@k8s-master01 cephfs]# curl 10.102.186.144
123 mutouren
[root@k8s-master01 cephfs]# curl 10.102.186.144
123 mutouren
[root@k8s-master01 cephfs]# kubectl exec web-7bf54cbc8d-xnb2h -- cat /usr/share/nginx/html/index.html    #查看另一个容器的文件也存在
123 mutouren

(四) PVC扩容、快照、回滚、克隆

1.PVC扩容

文件共享类型的PVC扩容需要k8s 1.15+ 块存储类型的PVC扩容需要k8s 1.16+
PVC扩容需要开启ExpandCSIVolumes，新版本的k8s已经默认打开了这个功能，可以查看自己的k8s版本是否已经默认打开了该功能：

[root@k8s-master01 ~]# kubectl exec kube-apiserver-k8s-master01 -n kube-system -- kube-apiserver -h |grep ExpandCSIVolumes
 ExpandCSIVolumes=true|false (BETA - default=true)

如果default为true就不需要打开此功能，如果default为false，需要开启该功能。
块存储和文件存储类似，已块存储为例。扩容前容量：

[root@k8s-master01 kubernetes]# kubectl exec wordpress-mysql-6965fc8cc8-ph6nn  -- df -h | grep mysql
/dev/rbd0                 20G  160M   20G   1% /var/lib/mysql

扩容直接edit pvc修改其大小为25G,可以看到此时pvc并没有扩容，但是pv已经扩容,也可以看到ceph dashboard的image也完成了扩容，但是pvc和pod里面的状态会有延迟，大概等待5-10分钟后，即可完成扩容：

[root@k8s-master01 kubernetes]# kubectl edit pvc mysql-pv-claim 
persistentvolumeclaim/mysql-pv-claim edited
...
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi  #20改为25G

[root@k8s-master01 kubernetes]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS      REASON   AGE
pvc-ef45d834-eaad-40dc-a6fa-83b567182924   25Gi       RWO            Delete           Bound    default/mysql-pv-claim   rook-ceph-block            4m26s

几分钟后查看，已经扩容

[root@k8s-master01 kubernetes]# kubectl get pvc  
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-ef45d834-eaad-40dc-a6fa-83b567182924   25Gi       RWO            rook-ceph-block   

[root@k8s-master01 kubernetes]# kubectl exec wordpress-mysql-6965fc8cc8-ph6nn  -- df -h | grep mysql 
/dev/rbd0                 25G  160M   25G   1% /var/lib/mysql

2.PVC快照

PVC快照功能需要k8s 1.17+
以块存储为例：
先创建snapshotclass

[root@k8s-master01 rbd]# pwd
/root/rook/cluster/examples/kubernetes/ceph/csi/rbd
[root@k8s-master01 rbd]# kubectl create -f snapshotclass.yaml 
volumesnapshotclass.snapshot.storage.k8s.io/csi-rbdplugin-snapclass created

创建之前先在容器里面写入数据

[root@k8s-master01 rbd]# kubectl exec -it wordpress-mysql-6965fc8cc8-ph6nn -- bash
root@wordpress-mysql-6965fc8cc8-ph6nn:/# df -h
/dev/rbd0                 25G  160M   25G   1% /var/lib/mysql

root@wordpress-mysql-6965fc8cc8-ph6nn:/var/lib/mysql# touch  123.log
root@wordpress-mysql-6965fc8cc8-ph6nn:/var/lib/mysql# echo 123 > 123.log

然后创建，查看

[root@k8s-master01 rbd]# vim snapshot.yaml
---
# 1.17 <= K8s <= v1.19
# apiVersion: snapshot.storage.k8s.io/v1beta1
# K8s >= v1.20
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: rbd-pvc-snapshot  #需要打快照的PVC
spec:
  volumeSnapshotClassName: csi-rbdplugin-snapclass
  source:
    persistentVolumeClaimName: mysql-pv-claim

[root@k8s-master01 rbd]# kubectl create -f snapshot.yaml 
volumesnapshot.snapshot.storage.k8s.io/rbd-pvc-snapshot created

[root@k8s-master01 rbd]# kubectl get volumesnapshot  #创建成功
NAME               READYTOUSE   SOURCEPVC        SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS             SNAPSHOTCONTENT                                    CREATIONTIME   AGE
rbd-pvc-snapshot   true         mysql-pv-claim                           25Gi          csi-rbdplugin-snapclass   snapcontent-e4412f69-eba4-48db-8a55-4a9ad65c1d62   16s            17s

可以到web端查看快照

3.PVC快照回滚

如果想要创建一个具有某个数据的PVC，可以从某个快照恢复：

[root@k8s-master01 rbd]# cat pvc-restore.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc-restore  #新建pvc快照的名字
spec:
  storageClassName: rook-ceph-block    #新建pvc的storageClass,与来源的sc一致
  dataSource:
    name: rbd-pvc-snapshot  #来源pvc快照的名字
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi #不能低于原pvc的大小

[root@k8s-master01 rbd]# kubectl create -f pvc-restore.yaml 
persistentvolumeclaim/rbd-pvc-restore created

[root@k8s-master01 rbd]# kubectl get pvc  #查看PVC
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim    Bound    pvc-ef45d834-eaad-40dc-a6fa-83b567182924   25Gi       RWO            rook-ceph-block   
rbd-pvc-restore   Bound    pvc-b7be06bd-fa78-415e-a153-8167ecd1ecf0   25Gi       RWO            rook-ceph-block   5s

第二步，从快照创建的pvc创建一个pod,然后进行快照恢复。

[root@k8s-master01 kubernetes]# vim mysql-restore.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: check-mysql
spec:
  selector:
    matchLabels:
      app: check
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: check
    spec:
      containers:
        - image: mysql:5.6
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: changeme
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: check-mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: check-mysql-persistent-storage
          persistentVolumeClaim:
            claimName: rbd-pvc-restore   #换成新的PVC

[root@k8s-master01 kubernetes]# kubectl create -f mysql-restore.yaml   #然后创建
deployment.apps/check-mysql created
[root@k8s-master01 kubernetes]# kubectl exec -it check-mysql-656b9f555b-dfzsb -- bash
root@check-mysql-656b9f555b-dfzsb:/# cd /var/lib/mysql
root@check-mysql-656b9f555b-dfzsb:/var/lib/mysql# ls  #文件和拍快照时一致，可以进行文件还原
123.log  auto.cnf  ib_logfile0  ib_logfile1  ibdata1  lost+found  mysql  performance_schema

4.PVC克隆

pvc克隆和pvc回滚操作类似，都是根据快照先创建一个新的PVC，然后通过PVC创建pod。
区别是克隆是用现在用着的PVC作来源dataSource，回滚是用备份快照的数据。kind一个是VolumeSnapshot，一个是PersistentVolumeClaim 。

[root@k8s-master01 rbd]# cat pvc-clone.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc-clone  #新建pvc快照的名字
spec:
  storageClassName: rook-ceph-block  #新建pvc的storageClass,与来源的sc一致
  dataSource:
    name: mysql-pv-claim     #来源pvc快照的名字，你需要克隆的PVC
    kind: PersistentVolumeClaim    
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi

[root@k8s-master01 rbd]# kubectl create -f pvc-clone.yaml 
persistentvolumeclaim/rbd-pvc-clone created

[root@k8s-master01 rbd]# kubectl get pvc 
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim    Bound    pvc-ef45d834-eaad-40dc-a6fa-83b567182924   25Gi       RWO            rook-ceph-block   
rbd-pvc-clone     Bound    pvc-74fdc858-8bc3-4235-9255-c7c79a940db0   25Gi       RWO            rook-ceph-block   4s
rbd-pvc-restore   Bound    pvc-b7be06bd-fa78-415e-a153-8167ecd1ecf0   25Gi       RWO            rook-ceph-block   28m

[root@k8s-master01 rbd]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS      REASON   AGE
pvc-74fdc858-8bc3-4235-9255-c7c79a940db0   25Gi       RWO            Delete           Bound    default/rbd-pvc-clone     rook-ceph-block            9s
pvc-b7be06bd-fa78-415e-a153-8167ecd1ecf0   25Gi       RWO            Delete           Bound    default/rbd-pvc-restore   rook-ceph-block            28m
pvc-ef45d834-eaad-40dc-a6fa-83b567182924   25Gi       RWO            Delete           Bound    default/mysql-pv-claim    rook-ceph-block            84m

(五) 数据清理

如果Rook要继续使用，可以只清理创建的deploy、pod、pvc即可。之后可以直接投入使用
数据清理步骤：
1.首先清理挂载了PVC和Pod，可能需要清理单独创建的Pod和Deployment或者是其他的高级资源
2.之后清理PVC，清理掉所有通过ceph StorageClass创建的PVC后，最好检查下PV是否被清理
3.之后清理快照：kubectl delete volumesnapshot XXXXXXXX
4.之后清理创建的Pool，包括块存储和文件存储
a)kubectl delete -n rook-ceph cephblockpool replicapool
b)kubectl delete -n rook-ceph cephfilesystem myfs
5.清理StorageClass：kubectl delete sc rook-ceph-block rook-cephfs
6.清理Ceph集群：kubectl -n rook-ceph delete cephcluster rook-ceph
7.删除Rook资源：
a)kubectl delete -f operator.yaml
b)kubectl delete -f common.yaml
c)kubectl delete -f crds.yaml
8.如果卡住需要参考Rook的troubleshooting
a)https://rook.io/docs/rook/v1.6/ceph-teardown.html#troubleshooting
for CRD in $(kubectl get crd -n rook-ceph | awk '/ceph.rook.io/ {print $1}'); do kubectl get -n rook-ceph "$CRD" -o name | xargs -I {} kubectl patch {} --type merge -p '{"metadata":{"finalizers": [null]}}' -n rook-ceph; done
9.清理数据目录和磁盘
参考链接：https://rook.io/docs/rook/v1.6/ceph-teardown.html#delete-the-data-on-hosts
参考链接：https://rook.io/docs/rook/v1.6/ceph-teardown.html

十三云原生存储及存储进阶
存储类，由K8s管理员创建，用于动态PV的管理，可以链接至不同的后端存储，比如Ceph、GlusterFS等。之后...
云存储原理及系统构架原理是什么？
云存储原理及系统构架灵动区块链云存储[https://www.lindensys.com]提供文章摘要：云存储...
小程序·云开发数据库
一、小程序云原生的探索云开发：云函数、云存储、云数据库+链路原生、场景原生、开发原生。云数据库特点：弹性收缩...
五个Taurus垃圾回收compactor优化方案，减少系统资源
简介 TaurusDB是一种基于MySQL的计算与存储分离架构的云原生数据库，一个集群中包含多个存储几点，每个存储...
为数据计算提供强力引擎，阿里云文件存储HDFS v1.0公测发布
在2019年3月的北京云栖峰会上，阿里云正式推出全球首个云原生HDFS存储服务—文件存储HDFS，为数据分析业务在...
MySQL进阶——存储引擎
上篇文章我们学习了MySQL基础——事务，这篇文章学习MySQL进阶——存储引擎。存储引擎存储引擎就是存储数据...
【Filecoin周报-25】：go-filecoin将实现与l
Filecoin是一个将云存储转变为一个算法市场的去中心化存储网络。矿工通过提供数据存储和检索来获得原生代币（称...
Filecoin 项目简介
Filecoin是一个将云存储转变为一个算法市场的去中心化存储网络。矿工通过提供数据存储和检索来获得原生代币（称为...
Cloud Team：上能修 DB，下能改容器的云原生信仰者｜
TiDB 从诞生之时就带着云原生的标签，并且底层存储引擎 TiKV 也在 2019 年成为了 CNCF （云原生计...
云原生存储和云存储有什么区别？
作者 | 李鹏（壮怀）阿里云智能事业群高级技术专家导读：新的企业负载／智能工作负载容器化、迁云、存储方面遇到的...