美文网首页
k8s集群使用cinder作为后端存储

k8s集群使用cinder作为后端存储

作者: zhangzhifei | 来源:发表于2020-07-14 13:12 被阅读0次

    说明

    官方的部署文档中说明,使用cinder csi需要OpenStack作为Kubernetes Provider(这点还是很限制cinder在k8s中的使用)。因为对OpenStack不了解(也对“需要OpenStack作为Kubernetes Provider”不太理解),我直接在自己的k8s测试集群中部署cinder csi,csi驱动的容器都能正常启,创建pvc也可以动态绑定pv(cinder 的volume可以创建成功),但是创建pod时,csi controller attach这个volume到node上时失败了。原因是:自己本地搭建k8s的话,它申请的vloume资源 openstack没办法挂载到k8s服务器。下面记录下OpenStack作为Provider的k8s如何使用cinder。

    环境

    可以按官方文档搭建OpenStack和k8s集群。我用OpenStack是虚拟化团队创建的。k8s版本v1.18.3,CSI Cinder driver release.1.17

    部署cinder csi

    1. clonecloud-provider-openstack项目
    [root@kubernetes-master home]# git clone -b release-1.17 https://github.com/kubernetes/cloud-provider-openstack.git
    [root@kubernetes-master home]# cd cloud-provider-openstack/manifests/cinder-csi-plugin
    [root@kubernetes-master cinder-csi-plugin]# ls -lrt
    total 28
    -rw-r--r--. 1 root root  199 Jul 14 10:11 csi-cinder-driver.yaml
    -rw-r--r--. 1 root root 3467 Jul 14 10:11 cinder-csi-nodeplugin.yaml
    -rw-r--r--. 1 root root  694 Jul 14 10:11 cinder-csi-nodeplugin-rbac.yaml
    -rw-r--r--. 1 root root 3545 Jul 14 10:11 cinder-csi-controllerplugin.yaml
    -rw-r--r--. 1 root root 5555 Jul 14 10:11 cinder-csi-controllerplugin-rbac.yaml
    -rw-r--r--. 1 root root  800 Jul 14 10:20 csi-secret-cinderplugin.yaml
    
    1. 根据openstack接口信息配置cloud.conf, 放到cat /etc/kubernetes/cloud.conf
    [root@kubernetes-master cinder-csi-plugin]# cat /etc/kubernetes/cloud.conf
    [Global]
    username = admin
    password = ZE0LNthHkgfS7M13KjA6aBdZ4A1x3w3nd
    domain-name = default
    auth-url = http://10.145.69.61:35357/v3
    tenant-id = 6441367ec4ae4ce39e5f265dd
    region = bjcc_region_2
    
    1. 修改csi-secret-cinderplugin.yam
    • 在OpenStack中,compute实例使用配置驱动器或元数据服务来检索特定于实例的数据,所以按照官方文档我们将[Metadata]节加到cloud.conf(为了给csi驱动使用,所以不需要更改我们之前创建的/etc/kubernetes/cloud.conf)
    [root@kubernetes-master home]# cat cloud.conf 
    [Global]
    username = admin
    password = ZE0LNthHkgfS7M13KjA6aBdZ4A1x3w3nd
    domain-name = default
    auth-url = http://10.145.69.61:35357/v3
    tenant-id = 6441367ec4ae4ce39e5f265dd
    region = bjcc_region_2
    
    [Metadata]
    search-order = configDrive,metadataService
    
    • 对cloud.conf做base64
    [root@kubernetes-master home]# cat cloud.conf | base64 -w 0 
    W0dsb2JhbF0KdXNlcm5hbWUgPSBhZG1pbgpwYXNzd29yZCA9IFpFMExOdGhIa2dmUzdNMTNLakE2YUJkWjRBMXgzdzNuZE5VS2x1WU8KZG9tYWluLW5hbWUgPSBkZWZhdWx0CmF1dGgtdXJsID0gaHR0cDovLzEwLjE0NS42OS42OjM1MzU3L3YzCnRlbmFudC1pZCA9IDY0NDEzNjdlYzRhZTRjZTM5ZTVmMjY1ZGQwMWRiY2IxCnJlZ2lvbiA9IGJqY2NfcmVnaW9uXzIKCltNZXRhZGF0YV0Kc2VhcmNoLW9yZGVyID0gY29uZmlnRHJpdmUsbWV0YWRhdGFTZXJ2aWNlCg==
    
    • 修改csi-secret-cinderplugin.yaml
    [root@kubernetes-master cinder-csi-plugin]# cat csi-secret-cinderplugin.yaml 
    # This YAML file contains secret objects,
    # which are necessary to run csi cinder plugin.
    
    kind: Secret
    apiVersion: v1
    metadata:
      name: cloud-config
      namespace: kube-system
    data:
      #cloud.conf: W0dsb2JhbF0KdXNlcm5hbWUgPSBhZG1pbgpwYXNzd29yZCA9IG5vbW9yZXNlY3JldApkb21haW4tbmFtZSA9IGRlZmF1bHQKYXV0aC11cmwgPSBodHRwOi8vMTkyLjE2OC4yMDAuOS9pZGVudGl0eQp0ZW5hbnQtaWQgPSBjYzM0YjExZmY5NWQ0MjMwOTA4MWQwYmQ0NmMwZmY4OQpyZWdpb24gPSBSZWdpb25PbmUK
      cloud.conf: W0dsb2JhbF0KdXNlcm5hbWUgPSBhZG1pbgpwYXNzd29yZCA9IFpFMExOdGhIa2dmUzdNMTNLakE2YUJkWjRBMXgzdzNuZE5VS2x1WU8KZG9tYWluLW5hbWUgPSBkZWZhdWx0CmF1dGgtdXJsID0gaHR0cDovLzEwLjE0NS42OS42OjM1MzU3L3YzCnRlbmFudC1pZCA9IDY0NDEzNjdlYzRhZTRjZTM5ZTVmMjY1ZGQwMWRiY2IxCnJlZ2lvbiA9IGJqY2NfcmVnaW9uXzIKCltNZXRhZGF0YV0Kc2VhcmNoLW9yZGVyID0gY29uZmlnRHJpdmUsbWV0YWRhdGFTZXJ2aWNlCg==
    
    1. 修改k8s master参数
    • 修改kube-controller-manager参数,并重启
      # 启动参数添加两行
      --cloud-provider=openstack \
      --cloud-config=/etc/kubernetes/cloud.conf
    

    如果是kubeadm 以pod的形式启动的kube-controller-manager,修改/etc/kubernetes/manifests/kube-controller-manager.yaml文件就会自动重建pod
    注: 官方已经不推荐修改kube-controller-manager的cloud-provider选项了,以后会废弃。而是直接修改kubelet,由于本人没有测试过,大家可以按照官网跑一把,写的很清楚https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/

    1. 部署csi
    /home/cloud-provider-openstack/manifests/cinder-csi-plugin
    [root@kubernetes-master cinder-csi-plugin]# kubectl apply -f .
    

    查看csi pod

    [root@kubernetes-master cinder-csi-plugin]# kubectl get pod -n kube-system 
    NAME                                         READY   STATUS    RESTARTS   AGE
    coredns-65dbdb44db-56l6t                     1/1     Running   0          13h
    csi-cinder-controllerplugin-0                5/5     Running   0          152m
    csi-cinder-nodeplugin-6j2l5                  2/2     Running   0          152m
    csi-cinder-nodeplugin-fx9zq                  2/2     Running   0          152m
    csi-cinder-nodeplugin-pz2dl                  2/2     Running   0          152m
    dashboard-metrics-scraper-545bbb8767-5ch8h   1/1     Running   0          13h
    kube-flannel-ds-amd64-5qkhq                  1/1     Running   8          14h
    kube-flannel-ds-amd64-5tmqv                  1/1     Running   8          14h
    kube-flannel-ds-amd64-z9ntk                  1/1     Running   8          14h
    kubernetes-dashboard-65665f84db-dgk6g        1/1     Running   0          13h
    metrics-server-869ffc99cd-f2b8v              1/1     Running   0          13h
    

    csi driver

    [root@kubernetes-master cinder-csi-plugin]# kubectl get csidrivers.storage.k8s.io
    NAME                       ATTACHREQUIRED   PODINFOONMOUNT   MODES                  AGE
    cinder.csi.openstack.org   true             true             Persistent,Ephemeral   154m
    

    到此cinder csi已经部署好了,下面测试一下

    测试

    • 使用官方的examples
    [root@kubernetes-master cinder-csi-plugin]# kubectl -f examples/cinder-csi-plugin/nginx.yaml create
    [root@kubernetes-master cinder-csi-plugin]# kubectl get pod
    NAME     READY   STATUS    RESTARTS   AGE
    nginx   1/1     Running   0          135m
    
    • 查看pvc是否bound
    [root@kubernetes-master cinder-csi-plugin]# kubectl get pvc
    NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
    csi-pvc-cinderplugin    Bound    pvc-18b3ef69-e2ef-4be8-a579-f2b37e1e53f2   1Gi        RWO            csi-sc-cinderplugin   146m
    pvc-snapshot-demo       Bound    pvc-ed0de1a3-9a64-4984-912b-0a81d4a3c8e4   1Gi        RWO            csi-sc-cinderplugin   114m
    snapshot-demo-restore   Bound    pvc-f6e8e9b2-da5c-45a1-ba33-a230ee9cc5ba   1Gi        RWO            csi-sc-cinderplugin   110m
    
    
    • 计入容器查看挂载情况
    [root@kubernetes-master cinder-csi-plugin]# kubectl exec -it nginx2 bash
    kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
    root@nginx2:/# mount  | grep sdb
    /dev/sdb on /var/lib/www/html type ext4 (rw,relatime,seclabel,data=ordered)
    root@nginx2:/# ls /var/lib/www/html/
    index.html  lost+found  test
    root@nginx2:/# echo "test cinder" > /var/lib/www/html/newfile
    root@nginx2:/# ls /var/lib/www/html/
    index.html  lost+found  newfile  test
    root@nginx2:/# 
    

    快照等一些列测试请参考官方文档,文末附链接

    参考

    https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-cinder-csi-plugin.md
    https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-openstack-cloud-controller-manager.md#steps
    https://stackoom.com/question/3zJL9/%E5%A6%82%E4%BD%95%E5%B0%86Openstack-Cinder%E5%8D%B7%E7%94%A8%E4%BD%9CKubernetes%E6%8C%81%E4%B9%85%E5%8D%B7
    https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

    相关文章

      网友评论

          本文标题:k8s集群使用cinder作为后端存储

          本文链接:https://www.haomeiwen.com/subject/dduxhktx.html