k8s使用openebs实现动态持久化存储

简介

本文章介绍如何使用openebs为k8s提供动态申请pv的功能。iscsi提供底层存储功能，openebs管理iscsi。目前只支持pv的ReadWriteOnce访问模式

访问模式只是能力描述，并不是强制执行的，对于没有按pvc声明的方式使用pv，存储提供者应该负责访问时的运行错误。例如如果设置pvc的访问模式为ReadOnlyMany ，pod挂载后依然可写，如果需要真正的不可写，申请pvc是需要指定 readOnly: true 参数

安装

实验用的Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

ENV["LC_ALL"] = "en_US.UTF-8"

Vagrant.configure("2") do |config|
    (1..3).each do |i|
      config.vm.define "lab#{i}" do |node|
        node.vm.box = "centos-7.4-docker-17"
        node.ssh.insert_key = false
        node.vm.hostname = "lab#{i}"
        node.vm.network "private_network", ip: "11.11.11.11#{i}"
        node.vm.provision "shell",
          inline: "echo hello from node #{i}"
        node.vm.provider "virtualbox" do |v|
          v.cpus = 2
          v.customize ["modifyvm", :id, "--name", "lab#{i}", "--memory", "3096"]
          file_to_disk = "lab#{i}_vdb.vdi"
          unless File.exist?(file_to_disk)
            # 50GB
            v.customize ['createhd', '--filename', file_to_disk, '--size', 50 * 1024]
          end
          v.customize ['storageattach', :id, '--storagectl', 'IDE', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', file_to_disk]
        end
      end
    end
end

安装配置iscsi

# 安装 iscsi
yum install iscsi-initiator-utils -y

# 查看 InitiatorName 是否正常配置
cat /etc/iscsi/initiatorname.iscsi

# 启动查看状态
systemctl start iscsid.service
systemctl status iscsid.service

systemctl start iscsi.service
systemctl status iscsi.service

安装openebs

# 部署
mkdir openebs && cd openebs
wget https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-operator.yaml
wget https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-storageclasses.yaml
kubectl apply -f openebs-operator.yaml
kubectl apply -f openebs-storageclasses.yaml

# 查看 openebs 状态
kubectl get pods -n openebs -o wide
kubectl get svc -n openebs
kubectl get crd

测试

# 查看 storage class
kubectl get sc

# 创建pvc测试
cat >openebs-pvc-test.yaml<<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: openebs1
spec:
 storageClassName: openebs-standard
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 5Gi
EOF
kubectl apply -f openebs-pvc-test.yaml
 
# 查看
kubectl get pvc
kubectl get pv
 
# 创建 nginx pod 挂载测试
cat >nginx-pod.yaml<<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod1
  labels:
    name: nginx-pod1
spec:
  containers:
  - name: nginx-pod1
    image: nginx:alpine
    ports:
    - name: web
      containerPort: 80
    volumeMounts:
    - name: openebs1-vol1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: openebs1-vol1
    persistentVolumeClaim:
      claimName: openebs1
EOF
kubectl apply -f nginx-pod.yaml
 
# 查看
kubectl get pods -o wide
 
# 修改文件内容
kubectl exec -ti nginx-pod1 -- /bin/sh -c 'echo Hello World from Openebs!!! > /usr/share/nginx/html/index.html'
 
# 访问测试
POD_ID=$(kubectl get pods -o wide | grep nginx-pod1 | awk '{print $(NF-1)}')
curl http://$POD_ID

参考文档

• https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md
• https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md
• https://github.com/gluster/gluster-kubernetes/blob/master/docs/examples/hello_world/README.md
• https://jimmysong.io/kubernetes-handbook/practice/using-heketi-gluster-for-persistent-storage.html
• https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• https://docs.openshift.com/enterprise/3.1/architecture/additional_concepts/storage.html#pv-access-modes