longhorn 部署及使用

1. 概述

该教程，是基于 k8s 集群的部署，事先准备好 k8s 集群，此处k8s集群部署略过

2. 部署

官网教程

2.1 安装基础组件

yum install iscsi-initiator-utils

2.2 下载 部署文件 yml

wget https://raw.githubusercontent.com/longhorn/longhorn/v1.3.2/deploy/longhorn.yaml

2.3 修改 yml 文件

2.3.1 修改pv为保留策略，否则删除pvc的时候pv就会被删除

data:
  storageclass.yaml: |
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: longhorn
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
    provisioner: driver.longhorn.io
    allowVolumeExpansion: true
    # 找到文件中的StorageClass.data.reclaimPolicy字段,修改为Retain
    reclaimPolicy: "Retain"
    volumeBindingMode: Immediate

2.3.2 修改longhorn-frontend 服务的svc资源类型和端口，便于web界面查看

kind: Service
apiVersion: v1
metadata:
  labels:
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/version: v1.3.2
    app: longhorn-ui
  name: longhorn-frontend
  namespace: longhorn-system
spec:
  # 修改为 NodePort，默认为ClusterIP
  #type: ClusterIP
  type: NodePort
  selector:
    app: longhorn-ui
  ports:
  - name: http
    port: 80
    targetPort: http
    # 修改端口号，默认为 null
    #nodePort: null
    nodePort: 30080

2.3.3 修改存储路径

默认存储路径为：/var/lib/longhorn，可修改为自己想要路径

找到DaemonSet资源下的spec.template.containers.volumeMounts字段

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/version: v1.3.2
    app: longhorn-manager
  name: longhorn-manager
  namespace: longhorn-system
spec:
  selector:
    matchLabels:
      app: longhorn-manager
  template:
    metadata:
      labels:
        app.kubernetes.io/name: longhorn
        app.kubernetes.io/instance: longhorn
        app.kubernetes.io/version: v1.3.2
        app: longhorn-manager
    spec:
      initContainers:
      - name: wait-longhorn-admission-webhook
        image: longhornio/longhorn-manager:v1.3.2
        command: ['sh', '-c', 'while [ $(curl -m 1 -s -o /dev/null -w "%{http_code}" -k https://longhorn-admission-webhook:9443/v1/healthz) != "200" ]; do echo waiting; sleep 2; done']
      containers:
      - name: longhorn-manager
        image: longhornio/longhorn-manager:v1.3.2
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        command:
        - longhorn-manager
        - -d
        - daemon
        - --engine-image
        - "longhornio/longhorn-engine:v1.3.2"
        - --instance-manager-image
        - "longhornio/longhorn-instance-manager:v1_20221003"
        - --share-manager-image
        - "longhornio/longhorn-share-manager:v1_20221003"
        - --backing-image-manager-image
        - "longhornio/backing-image-manager:v3_20221003"
        - --manager-image
        - "longhornio/longhorn-manager:v1.3.2"
        - --service-account
        - longhorn-service-account
        ports:
        - containerPort: 9500
          name: manager
        readinessProbe:
          tcpSocket:
            port: 9500
        volumeMounts:
        - name: dev
          mountPath: /host/dev/
        - name: proc
          mountPath: /host/proc/
        - name: longhorn
          # 修改为自己的路径
          #mountPath: /var/lib/longhorn/
          mountPath: /data/longhorn/
          mountPropagation: Bidirectional
        - name: longhorn-grpc-tls
          mountPath: /tls-files/
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
      volumes:
      - name: dev
        hostPath:
          path: /dev/
      - name: proc
        hostPath:
          path: /proc/
      - name: longhorn
        hostPath:
          # 修改为自己的路径
          #path: /var/lib/longhorn/
          path: /data/longhorn/
      - name: longhorn-grpc-tls
        secret:
          secretName: longhorn-grpc-tls
          optional: true
      serviceAccountName: longhorn-service-account
  updateStrategy:
    rollingUpdate:
      maxUnavailable: "100%"

2.4 部署

kubectl apply -f longhorn.yml

2.5 检查

web界面管理

image-20221122144530823.png

查看所有 pod 运行状态

image-20221122144603579.png

查看 sc 资源

image-20221122144720913.png

3. 使用 longhorn 存储

3.1 创建一个pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc
  namespace: longhorn-system
  annotations:
    volume.beta.kubernetes.io/storage-class: "longhorn"
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

3.2 创建一个 pod，验证 pvc 使用情况

kind: Pod
apiVersion: v1
metadata:
  name: mysql-pod
  namespace: longhorn-system
  labels:
    app: mysql-server
spec:
  containers:
  - name: mysql-server
    imagePullPolicy: Never
    image: mysql:5.7
    volumeMounts:
    - name: data
      mountPath: "/var/lib/mysql"
    ports:
    - containerPort: 3306
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: '123456'
    args: 
    - "--ignore-db-dir=lost+found"
    
  restartPolicy: "Always"
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: test-pvc
  
---
kind: Service
apiVersion: v1
metadata:
  name: mysql-server
  namespace: longhorn-system
spec:
  selector:
    app: mysql-server
  type: NodePort
  ports:
  - port: 3306
    targetPort: 3306
    nodePort: 30060

image-20221122144910946.png image-20221122144927101.png image-20221122144940066.png image-20221122145043339.png