美文网首页
kubernetes csi 开发环境搭建 (MacOS +

kubernetes csi 开发环境搭建 (MacOS +

作者: allenhaozi | 来源:发表于2021-04-25 09:54 被阅读0次

    关于CSI 知识, 可以参考 郡宝 Kubernetes 存储架构及插件使用 介绍及 CSI官方文档

    原创禁止转载

    开发其他 Operator 一直基于 kind, 简单, 方便, 开箱即用, 所以
    本文直接介绍基于kind 的环境搭建

    俊宝-Kubernetes 存储架构及插件使用-示例图

    以 fluid 为例:
    fluid 目前包含四个组件:

    1. alluxio
    2. csi-driver + node-driver-registrar
    3. dataset
    4. jindo

    如上图 因为 csi-driver 和 node-driver-registrar 需要和 kubelet 通信, 所以我们会在 kind node 的容器里面运行他们,
    即在容器里面通过 go run main.go 的方式运行 csi-driver 和 node-driver-registrar
    其余组件直接在 Mac 运行, 他们通过 ~/.kube/config 和 apiserver 通信即可

    1. 创建 kind 集群

    $ kind create cluster --config=cluster.yaml

# cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: fluid-dev
nodes:
- role: control-plane
  image: kindest/node:v1.16.15
  extraMounts:
  - hostPath: /Users/mahao/go
    containerPath: /home/work/go

$ k get nodes
NAME                      STATUS   ROLES    AGE   VERSION
fluid-dev-control-plane   Ready    master   74s   v1.16.15

    重点介绍一下 映射到 kind 节点的 go 目录
    上面提到 csi-driver 和 node-driver-registrar 需要和 kubelet 通信
    尝试了几种方式, 发现直接开两个窗口在 kind 节点容器里面, 然后直接 go run main.go 运行成本最低
    除了多了进入容器这一步, 没有额外工作

    $ docker exec -ti fluid-dev-control-plane /bin/bash 
$ go run main.go

    /User/mahao/go 是我 Mac 上 GOPATH 的目录

    /User/mahao/go
    |__bin
    |__pkg
    |__src
    |   |__ github.com
    |   |__ sigs.k8s.io
    |   |__ k8s.io
    |__local
    |  |__go #(e.g.: go1.16.3.linux-amd64.tar.gz ) 
    |__tmp

    这里面多了一个 local 因为我要在 kind 节点容器里面运行 csi-driver 和 node-driver-registrar
    需要提前下载 go 放在 local 目录下, 在 kind 节点里面运行 go 程序
    搭建过程中使用的版本是 go1.16.3.linux-amd64.tar.gz

    2. 安装 fluid 依赖的 crd 和 csi driver

    • alluxioruntimes.data.fluid.io
    • databackups.data.fluid.io
    • dataloads.data.fluid.io
    • datasets.data.fluid.io
    • jindoruntimes.data.fluid.io
    • fuse.csi.fluid.io
    $k apply -f crds.yaml
customresourcedefinition.apiextensions.k8s.io/alluxioruntimes.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/databackups.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/dataloads.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/datasets.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/jindoruntimes.data.fluid.io created

$ k apply -f csi.yaml
$ k get csidriver
NAME                CREATED AT
fuse.csi.fluid.io   2021-04-24T15:30:38Z

    3. 进到 kind 节点容器里面运行 csi-driver 和 node-driver-registrar

    为了方便 写了一个 start.sh

    3.1 启动 node-driver-registrar

    针对官方的 node-driver-registrar 做了一个很小的修改, 支持传入 reg path
    在这里下载 fluid-dev-v1.3.0 这个只针对开发, 上线继续使用官方的即可

    $ docker exec -ti fluid-dev-control-plane /bin/bash
$ cd /home/work/go/src/github.com/allenhaozi/fluid/cmd/csi && sh start.sh

### 
### start.sh
### 
#! /bin/bash
set -x

export TMPDIR=/root/go/tmp
export GO111MODULE=on
export GOMODCACHE=/root/go/pkg/mod
export GOPROXY=https://goproxy.io
export GOPATH=/home/work/go
export GOROOT=/home/work/go/local/go
export GOBIN=/home/work/go/bin
export PATH=$PATH:$GOBIN:$GOROOT/bin

if [ ! -d $TMPDIR ]; then
  mkdir -p $TMPDIR
fi

# delete reg socket if exist
rm -rf /var/lib/kubelet/plugins_registry/fuse.csi.fluid.io-reg.sock

go run main.go \
    --kubelet-registration-path="/var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock" \
    --csi-address="/var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock" \
    --reg-path="/var/lib/kubelet/plugins_registry" \
    --v=5

# stdout message
I0424 15:49:46.106452    5692 main.go:114] Version: unknown
I0424 15:49:46.106529    5692 main.go:124] Attempting to open a gRPC connection with: "/var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock"
I0424 15:49:46.106565    5692 connection.go:151] Connecting to unix:///var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock
W0424 15:49:56.107877    5692 connection.go:170] Still connecting to unix:///var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock
W0424 15:50:06.106984    5692 connection.go:170] Still connecting to unix:///var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock
I0424 15:53:57.499425    5692 main.go:131] Calling CSI driver to discover driver name
I0424 15:53:57.499555    5692 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0424 15:53:57.499594    5692 connection.go:181] GRPC request: {}
I0424 15:53:57.551231    5692 connection.go:183] GRPC response: {"name":"fuse.csi.fluid.io","vendor_version":"1.0.0"}
I0424 15:53:57.551422    5692 connection.go:184] GRPC error: <nil>
I0424 15:53:57.551472    5692 main.go:141] CSI driver name: "fuse.csi.fluid.io"
I0424 15:53:57.551526    5692 main.go:167] Starting Registration Server at: //var/lib/kubelet/plugins_registry/fuse.csi.fluid.io-reg.sock
I0424 15:53:57.556255    5692 main.go:176] Registration Server started at: //var/lib/kubelet/plugins_registry/fuse.csi.fluid.io-reg.sock
I0424 15:53:57.989196    5692 main.go:81] Received GetInfo call: &InfoRequest{}
I0424 15:53:58.059227    5692 main.go:91] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

    3.2 启动 fluid csi-driver

    $ docker exec -ti fluid-dev-control-plane /bin/bash
$ cd /home/work/go/src/github.com/allenhaozi/node-driver-registrar/cmd/csi-node-driver-registrar && sh start.sh

###
### start.sh
###
#! /bin/bash

set -x

export TMPDIR=/root/go/tmp
export GO111MODULE=on
export GOMODCACHE=/root/go/pkg/mod
export GOPROXY=https://goproxy.io
export GOPATH=/home/work/go
export GOROOT=/home/work/go/local/go
export GOBIN=/home/work/go/bin
export PATH=$PATH:$GOBIN:$GOROOT/bin

if [ ! -d $TMPDIR ]; then
  mkdir -p $TMPDIR
fi

rm -f /var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock
mkdir -p /var/lib/kubelet/csi-plugins/fuse.csi.fluid.io

cp /home/work/go/src/github.com/allenhaozi/fluid/csi/shell/check_mount.sh /usr/local/bin/check_mount.sh && chmod +x /usr/local/bin/check_mount.sh 

go run main.go start \
    --endpoint="unix://var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock" \
    --nodeid="fluid-dev-control-plane" \
    --kubeconfig=/etc/kubernetes/kubelet.conf \
    --v=5
# stdout
2021/04/24 15:53:57 BuildDate: 1970-01-01T00:00:00Z
2021/04/24 15:53:57 GitCommit: 
2021/04/24 15:53:57 GitTreeState: 
2021/04/24 15:53:57 GoVersion: go1.16.3
2021/04/24 15:53:57 Compiler: gc
2021/04/24 15:53:57 Platform: linux/amd64
I0424 15:53:57.006344    8162 driver.go:56] Driver: fuse.csi.fluid.io version: 1.0.0
I0424 15:53:57.006885    8162 driver.go:59] protocol: unix addr: var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock
I0424 15:53:57.007198    8162 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0424 15:53:57.007455    8162 driver.go:93] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0424 15:53:57.008811    8162 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock", Net:"unix"}
I0424 15:53:57.543896    8162 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0424 15:53:57.544237    8162 utils.go:98] GRPC request: {}
I0424 15:53:57.546983    8162 identityserver-default.go:32] Using default GetPluginInfo
I0424 15:53:57.547009    8162 utils.go:103] GRPC response: {"name":"fuse.csi.fluid.io","vendor_version":"1.0.0"}
I0424 15:53:57.998783    8162 utils.go:97] GRPC call: /csi.v1.Node/NodeGetInfo
I0424 15:53:57.998838    8162 utils.go:98] GRPC request: {}
I0424 15:53:58.002267    8162 nodeserver-default.go:40] Using default NodeGetInfo
I0424 15:53:58.002298    8162 utils.go:103] GRPC response: {"node_id":"fluid-dev-worker"}

    4. 直接在 Mac 上启动 alluxio 和 dataset

    4.1 启动 alluxio

    #! /bin/bash

$ cd ~/go/src/github.com/allenhaozi/fluid/cmd/alluxio && sh start.sh

# start.sh
export ALLUXIO_INIT_IMAGE_ENV=registry.cn-hangzhou.aliyuncs.com/fluid/init-users:v0.4.0-a8ba7c9
export MOUNT_ROOT=/alluxio-mnt
go run main.go start

    4.2 启动 dataset

    #! /bin/bash

$ cd ~/go/src/github.com/allenhaozi/fluid/cmd/dataset && sh start.sh

# start.sh
export ALLUXIO_INIT_IMAGE_ENV=registry.cn-hangzhou.aliyuncs.com/fluid/init-users:v0.4.0-a8ba7c9
go run main.go start \
     --metrics-addr=127.0.0.1:8082

    预加载镜像, 将镜像加入到kind集群中, 这样可以加速服务启动

    $ kind load docker-image registry.cn-hangzhou.aliyuncs.com/fluid/init-users:v0.4.0-a8ba7c9 --name=fluid-dev
$ kind load docker-image registry.cn-huhehaote.aliyuncs.com/alluxio/alluxio:2.3.0-SNAPSHOT-2c41226 --name=fluid-dev
$ kind load docker-image registry.cn-huhehaote.aliyuncs.com/alluxio/alluxio-fuse:2.3.0-SNAPSHOT-2c41226 --name=fluid-dev                                 
$ kind load docker-image nginx:latest --name=fluid-dev

    5. 运行 fluid demo 

    $ k apply -f dateset.yaml
$ k apply -f runtime.yaml
$ k apply -f pod.yaml
# check pod list
NAME                READY   STATUS    RESTARTS   AGE
demo-app            1/1     Running   0          31m
demo-fuse-ksdsr     1/1     Running   0          31m
demo-master-0       2/2     Running   0          36m
demo-worker-k2xhh   2/2     Running   0          31m
# check data
$ k exec -ti demo-app sh
$ ls /data/spark/
SparkR_3.0.2.tar.gz   spark-3.0.2-bin-hadoop2.7-hive1.2.tgz  spark-3.0.2-bin-hadoop3.2.tgz  spark-3.0.2.tgz
pyspark-3.0.2.tar.gz  spark-3.0.2-bin-hadoop2.7.tgz      spark-3.0.2-bin-without-hadoop.tgz

    搭建完成, 如果你有更简单的方法, 欢迎留言

    相关文章

      网友评论

          本文标题：kubernetes csi 开发环境搭建 (MacOS +

          本文链接：https://www.haomeiwen.com/subject/vsalrltx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|kubernetes csi 开发环境搭建 (MacOS + |投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！