本文基于k8s 1.22版本。
流程图
kubelet root dir
[root@glusterfs-2 kubelet]# tree /var/lib/kubelet/plugins*
/var/lib/kubelet/plugins
└── kubernetes.io
└── csi
└── pv
├── pvc-1dfaf0ee-d88e-4b94-8a36-4e7bde2e10ab
│ ├── globalmount
│ └── vol_data.json
├── pvc-61a403de-cb24-4d2d-ab7e-df75f643366a
│ ├── globalmount
│ └── vol_data.json
└── pvc-8b86f8b4-cc62-4c3c-ba48-1e235215dbd8
├── globalmount
└── vol_data.json
/var/lib/kubelet/plugins_registry
├── kadalu
│ └── csi.sock
└── kadalu-reg.sock
10 directories, 5 files
CSI plugin register相关代码
pkg/kubelet/kubelet.go 1411 initializeRuntimeDependentModules()
pkg/kubelet/kubelet.go 784 klet.pluginManager
pkg/kubelet/pluginmanager/reconciler/reconciler.go 110 reconcile()
pkg/kubelet/pluginmanager/operationexecutor/operation_generator.go 74 GenerateRegisterPluginFunc()
pkg/kubelet/pluginmanager/cache/types.go
pkg/volume/csi/csi_plugin.go
pkg/volume/csi/csi_client.go
pkg/kubelet/pluginmanager/plugin_manager.go 108 Run()
staging/src/k8s.io/kubelet/pkg/apis/pluginregistration/v1/api.proto
plugin的注册逻辑
遍历kubelet root dir下的plugins_registry
目录中的所有socket文件,其实只需要调用<driver>-reg.sock
文件即可,其他的socket文件在dial
之后调用GetInfo()
会报错"Method not found!"
。
-
reconciler
对象会rpc调用node-driver-registrar sidecar container中rpc server提供的的GetInfo
。 - 然后根据返回字段的type,从最开始注册的
pluginHandlers
中查找对应的handler,这里就是上文说的CSIPlugin
type的csi.RegistrationHandler{}
对象。 - 调用该对象的
ValidatePlugin
和RegisterPlugin
来注册插件,这里的注册插件其实就是设置node annotation
和label
, 以及创建/更新CSINode
对象(CSINodeDrivers
)。 - 最后rpc调用
NotifyRegistrationStatus
告知注册结果。
对于上述第三点,解释如下:
更新node annotation;创建更新CSINode对象。
- 更新node annotation主要是往当前Node中增加一个annotation
csi.volume.kubernetes.io/nodeid:{"$csiDriverName":"$driverNodeID"}
,csiDriverName是之前rpc调用node-driver-registrar sidecar container的GetInfo
获得的,
driverNodeID是直接rpc调用自定义csi-plugin的node serviceNodeGetInfo
获得的 。后面会详细介绍。
(dlv) p socketPath
"/var/lib/kubelet/plugins_registry/kadalu-reg.sock"
(dlv) p infoResp
*k8s.io/kubelet/pkg/apis/pluginregistration/v1.PluginInfo {
Type: "CSIPlugin",
Name: "kadalu",
Endpoint: "/var/lib/kubelet/plugins_registry/kadalu/csi.sock",
SupportedVersions: []string len: 1, cap: 1, ["1.0.0"],}
(dlv) p c
*k8s.io/kubernetes/pkg/volume/csi.csiDriverClient {
driverName: "kadalu",
addr: "/var/lib/kubelet/plugins_registry/kadalu/csi.sock",
metricsManager: *k8s.io/kubernetes/pkg/volume/csi.MetricsManager {driverName: "kadalu"},
nodeV1ClientCreator: k8s.io/kubernetes/pkg/volume/csi.newV1NodeClient,}
(dlv) p res
*github.com/container-storage-interface/spec/lib/go/csi.NodeGetInfoResponse {
NodeId: "glusterfs-2.deeproute.ai",
MaxVolumesPerNode: 0,
AccessibleTopology: *github.com/container-storage-interface/spec/lib/go/csi.Topology nil,
XXX_NoUnkeyedLiteral: struct {} {},
XXX_unrecognized: []uint8 len: 0, cap: 0, nil,
XXX_sizecache: 0,}
- 更新node label主要是将topology信息保存在metadata.labels中。
- apiserver中创建/更新CSINode对象,创建CSINode对象逻辑可见
CreateCSINode()
,更新CSINode对象逻辑可见installDriverToCSINode()
。-
CreateCSINode()
会创建CSINode,设置metadata中的name
和ownerReferences
,以及创建空的spec.drivers,然后设置annotation,storage.alpha.kubernetes.io/migrated-plugins:<migratedPlugin1, migratedPlugin2>
-
installDriverToCSINode()
主要是更新spec.driver
信息,其中包括name
,nodeID
,topologyKeys
,allocatable
。如果migration annotation
有变化的话也会做更新。
-
🍺 /root/go/src/k8s.io/kubernetes ☞ git:(master) ✗ kubectl get nodes glusterfs-2.deeproute.ai -o yaml | yq e '.metadata' -
annotations:
csi.volume.kubernetes.io/nodeid: '{"kadalu":"glusterfs-2.deeproute.ai"}'
flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"9e:b1:47:3f:c2:cd"}'
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: "true"
flannel.alpha.coreos.com/public-ip: 10.9.9.151
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: "0"
volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2021-08-11T06:50:02Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: glusterfs-2.deeproute.ai
kubernetes.io/os: linux
name: glusterfs-2.deeproute.ai
resourceVersion: "7888711"
uid: ab7b5284-984c-4259-978d-f954151e300a
🍺 /root/go/src/k8s.io/kubernetes ☞ git:(master) ✗ kubectl get csinodes.storage.k8s.io glusterfs-2.deeproute.ai -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/azure-disk,kubernetes.io/cinder
creationTimestamp: "2021-08-11T06:50:03Z"
name: glusterfs-2.deeproute.ai
ownerReferences:
- apiVersion: v1
kind: Node
name: glusterfs-2.deeproute.ai
uid: ab7b5284-984c-4259-978d-f954151e300a
resourceVersion: "7878094"
uid: 51efb47e-340d-4b28-98fd-485164259dc9
spec:
drivers:
- name: kadalu
nodeID: glusterfs-2.deeproute.ai
topologyKeys: null
node-driver-registrar
前面提到过GetInfo
是通过node-driver-registrar sidecar container
的GetInfo
获得的,通过与/var/lib/kubelet/plugins_registry/<csi-driver>-reg.sock
进行通信来获取的。
node-driver-registrar主要有以下两个作用:
- 连接到
/var/lib/kubelet/plugins_registry/<csi-driver>/csi.sock
Unix domain socket
用于获取csi driver name
。其会调用csi driver的GetPluginInfo
来获取。 - 创建
/var/lib/kubelet/plugins_registry/<csi-driver>-reg.sock
Unix domain socket来监听kubelet的GetInfo
和NotifyRegistrationStatus
请求,其实现了RegistrationServer
接口。
网友评论