从0到1使用kubebuilder创建crd

• 0. 下载kubebuilder
• 1. 创建目录
• 2. 初始化项目
• 3. 创建api和controller
• 4. 实现自己的crd和控制器逻辑
• 5. make manifests, 创建crd的相关yaml
• 6. 在集群中部署crd
• 7. 部署controller

简介

从0到1，手把手教会如何使用kubebuilder创建crd, 并且定制自己的控制器。

代码：https://github.com/zoux86/operator-example

0. 下载kubebuilder

# download kubebuilder and install locally.
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

1. 创建目录

~/go/src 是我的go src目录

github.com/zoux86/operator-example是想自定义的crd项目

// 可以看出来go mod init 指定的字符串就是mod文件里面的module目录
~/go/src/github.com/zoux86/operator-example#  go mod init github.com/zoux86/operator-example
go: creating new go.mod: module github.com/zoux86/operator-example

 ~/go/src/github.com/zoux86/operator-example # ls
go.mod

 ~/go/src/github.com/zoux86/operator-example # cat go.mod
module github.com/zoux86/operator-example   

go 1.18

2. 初始化项目

执行kubebuilder init这一条命令就行了

 ~/go/src/github.com/zoux86/operator-example # ~/kubebuilder init --domain github.com --license apache2 --owner "zoux86"
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.11.2
go: downloading sigs.k8s.io/controller-runtime v0.11.2
go: downloading k8s.io/apimachinery v0.23.5
go: downloading k8s.io/client-go v0.23.5
go: downloading k8s.io/utils v0.0.0-20211116205334-6203023598ed
go: downloading k8s.io/component-base v0.23.5
go: downloading k8s.io/api v0.23.5
go: downloading k8s.io/apiextensions-apiserver v0.23.5
go: downloading sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
go: downloading golang.org/x/net v0.0.0-20211209124913-491a49abca63
go: downloading golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
Update dependencies:
$ go mod tidy
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
Next: define a resource with:
$ kubebuilder create api

查看文件目录

k8s apis通常有三个组件Resource, Controller, Manager，它们分别定义/实现在以下的三个package当中：

• cmd/...：主流程程序Manager入口，负责初始化依赖包、启停Controller。用户通常不需要编辑此包，可以依赖脚手架。通过kubebuilder init自动创建生成。
• pkg/apis/...：包含API资源的定义。编辑*_types.go文件来修改资源定义。每个资源的定义文件存在于pkg/apis/<api-group-name>/<api-version-name>/<api-kind-name>_types.go中。通过kubebuilder create api自动创建生成。
• pkg/controller/...：包含Controller的实现。编辑*_controller.go实现Controller。通过kubebuilder create api自动创建生成。

 ~/go/src/github.com/zoux86/operator-example  tree
.
├── Dockerfile
├── Makefile
├── PROJECT
├── README.md
├── config
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role.yaml
│       ├── leader_election_role_binding.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
└── main.go

3. 创建api和controller

其实从create api 后的输出我们可以看出来：我们修改逻辑后就可以部署了

 ~/go/src/github.com/zoux86/operator-example # ~/kubebuilder create api --group zouxapp --kind PodCount --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...     // 先修改这2个文件
Writing scaffold for you to edit...
api/v1/podcount_types.go
controllers/podcount_controller.go
Update dependencies:
$ go mod tidy                                   
Running make:
$ make generate                                     
mkdir -p /Users/game-netease/go/src/github.com/zoux86/operator-example/bin
GOBIN=/Users/game-netease/go/src/github.com/zoux86/operator-example/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.8.0
go: downloading sigs.k8s.io/controller-tools v0.8.0
go: downloading github.com/spf13/cobra v1.2.1
go: downloading golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go: downloading github.com/fatih/color v1.12.0
go: downloading k8s.io/api v0.23.0
go: downloading k8s.io/apimachinery v0.23.0
go: downloading github.com/gobuffalo/flect v0.2.3
go: downloading k8s.io/apiextensions-apiserver v0.23.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go: downloading golang.org/x/mod v0.4.2
/Users/game-netease/go/src/github.com/zoux86/operator-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

执行create api后，生成以下文件：

api/v1/groupversion_info.go
api/v1/podcount_types.go                      // 需要修改这个文件中crd的定义
api/v1/zz_generated.deepcopy.go
config/crd/kustomization.yaml
config/crd/kustomizeconfig.yaml
config/crd/patches/cainjection_in_podcounts.yaml
config/crd/patches/webhook_in_podcounts.yaml
config/rbac/podcount_editor_role.yaml
config/rbac/podcount_viewer_role.yaml
config/samples/zouxapp_v1_podcount.yaml
controllers/podcount_controller.go            // 需要修改这个文件的controller运行逻辑
controllers/suite_test.go
go.mod
main.go

4. 实现自己的crd和控制器逻辑

根据实际情况而定，这里的控制器逻辑很简单，就是创建同步podCount的spec.count到status里面。

func (r *PodCountReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    rlog := log.FromContext(ctx)
    rlog.Info("start to reconciling podCount %s", req.Name)
    podCount := &zouxappv1.PodCount{}
    err := r.Client.Get(ctx, req.NamespacedName, podCount)
    if err != nil {
        rlog.Error(err, fmt.Sprintf("get podcount %s/%s err during reconcile.", req.Namespace, req.Name))
        return ctrl.Result{}, nil
    }
    podCountCopy := podCount.DeepCopy()
    if podCount.Spec.Count <= 0 {
        podCountCopy.Status.Count = 0
    } else {
        podCountCopy.Status.Count = podCount.Spec.Count
    }

    err = r.Client.Status().Update(ctx, podCountCopy)
    if err != nil {
        rlog.Error(err, fmt.Sprintf("update crd podcount status error %s/%s  during reconcile.", req.Namespace, req.Name))
    }
    //r.Status().Update(ctx, podCountCopy, metav1.UpdateOptions{})
    // TODO(user): your logic here

    return ctrl.Result{}, err
}

5. make manifests, 创建crd的相关yaml

 ~/go/src/github.com/zoux86/operator-example # make manifests
/Users/game-netease/go/src/github.com/zoux86/operator-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

执行make manifests之后，我们会得到2个文件。

config/crd/bases/
config/rbac/role.yaml


# cat config/rbac/role.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - zouxapp.github.com
  resources:
  - podcounts
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - zouxapp.github.com
  resources:
  - podcounts/finalizers
  verbs:
  - update
- apiGroups:
  - zouxapp.github.com
  resources:
  - podcounts/status
  verbs:
  - get
  - patch
  - update

# cat config/crd/bases/zouxapp.github.com_podcounts.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.8.0
  creationTimestamp: null
  name: podcounts.zouxapp.github.com
spec:
  group: zouxapp.github.com
  names:
    kind: PodCount
    listKind: PodCountList
    plural: podcounts
    singular: podcount
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: PodCount is the Schema for the podcounts API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: PodCountSpec defines the desired state of PodCount
            properties:
              count:
                description: Foo is an example field of PodCount. Edit podcount_types.go
                  to remove/update
                type: integer
            type: object
          status:
            description: PodCountStatus defines the observed state of PodCount
            properties:
              count:
                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
                  of cluster Important: Run "make" to regenerate code after modifying
                  this file'
                type: integer
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

6. 在集群中部署crd

 ~/go/src/github.com/zoux86/operator-example #kubectl  --kubeconfig=kubeconfig get node create -f config/crd/bases
customresourcedefinition.apiextensions.k8s.io/podcounts.zouxapp.github.com created

 ~/go/src/github.com/zoux86/operator-example # kubectl --kubeconfig=kubeconfig create -f config/samples/zouxapp_v1_podcount.yaml 
podcount.zouxapp.github.com/podcount-sample created

上集群验证，可以看到创建成功了，但是可以看出来没有status.count，这个因为集群还没部署控制器

root# kubectl get crd   | grep podc
podcounts.zouxapp.github.com                            2022-08-25T06:57:09Z


root # kubectl get podcounts.zouxapp.github.com
NAME              AGE
podcount-sample   11s

root # kubectl get podcounts.zouxapp.github.com -oyaml
apiVersion: v1
items:
- apiVersion: zouxapp.github.com/v1
  kind: PodCount
  metadata:
    creationTimestamp: "2022-08-25T07:01:16Z"
    generation: 1
    name: podcount-sample
    namespace: default
    resourceVersion: "467368378"
    selfLink: /apis/zouxapp.github.com/v1/namespaces/default/podcounts/podcount-sample
    uid: a8b42a4c-1ebd-430a-890f-b0238f4ad125
  spec:
    count: 3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

7. 部署controller

之前 CRD 并不会完成任何工作，只是在 ETCD 中创建了一条记录。所以我们需要部署写的controller。

运行CRD controller

 ~/go/src/github.com/zoux86/operator-example ## go run ./main.go
I0825 15:35:12.827589   63628 request.go:665] Waited for 1.000074041s due to client-side throttling, not priority and fairness, request: GET:https://xxx/apis/apiextensions.k8s.io/v1?timeout=32s
1.6614129137223601e+09  INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
1.6614129137230568e+09  INFO    setup   starting manager
1.661412913723448e+09   INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.661412913723448e+09   INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.661412913723506e+09   INFO    controller.podcount     Starting EventSource    {"reconciler group": "zouxapp.github.com", "reconciler kind": "PodCount", "source": "kind source: *v1.PodCount"}
1.661412913723542e+09   INFO    controller.podcount     Starting Controller     {"reconciler group": "zouxapp.github.com", "reconciler kind": "PodCount"}
1.661412913825421e+09   INFO    controller.podcount     Starting workers        {"reconciler group": "zouxapp.github.com", "reconciler kind": "PodCount", "worker count": 1}
I0825 15:35:13.825542   63628 podcount_controller.go:50] start to reconciling podCount podcount-sample
I0825 15:35:13.868618   63628 podcount_controller.go:50] start to reconciling podCount podcount-sample

查看发现生效了

root# kubectl get podcounts.zouxapp.github.com -oyaml
apiVersion: v1
items:
- apiVersion: zouxapp.github.com/v1
  kind: PodCount
  metadata:
    creationTimestamp: "2022-08-25T07:01:16Z"
    generation: 1
    name: podcount-sample
    namespace: default
    resourceVersion: "467385745"
    selfLink: /apis/zouxapp.github.com/v1/namespaces/default/podcounts/podcount-sample
    uid: a8b42a4c-1ebd-430a-890f-b0238f4ad125
  spec:
    count: 3
  status:
    count: 3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""