Kubernetes 开发环境搭建

作者: 河码匠 | 来源:发表于2023-12-17 11:14 被阅读0次

一. 基本环境

操作系统环境 ubuntu20.04 (kubernetes的kube kubelet 不支持 macos M1。项目也能起来，但是没法创建节点 )
Kubernetes 版本 1.29
go 版本 1.21.4

二. github 下载 Kubernetes

根据 GOPATH 目录将 Kubernetes 源码下载到 {$GOPATH}/src/k8s.io/ 里面
我的 GOPATH="/root/go"

└── go
    ├── pkg
    └── src
        └── k8s.io
            └── kubernetes
                ├── CHANGELOG
                ├── CHANGELOG.md -> CHANGELOG/README.md
                ├── CONTRIBUTING.md
                ├── LICENSE
                ├── LICENSES
                ├── Makefile -> build/root/Makefile
                ├── OWNERS
                ├── OWNERS_ALIASES
                ├── README.md
                ├── SECURITY_CONTACTS
                ├── SUPPORT.md
                ├── _output
                ├── api
                ├── build
                ├── cluster
                ├── cmd
                ├── code-of-conduct.md
                ├── docs
                ├── go.mod
                ├── go.sum
                ├── hack
                ├── logo
                ├── pkg
                ├── plugin
                ├── staging
                ├── test
                ├── third_party
                └── vendor

三. 安装环境需要的软件

官网开发环境说明

1. 安装软件

ubuntu

apt install jq
apt install gnome-terminal
apt-get install ca-certificates curl gnupg

mac

brew install coreutils ed findutils gawk gnu-sed gnu-tar grep make jq

2. 安装 `Docker`

ubuntu

apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

mac
直接下载 Docker 客户端安装即可

3. 安装 Go

安装 go 需要的库

go get ./...

安装 cfssl

go install github.com/cloudflare/cfssl/cmd/...@latest

4. 安装 `PyYAML`

pip3 install PyYAML

5. 执行 `etcd` 安装脚本 `./kubernetes/hack/install-etcd.sh`

# ./hack/install-etcd.sh

Downloading https://github.com/etcd-io/etcd/releases/download/v3.5.10/etcd-v3.5.10-darwin-arm64.zip succeed
Archive:  etcd-v3.5.10-darwin-arm64.zip
   creating: etcd-v3.5.10-darwin-arm64/
  inflating: etcd-v3.5.10-darwin-arm64/README.md
  inflating: etcd-v3.5.10-darwin-arm64/READMEv2-etcdctl.md
  inflating: etcd-v3.5.10-darwin-arm64/etcdutl
  inflating: etcd-v3.5.10-darwin-arm64/etcdctl
   creating: etcd-v3.5.10-darwin-arm64/Documentation/
  inflating: etcd-v3.5.10-darwin-arm64/Documentation/README.md
   creating: etcd-v3.5.10-darwin-arm64/Documentation/dev-guide/
   creating: etcd-v3.5.10-darwin-arm64/Documentation/dev-guide/apispec/
   creating: etcd-v3.5.10-darwin-arm64/Documentation/dev-guide/apispec/swagger/
  inflating: etcd-v3.5.10-darwin-arm64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
  inflating: etcd-v3.5.10-darwin-arm64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
  inflating: etcd-v3.5.10-darwin-arm64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
  inflating: etcd-v3.5.10-darwin-arm64/README-etcdutl.md
  inflating: etcd-v3.5.10-darwin-arm64/README-etcdctl.md
  inflating: etcd-v3.5.10-darwin-arm64/etcd
etcd v3.5.10 installed. To use:
export PATH="/root/g/src/k8s.io/kubernetes/third_party/etcd:${PATH}"

将 PATH="/root/go/src/k8s.io/kubernetes/third_party/etcd:${PATH}" 添加到环境变量里面去

四. 添加环境变量

1. 定义容器运行时

export CONTAINER_RUNTIME_ENDPOINT="unix:///run/containerd/containerd.sock"

2. 其他环境变量

ubuntu
vim ~/.profile

export GOPATH="/root/go/"
export PATH="/root/go/src/k8s.io/kubernetes/third_party/etcd:${PATH}"
export PATH="$PATH:/usr/local/go/bin"
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig

mac
vim ~/.bashrc

GNUBINS="$(find `brew --prefix`/opt -type d -follow -name gnubin -print)"

for bindir in ${GNUBINS[@]}
do
  export PATH=$bindir:$PATH
done

export GOPATH="/Users/machao/Desktop/Projects/kubernetes"
# 这是 go 的 bin 目录
PATH="$/xxx/go/bin:$PATH"

export PATH

五. 执行 `make verify` 检测系统环境。

FAILED TESTS
========================
# 我这里报下面错这个错误，需要"上网"才能下载镜像
hack/make-rules/../../hack/verify-codegen.sh

# 提示可以无视这个警告
hack/make-rules/../../hack/verify-golangci-lint.sh

# 这个报错一般都是 "GOPATH" 配置和项目所在位置不匹配
hack/make-rules/../../hack/verify-import-aliases.sh

# 这个报错执行"hack/update-openapi-spec.sh" 就好了
hack/make-rules/../../hack/verify-openapi-spec.sh

# 这里报错是没安装 pyyaml
hack/make-rules/../../hack/verify-publishing-bot.py

六. 构建 Kubernetes

1. `make` 构建 Kubernetes

make all 构建所有。也可以指定想要构建的组件，如： make WHAT=cmd/kubectl

# make all

go version go1.21.4 darwin/arm64
+++ [1206 11:39:39] Building go targets for darwin/arm64
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kubeadm (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
    k8s.io/component-base/logs/kube-log-runner (static)
    k8s.io/kube-aggregator (static)
    k8s.io/apiextensions-apiserver (static)
    k8s.io/kubernetes/cluster/gce/gci/mounter (static)
    k8s.io/kubernetes/cmd/kubectl (non-static)
    k8s.io/kubernetes/cmd/kubectl-convert (static)
    github.com/onsi/ginkgo/v2/ginkgo (non-static)
    k8s.io/kubernetes/test/e2e/e2e.test (test)
    k8s.io/kubernetes/test/conformance/image/go-runner (non-static)
    k8s.io/kubernetes/cmd/kubemark (static)
    github.com/onsi/ginkgo/v2/ginkgo (non-static)

2. 启动本地集群 `ENABLE_DAEMON=true DBG=1 ./hack/local-up-cluster.sh`

DBG=1 开启调试模式在启动过程中可以看到很多日志
ENABLE_DAEMON=true 启动守护进城，不然自己会退出

# ENABLE_DAEMON=true DBG=1 ./hack/local-up-cluster.sh 
.................
No resources found
No resources found
No resources found
No resources found
...........
Create default storage class for
storageclass.storage.k8s.io/standard created
Local Kubernetes cluster is running. Press Ctrl-C to shut it down.

Configurations:
  /private/var/folders/81/r5d92sr51yv2cn4h4b8fdd5c0000gn/T/local-up-cluster.sh.TMd54w/kube-audit-policy-file
  /private/var/folders/81/r5d92sr51yv2cn4h4b8fdd5c0000gn/T/local-up-cluster.sh.TMd54w/kube-scheduler.yaml
  /private/var/folders/81/r5d92sr51yv2cn4h4b8fdd5c0000gn/T/local-up-cluster.sh.TMd54w/kube-serviceaccount.key
  /private/var/folders/81/r5d92sr51yv2cn4h4b8fdd5c0000gn/T/local-up-cluster.sh.TMd54w/kube_egress_selector_configuration.yaml

Logs:
  /tmp/etcd.log
  /tmp/kube-apiserver.log
  /tmp/kube-controller-manager.log


  /tmp/kube-scheduler.log


To start using your cluster, you can open up another terminal/tab and run:

  export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
  cluster/kubectl.sh

Alternatively, you can write to the default kubeconfig:

  export KUBERNETES_PROVIDER=local

  cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
  cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
  cluster/kubectl.sh config set-context local --cluster=local --user=myself
  cluster/kubectl.sh config use-context local
  cluster/kubectl.sh

中途会有 No resources found 这个提示，不用管他，这个是在获取 node 节点信息。因为没创建出来所以提示 No resources found。等一会儿就好了

3. 配置 `kubectl` 配置文件

# export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig

七. 查看是否成功

1. 查看进城

# ps -a | grep kube
 248451 pts/0    00:00:10 kube-apiserver
 248595 pts/0    00:00:03 kube-controller
 248597 pts/0    00:00:01 kube-scheduler
 248708 pts/0    00:00:03 kubelet
 248857 pts/0    00:00:00 kube-proxy

2. 查看资源信息

# ./cluster/kubectl.sh cluster-info

Kubernetes control plane is running at https://localhost:6443/
CoreDNS is running at https://localhost:6443//api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

# ./cluster/kubectl.sh get nodes

NAME        STATUS   ROLES    AGE   VERSION
127.0.0.1   Ready    <none>   82s   v1.30.0-alpha.0.354+0a54839370b82d

# ./cluster/kubectl.sh get pods -A

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-69cbfb9798-xgwq9   1/1     Running   0          20m

八. 遇到的错误

1. 启动时发现 `kebelet` 没有启动

分析过程

查看日志 tmp/kubelet.log 发现下面这个错误

failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService

里面有一个 validate CRI v1 runtime API 发现没有运行

解决过程

重启 containerd 并查看日志 journalctl -f -u containerd 发现正常
查看 /etc/containerd/config.toml 发现 cri 禁用了。启用这个插件

Container Runtime Interface (CRI) 插件，它是 containerd 用于与 Kubernetes 通信的关键组件

#disabled_plugins = ["cri"]
disabled_plugins = []

重启 containerd

systemctl restart containerd

重启 kubernetes 集群，成功

2. `coredns` pod 为 `CrashLoopBackOff` 状态

# ./cluster/kubectl.sh get pods -A
NAMESPACE     NAME                       READY   STATUS             RESTARTS      AGE
kube-system   coredns-69cbfb9798-zph7c   0/1     CrashLoopBackOff   3 (35s ago)   80s

分析过程

查看 pod 日志

# ./cluster/kubectl.sh logs coredns-69cbfb9798-zph7c -n kube-system

.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/arm64, go1.20.7, ae2bbc2
[FATAL] plugin/loop: Loop (127.0.0.1:33907 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 1890283224315596440.6052109503740083691."

意思是说 DNS 是本地回环
查看 DNS

# cat /etc/resolv.conf
nameserver 127.0.0.53
search localdomain

coredns 官方解释

解决过程

尝试直接修改 /etc/resolv.conf

nameserver 192.168.17.2
search localdomain

然后重启 kubernetes 集群，发现 /etc/resolv.conf 又被改回去了（我发现只要重新 make all 就会被改回去。不知道怎么搞 - -）

安装 apt install resolvconf 他是管理系统的 DNS 解析配置

/run/resolvconf/interface/systemd-resolved 是 resolvconf 生成的
修改 /run/resolvconf/interface/systemd-resolved 这个是永久修改 DNS 方法

# cat /run/resolvconf/interface/systemd-resolved
nameserver 192.168.17.2
search localdomain

在尝试过程过发现 /run/systemd/resolve/resolv.conf 这里面也有一个 DNS

/run/systemd/resolve/resolv.conf 保存的是静态配置、DHCP 或 VPN 生成的 DNS
# ls -al /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Dec 22 02:42 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

sudo resolvconf -u 生效新的 dns
重启 kubernetes 集群（这里重启我是加 -O 参数，不重新 make），成功

Kubernetes 开发环境搭建

一. 基本环境

二. github 下载 Kubernetes

目录结构

三. 安装环境需要的软件

1. 安装软件

2. 安装 `Docker`

3. 安装 Go

4. 安装 `PyYAML`

5. 执行 `etcd` 安装脚本 `./kubernetes/hack/install-etcd.sh`

四. 添加环境变量

1. 定义容器运行时

2. 其他环境变量

五. 执行 `make verify` 检测系统环境。

六. 构建 Kubernetes

1. `make` 构建 Kubernetes

2. 启动本地集群 `ENABLE_DAEMON=true DBG=1 ./hack/local-up-cluster.sh`

3. 配置 `kubectl` 配置文件

七. 查看是否成功

1. 查看进城

2. 查看资源信息

八. 遇到的错误

1. 启动时发现 `kebelet` 没有启动

2. `coredns` pod 为 `CrashLoopBackOff` 状态

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

Kubernetes 开发环境搭建

一. 基本环境

二. github 下载 Kubernetes

目录结构

三. 安装环境需要的软件

1. 安装软件

2. 安装 Docker

3. 安装 Go

4. 安装 PyYAML

5. 执行 etcd 安装脚本 ./kubernetes/hack/install-etcd.sh

四. 添加环境变量

1. 定义容器运行时

2. 其他环境变量

五. 执行 make verify 检测系统环境。

六. 构建 Kubernetes

1. make 构建 Kubernetes

2. 启动本地集群 ENABLE_DAEMON=true DBG=1 ./hack/local-up-cluster.sh

3. 配置 kubectl 配置文件

七. 查看是否成功

1. 查看进城

2. 查看资源信息

八. 遇到的错误

1. 启动时发现 kebelet 没有启动

2. coredns pod 为 CrashLoopBackOff 状态

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

2. 安装 `Docker`

4. 安装 `PyYAML`

5. 执行 `etcd` 安装脚本 `./kubernetes/hack/install-etcd.sh`

五. 执行 `make verify` 检测系统环境。

1. `make` 构建 Kubernetes

2. 启动本地集群 `ENABLE_DAEMON=true DBG=1 ./hack/local-up-cluster.sh`

3. 配置 `kubectl` 配置文件

1. 启动时发现 `kebelet` 没有启动

2. `coredns` pod 为 `CrashLoopBackOff` 状态