构建Kubernetes集群

作者: bear_small | 来源:发表于2021-07-08 13:39 被阅读0次

对应的每台设备都需要操作 kubernetes-master kubernetes-node1 kubernetes-node2

1. 修改主机名

更改主机名
hostnamectl set-hostname kubernetes-master
配置 hosts

cat >> /etc/hosts << EOF
192.168.141.150 kubernetes-master
EOF

2. 配置静态IP

为 Master 和 Node 节点单独配置对应的 IP
编辑配置文件
vim /etc/netplan/50-cloud-init.yaml

image.png

配置生效
netplan apply

3. 配置kubeadm

安装概述
安装 kubernetes 主要是安装它的各个镜像，而 kubeadm 已经为我们集成好了运行 kubernetes 所需的基本镜像。但由于国内的网络原因，在搭建环境时，无法拉取到这些镜像。此时我们只需要修改为阿里云提供的镜像服务即可解决该问题。

创建并修改配置

新建配置目录（后续自定义配置都置于此目录）
mkdir -p /usr/local/kubernetes

在上述目录导出配置文件
kubeadm config print init-defaults --kubeconfig ClusterConfiguration > kubeadm.yml

修改配置如下
#修改为当前服务器（主节点）的IP地址 
advertiseAddress: 192.168.141.150 
#国内不能访问 Google，修改为阿里云 
imageRepository: registry.aliyuncs.com/google_containers  
#默认为当前下载kubernetes版本的大版本中的第一个版本，如我下载的是1.20.1，此处为1.20.0,更新为 > 1.20.1即可 
kubernetesVersion: v1.20.1   
 # 配置成 Calico 的默认网段，属性没有则添加 
podSubnet: "10.244.0.0/16"   

以上配置会有个小坑，即Calico 的默认网段，此处，因为我虚拟机的网段是192.168.141.0/24,此处就会和Calico 的默认网段产生冲突，建议更换其他网段（此处说明是因为我安装单节点集群时没问题，但是安装高可用时就产生了问题，具体问题后面会单独说明分析）

拉取镜像

查看所需镜像列表
kubeadm config images list --config kubeadm.yml

拉取镜像
kubeadm config images pull --config kubeadm.yml
image.png

4. 使用 kubeadm 构建 kubernetes 集群-主节点

安装kubernetes主节点
kubeadm init --config=kubeadm.yml --upload-certs | tee kubeadm-init.log
执行以上命令初始化主节点，该命令指定了初始化时需要使用的配置文件
--experimental-upload-certs(在v1.16及以上版本会报错, 新版本不再支持此参数了，变更为：--upload-certs即可,见: http://www.suoniao.com/article/5e7b9559ab939f14c42a9784) 参数可以在后续执行加入节点时自动分发证书文件。
追加的 tee kubeadm-init.log 用以输出日志。

安装成功则会有如下输出

[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
  [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The > recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes-master kubernetes.default > kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.129]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubernetes-master localhost] and IPs > [192.168.2.129 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubernetes-master localhost] and IPs > [192.168.2.129 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory > "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 73.009161 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" > Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for > the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
4def8ffa89bd642c033713e0b715ee074341987164c2ae23fc4394536203c421
[mark-control-plane] Marking the node kubernetes-master as control-plane by adding the labels "node-> role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node kubernetes-master as control-plane by adding the taints [node-> role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes > to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs > from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the > cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate > and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.129:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash > sha256:1d5c20e7497d1c710c2f83aa1de8cdc2a8c73c7dfd005e1d155a0360cc510d72 
root@kubernetes-master:/usr/local/kubernetes#

注意：如果安装 kubernetes 版本和下载的镜像版本不统一则会出现 timed out waiting for the condition 错误。
中途失败或是想修改配置可以使用 kubeadm reset 命令重置配置，再做初始化操作即可。
如果执行kubeadm reset后,执行kubectl get nodes报错(权限相关),
则删除$HOME/.kube目录下的.kube文件(rm -rf $HOME/.kube)
重新执行mkdir -p $HOME/.kube和cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

配置 kubectl
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
非 ROOT 用户执行
chown $(id -u):$(id -g) $HOME/.kube/config

验证
kubectl get node
能够打印出节点信息即表示成功

至此主节点配置完成

说明：
kubeadm init 的执行过程
init：指定版本进行初始化操作
preflight：初始化前的检查和下载所需要的 Docker 镜像文件
kubelet-start：生成 kubelet 的配置文件 var/lib/kubelet/config.yaml，没有这个文件 kubelet 无法启动，所以初始化之前的 kubelet 实际上启动不会成功
certificates：生成 Kubernetes 使用的证书，存放在 /etc/kubernetes/pki 目录中
kubeconfig：生成 KubeConfig 文件，存放在 /etc/kubernetes 目录中，组件之间通信需要使用对应文件
control-plane：使用 /etc/kubernetes/manifest 目录下的 YAML 文件，安装 Master 组件
etcd：使用 /etc/kubernetes/manifest/etcd.yaml 安装 Etcd 服务
wait-control-plane：等待 control-plan 部署的 Master 组件启动
apiclient：检查 Master 组件服务状态。
uploadconfig：更新配置
kubelet：使用 configMap 配置 kubelet
patchnode：更新 CNI 信息到 Node 上，通过注释的方式记录
mark-control-plane：为当前节点打标签，打了角色 Master，和不可调度标签，这样默认就不会使用 Master 节点来运行 Pod
bootstrap-token：生成 token 记录下来，后边使用 kubeadm join 往集群中添加节点时会用到
addons：安装附加组件 CoreDNS 和 kube-proxy

5. 使用 kubeadm 配置 slave 节点

将 slave 节点加入到集群中很简单，只需要在 slave 服务器上安装 kubeadm，kubectl，kubelet 三个工具，然后使用 kubeadm join 命令加入即可。准备工作如下：
- 修改主机名
- 配置软件源
- 安装三个工具(kubectl,kubeadm,kubelet)
子节点加入集群

kubeadm join 192.168.141.150:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:1d5c20e7497d1c710c2f83aa1de8cdc2a8c73c7dfd005e1d155a0360cc510d72

在master节点查看集群节点信息，如下
kubectl get nodes
image.png
如果配置子节点过程中出现问题,使用kubeadm reset重置,此时kubectl get nodes查看仍然存在已重置的节点,需要执行 kubectl delete nodes Kubernetes-node1进行删除,然后重新安装和进行join操作

6. 安装网络插件 Calico

参考官方文档安装：https://docs.projectcalico.org/getting-started/kubernetes/quickstart

下载文件，方便更改
wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml
更改默认的网段，前面有说明
vim calico.yaml
将 192.168.0.0/16 修改为 10.244.0.0/16
命令快速查找
- 显示行号：:set number
- 查找字符：/要查找的字符，输入小写 n 下一个匹配项，输入大写 N 上一个匹配项
安装（Master 节点操作）
kubectl apply -f calico.yaml
确认安装是否成功
watch kubectl get pods --all-namespaces
需要等待所有状态为 Running，注意时间可能较久，3 - 5 分钟的样子

7. 运行Kubernetes 容器

检查组件运行状态
scheduler和controller-manager显示Unhealthy
异常 scheduler controller-manager connection refused
解决方案:
出现这种情况，是/etc/kubernetes/manifests下的kube-controller-manager.yaml和kube-scheduler.yaml设置> 的默认端口是0，在文件中注释掉就可以了
重启kubelet服务: systemctl restart kubelet.service

检查 Master 状态,输出如下
kubectl cluster-info

# 主节点状态
Kubernetes control plane is running at [<u>https://192.168.2.129:6443</u>](https://192.168.2.129:6443)
# DNS 状态
KubeDNS is running at https://192.168.2.129:6443/api/v1/namespaces/kube-system/services/kube-> dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

检查 Nodes 状态
kubectl get nodes
输出如下，STATUS 为 Ready 即为正常状态
image.png

运行容器实例
replicas说明已经被废弃,未来版本中会删除, 推荐用 deployment 创建 pods创建yaml文件
image.png
创建pods
kubectl apply -f nginx-deployment.yaml
image.png

各类状态获取命令

查看全部 Pods 的状态(需要等待一小段实践，STATUS 为 Running 即为运行成功)
kubectl get pods
image.png

查看已部署的服务
kubectl get deployment
image.png

映射服务，让用户可以访问
kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer
image.png

查看已发布的服务
kubectl get services
image.png
Nginx 服务已成功发布并将 80 端口映射为 31218

验证
直接浏览器访问ip:3121
此时 Kubernetes 会以负载均衡的方式访问部署的 Nginx 服务，能够正常看到 Nginx 的欢迎页即表示成功。容器实际部署在其它 Node 节点上，通过访问 Node 节点的 IP:Port 也是可以的

停止服务
kubectl delete deployment nginx-deployment
kubectl delete service nginx-deployment
image.png
image.png