kubeadm升级集群版本

CentOS kubeadm升级集群版本

1，找到最新的稳定版本

yum list --showduplicates kubeadm  kubectl kubelet  --disableexcludes=kubernetes
# 在列表中查找最新的 1.19 版本
# 它看起来应该是 1.19.x-0 ，其中 x 是最新的补丁版本

2，安装稳定版本

yum --enablerepo=kubernetes install kubelet-1.19.0-0 kubeadm-1.19.0-0 kubectl-1.19.0-0 -y

3，验证下载的版本

[root@k8s-master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:28:32Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

4，将master节点设为不可调度，drain驱逐节点上的pod

kubectl drain k8s-master --ignore-daemonsets 

5，在master上运行kubeadm upgrade plan会看见一下信息

[root@k8s-master ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.18.0
[upgrade/versions] kubeadm version: v1.19.0
I0309 11:07:29.493505  101466 version.go:252] remote version is much newer: v1.20.4; falling back to: stable-1.19
[upgrade/versions] Latest stable version: v1.19.8
[upgrade/versions] Latest stable version: v1.19.8
[upgrade/versions] Latest version in the v1.18 series: v1.18.16
[upgrade/versions] Latest version in the v1.18 series: v1.18.16

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
kubelet     3 x v1.18.0   v1.18.16

Upgrade to the latest version in the v1.18 series:

COMPONENT                 CURRENT   AVAILABLE
kube-apiserver            v1.18.0   v1.18.16
kube-controller-manager   v1.18.0   v1.18.16
kube-scheduler            v1.18.0   v1.18.16
kube-proxy                v1.18.0   v1.18.16
CoreDNS                   1.6.7     1.7.0
etcd                      3.4.3-0   3.4.3-0

You can now apply the upgrade by executing the following command:

   kubeadm upgrade apply v1.18.16

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
kubelet     3 x v1.18.0   v1.19.8

Upgrade to the latest stable version:

COMPONENT                 CURRENT   AVAILABLE
kube-apiserver            v1.18.0   v1.19.8
kube-controller-manager   v1.18.0   v1.19.8
kube-scheduler            v1.18.0   v1.19.8
kube-proxy                v1.18.0   v1.19.8
CoreDNS                   1.6.7     1.7.0
etcd                      3.4.3-0   3.4.9-1

You can now apply the upgrade by executing the following command:

   kubeadm upgrade apply v1.19.8

Note: Before you can perform this upgrade, you have to update kubeadm to v1.19.8.

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

此命令检查你的集群版本是否可以升级，并获取升级的版本
kubeadm upgrade 也会自动对它在此节点上管理的证书进行续约。 如果选择不对证书进行续约，可以使用标志 --certificate-renewal=false

6，升级版本执行kubeadm upgrade apply

kubeadm upgrade apply v1.19.0会输出一下信息：

[root@k8s-master ~]# kubeadm upgrade apply v1.19.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.19.0"
[upgrade/versions] Cluster version: v1.18.0
[upgrade/versions] kubeadm version: v1.19.0
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.19.0"...
Static pod: kube-apiserver-k8s-master hash: 5819395e02ad4540ffe60ef0d5a679c3
Static pod: kube-controller-manager-k8s-master hash: c4d2dd4abfffdee4d424ce839b0de402
Static pod: kube-scheduler-k8s-master hash: ca2aa1b3224c37fa1791ef6c7d883bbe
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-master hash: c2cbe28f923fb79739b0aec2e85fd8b8
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-03-09-11-13-19/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-k8s-master hash: c2cbe28f923fb79739b0aec2e85fd8b8
Static pod: etcd-k8s-master hash: 25120d5aa2ebd8aa050647a332cd58b0
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests280098751"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-03-09-11-13-19/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8s-master hash: 5819395e02ad4540ffe60ef0d5a679c3
Static pod: kube-apiserver-k8s-master hash: 5819395e02ad4540ffe60ef0d5a679c3
Static pod: kube-apiserver-k8s-master hash: 5b7230c771e95a932ff0f3b4815061cc
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-03-09-11-13-19/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8s-master hash: c4d2dd4abfffdee4d424ce839b0de402
Static pod: kube-controller-manager-k8s-master hash: 8a39cf854f2a9e5732426c840246e36c
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-03-09-11-13-19/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8s-master hash: ca2aa1b3224c37fa1791ef6c7d883bbe
Static pod: kube-scheduler-k8s-master hash: 340ea85a0f34a4df64d62b1a784833ae
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
W0309 11:13:43.712989  103119 dns.go:282] the CoreDNS Configuration will not be migrated due to unsupported version of CoreDNS. The existing CoreDNS Corefile configuration and deployment has been retained.
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.19.0". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

7，恢复master为可调度并重启kubelet

kubectl uncordon k8s-master
systemctl daemon-reload 
systemctl restart kubelet.service

#查看集群状态
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   28m   v1.19.0
k8s-node1    Ready    <none>   27m   v1.18.0
k8s-node2    Ready    <none>   27m   v1.18.0

8，升级node节点

#在node节点安装稳定版
yum --enablerepo=kubernetes install kubelet-1.19.0-0 kubeadm-1.19.0-0 kubectl-1.19.0-0 -y

9，将node节点设为不可调度

#在master上执行
kubectl drain  k8s-node1 --ignore-daemonsets
#会输出一下信息
[root@k8s-master ~]# kubectl drain  k8s-node1 --ignore-daemonsets 
node/k8s-node1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-rw8z7
evicting pod kube-system/coredns-7ff77c879f-hxg68
pod/coredns-7ff77c879f-hxg68 evicted
node/k8s-node1 evicted

10，升级node节点

#在node节点上执行
[root@k8s-node1 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

11，重启kubelet并设置node节点为可调度

#在node上执行
systemctl daemon-reload 
systemctl restart kubelet.service
#在master上执行
kubectl uncordon k8s-node1

12，验证节点

[root@k8s-master ~]# kubectl get node
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   4h28m   v1.19.0
k8s-node1    Ready    <none>   4h27m   v1.19.0
k8s-node2    Ready    <none>   4h27m   v1.19.0

官方文档：

https://v1-18.docs.kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/