Kubeadm是Kubernetes官方提供的用于快速安装Kubernetes集群的工具，伴随Kubernetes每个版本的发布都会同步更新，在2018年将进入GA状态，说明离生产环境中使用的距离越来越近了。
使用Kubeadm搭建Kubernetes集群本来是件很简单的事，但由于众所周知的原因，在中国大陆是无法访问 k8s.gcr.io的。这就使我们无法按照官方的教程来创建集群。而国内的教程参差不齐，大多也无法运行成功，我也是踩了很多坑，才部署成功，故在此分享出来。
准备
安装环境准备
| IP地址 | 主机名 | CPU | 内存 | 磁盘 |
|---|---|---|---|---|
| 10.136.157.23 | kube-node1 | 32C | 128G | 2T |
| 10.136.157.24 | kube-node2 | 32C | 128G | 2T |
| 10.136.157.25 | kube-node3 | 32C | 128G | 2T |
关闭防火墙和selinux
# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
# 禁用SELINUX
setenforce 0
vim /etc/selinux/config
SELINUX=disabled
关闭系统的Swap
Kubernetes 1.8开始要求必须禁用Swap，如果不关闭，默认配置下kubelet将无法启动。
# 关闭系统的Swap方法如下:
# 编辑`/etc/fstab`文件，注释掉引用`swap`的行，保存并重启后输入:
sudo swapoff -a
验证Mac地址和product_uuid是否唯一（可选）
Kubernetes要求集群中所有机器具有不同的Mac地址、产品uuid、Hostname。可以使用如下命令查看：
# UUID
cat /sys/class/dmi/id/product_uuid
# Mac地址
ip link
# Hostname
cat /etc/hostname
安装Docker
Kubernetes从1.6开始使用CRI(Container Runtime Interface)容器运行时接口。默认的容器运行时仍然是Docker，是使用kubelet中内置dockershim CRI来实现的。
sudo yum install -y docker
安装kubeadm, kubelet 和 kubectl
部署之前，我们需要安装一下三个包：
- kubeadm: 引导启动k8s集群的命令行工具。
- kubelet: 在群集中所有节点上运行的核心组件, 用来执行如启动pods和containers等操作。
- kubectl: 操作集群的命令行工具。
修改yum源
安装kubeadm, kubelet 和 kubectl
查看可用版本
sudo yum list --showduplicates | grep 'kubeadm\|kubectl\|kubelet'
安装 kubeadm1.13.1, kubelet1.13.1 和 kubectl1.13.1
yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 --disableexcludes=kubernetes
此时还不能启动 kubelet，先设置开机启动：
systemctl enable kubelet
安装kubernetes Docker镜像
Kubernetes国内Docker镜像
kubeadm init从私有仓库拉取镜像
初始化 Master 节点
配置文件
# 生成配置文件
kubeadm config print-default > kubeadm.conf
# 修改配置文件
# 修改镜像仓储地址
sed -i "s/imageRepository: .*/imageRepository: registry.cn-beijing.aliyuncs.com/imcto/g" kubeadm.conf
# 修改版本号
sed -i "s/kubernetesVersion: .*/kubernetesVersion: v1.13.1/g" kubeadm.conf
sed -i "s/podSubnet: .*/podSubnet: \"10.244.0.0\/16\"/g" kubeadm.conf
初始化
拉取镜像
$ sudo kubeadm config images pull --config kubeadm.conf
[config/images] Pulled docker2.yidian.com:5000/k8simages/kube-apiserver:v1.13.1
[config/images] Pulled docker2.yidian.com:5000/k8simages/kube-controller-manager:v1.13.1
[config/images] Pulled docker2.yidian.com:5000/k8simages/kube-scheduler:v1.13.1
[config/images] Pulled docker2.yidian.com:5000/k8simages/kube-proxy:v1.13.1
[config/images] Pulled docker2.yidian.com:5000/k8simages/pause:3.1
[config/images] Pulled docker2.yidian.com:5000/k8simages/etcd:3.2.24
[config/images] Pulled docker2.yidian.com:5000/k8simages/coredns:1.2.6
可以看到，已经成功拉取了需要的镜像
开始初始化
此处可能会遇到一个坑 安装Kubernetes报错：The kubelet is not running
$ sudo kubeadm init --config kubeadm.conf
[init] Using Kubernetes version: v1.13.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube-node1 localhost] and IPs [10.136.157.23 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube-node1 localhost] and IPs [10.136.157.23 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube-node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.136.157.23]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.501577 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node1" as an annotation
[mark-control-plane] Marking the node kube-node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kube-node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 10.136.157.23:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952
开启8080端口
# 修改配置文件
sudo sed -i 's/insecure-port=0/insecure-port=8080/g' /etc/kubernetes/manifests/kube-apiserver.yaml
# 重启docker镜像
sudo docker ps |grep 'kube-apiserver_kube-apiserver'|awk '{print $1}'|head -1|xargs sudo docker restart
设置 Node 节点加入集群
kubeadm init 命令的输出日志中已经告知我们加入新节点的方式。如果忘记了 Master 的 Token，可以在 Master 上输入以下命令查看：
$ sudo kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
abcdef.0123456789abcdef 9h 2019-03-22T19:35:14+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Token
kubeadm token create
获取ca证书sha256编码hash值
$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952
加入新的Node节点
$ sudo kubeadm join 10.136.157.23:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.136.157.23:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.136.157.23:6443"
[discovery] Requesting info from "https://10.136.157.23:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.136.157.23:6443"
[discovery] Successfully established connection with API Server "10.136.157.23:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node2" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
启用IPV6
注意：需要在新加入的Node上启用IPV6，否则会导致 coredns 启动失败，始终处于 ContainerCreating 状态
sudo sed -i 's\ipv6.disable=1\ipv6.disable=0\g' /etc/default/grub
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
安装网络插件
为了让Pods间可以相互通信，我们必须安装一个网络插件，并且必须在部署任何应用之前安装，CoreDNS也是在网络插件安装之后才会启动的。
网络的插件完整列表，请参考 Networking and Network Policy。
在安装之前，我们先查看一下当前Pods的状态：
$ sudo kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58d6b7c8d7-f5mdg 0/1 Pending 0 57m
kube-system coredns-58d6b7c8d7-fkpmc 0/1 Pending 0 57m
kube-system etcd-kube-node1 1/1 Running 1 56m
kube-system kube-apiserver-kube-node1 1/1 Running 7 10m
kube-system kube-controller-manager-kube-node1 1/1 Running 2 56m
kube-system kube-proxy-2xrzt 1/1 Running 1 57m
kube-system kube-scheduler-kube-node1 1/1 Running 2 56m
如上，可以看到CoreDND的状态是Pending，就是因为我们还没有安装网络插件。
安装RBAC
wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
sudo kubectl apply -f rbac.yaml
安装Flannel
sudo sysctl net.bridge.bridge-nf-call-iptables=1
Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ sudo kubectl apply -f kube-flannel.yml
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
通过 ifconfig 可以看到新建了一张虚拟网卡：
$ ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.0 netmask 255.255.255.255 broadcast 0.0.0.0
ether 7a:d1:26:11:ca:ae txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Master隔离（可选）
默认情况下，由于安全原因，集群并不会将pods部署在Master节点上。但是在开发环境下，我们可能就只有一个Master节点，这时可以使用下面的命令来解除这个限制：
$ sudo kubectl taint nodes --all node-role.kubernetes.io/master-
## 输出
node/ubuntu1 untainted
等待一会，我们可以在Master节点上使用kubectl get nodes命令来查看节点的状态：
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-node1 Ready master 3h3m v1.13.3
kube-node2 Ready <none> 118s v1.13.3
kube-node3 Ready <none> 95s v1.13.3
安装Dashboard UI
Dashboard默认是不安装的，要想安装Dashboard，可以执行下面的命令：
$ wget https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml
# 修改kubernetes-dashboard.yaml
spec:
containers:
- name: kubernetes-dashboard
image: registry.cn-beijing.aliyuncs.com/imcto/kubernetes-dashboard-amd64:v1.10.1
# 创建Dashboard UI
$ sudo kubectl create -f kubernetes-dashboard.yaml
注意：如果没有必要，不用解除 --apiserver-host 的注释
访问Dashboard UI
修改kubernetes-dashboard配置
$ kubectl -n kube-system edit service kubernetes-dashboard
将type: ClusterIP 修改为 type: NodePort，并保存文件。
检查 Dashboard UI 暴露的服务端口
$ sudo kubectl -n kube-system get service kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.110.146.196 <none> 443:32275/TCP 42m
可以看到，Dashboard UI 的服务端口是32275，协议为HTTPS，我们可以https://<master-ip>:32275来进行访问。
遇到的问题
初始化 Kubernetes 问题（端口占用）
安装Kubernetes报错：The connection to the server localhost:8080 was refused
安装Kubernetes报错：STATUS NotReady
安装Kubernetes报错：/proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
安装Kubernetes报错：kubernetes-dashboard状态CrashLoopBackOff
参考文章
Centos7 使用 kubeadm 安装Kubernetes 1.13.3
使用Kubeadm搭建Kubernetes(1.12.2)集群
在国内如何巧妙获取kubernetes各镜像？
国内借助阿里云快速获取gcr.io上的镜像
解决kubeadm部署kubernetes集群镜像问题
kubeadm 生成的token过期后，集群增加节点
kubeadm 安装 kubernetes
网友评论