kubeadm是Kubernetes官方提供的快速安装和初始化Kubernetes集群的工具，目前的还处于孵化开发状态，伴随Kubernetes每个版本的发布都会同步更新。 当然，目前的kubeadm是不能用于生产环境的。 但伴随着Kubernetes每次版本升级，kubeadm都会对集群配置方面的一些实践做调整，通过实验kubeadm我们可以学习到Kubernetes官方在集群配置上一些新的最佳实践。
Kubernetes 1.8已经发布，为了跟上官方更新的脚本，接下来体验一下Kubernetes 1.8中的kubeadm。
1.准备
1.1系统配置
在安装之前，需要先做如下准备。两台CentOS 7.3主机如下：
cat/etc/hosts192.168.61.11node1192.168.61.12node2
如果各个主机启用了防火墙，需要开放Kubernetes各个组件所需要的端口，可以查看Installing kubeadm中的”Check required ports”一节。 这里简单起见在各节点禁用防火墙：
systemctl stop firewalld
systemctl disable firewalld
创建/etc/sysctl.d/k8s.conf文件，添加如下内容：
net.bridge.bridge-nf-call-ip6tables=1net.bridge.bridge-nf-call-iptables=1
执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
禁用SELINUX：
setenforce0
vi/etc/selinux/config
SELINUX=disabled
Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。可以通过kubelet的启动参数–fail-swap-on=false更改这个限制。 我们这里关闭系统的Swap:
swapoff-a
修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。
swappiness参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行：
vm.swappiness=0
执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
1.2安装Docker
yum install-y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
查看当前的Docker版本：
yum list docker-ce.x86_64--showduplicates|sort-r
docker-ce.x86_6417.09.0.ce-1.el7.centosdocker-ce-stable
docker-ce.x86_6417.06.2.ce-1.el7.centosdocker-ce-stable
docker-ce.x86_6417.06.1.ce-1.el7.centosdocker-ce-stable
docker-ce.x86_6417.06.0.ce-1.el7.centosdocker-ce-stable
docker-ce.x86_6417.03.2.ce-1.el7.centosdocker-ce-stable
docker-ce.x86_6417.03.1.ce-1.el7.centosdocker-ce-stable
docker-ce.x86_6417.03.0.ce-1.el7.centosdocker-ce-stable
Kubernetes 1.8已经针对Docker的1.11.2, 1.12.6, 1.13.1和17.03.2等版本做了验证。 因为我们这里在各节点安装docker的17.03.2版本。
yum makecache fast
yum install-y--setopt=obsoletes=0\
docker-ce-17.03.2.ce-1.el7.centos\
docker-ce-selinux-17.03.2.ce-1.el7.centossystemctl start docker
systemctl enable docker
Docker从1.13版本开始调整了默认的防火墙规则，禁用了iptables filter表中FOWARD链，这样会引起Kubernetes集群中跨Node的Pod无法通信，在各个Docker节点执行下面的命令：
iptables-P FORWARD ACCEPT
可在docker的systemd unit文件中以ExecStartPost加入上面的命令：
ExecStartPost=/usr/sbin/iptables-P FORWARD ACCEPT
systemctl daemon-reload
systemctl restart docker
2.安装kubeadm和kubelet
下面在各节点安装kubeadm和kubelet：
cat</etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpghttps://packages.cloud.google.com/yum/doc/rpm-package-key.gpgEOF
测试地址https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64是否可用，如果不可用需要科学上网。
curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
yum makecache fast
yum install-y kubelet kubeadm kubectl...Installed:kubeadm.x86_640:1.8.0-0kubectl.x86_640:1.8.0-0kubelet.x86_640:1.8.0-0DependencyInstalled:kubernetes-cni.x86_640:0.5.1-0socat.x86_640:1.7.3.2-2.el7
从安装结果可以看出还安装了kubernetes-cni和socat两个依赖：
可以看出官方Kubernetes 1.8依赖的cni还是0.5.1版本
socat是kubelet的依赖
我们之前在Kubernetes 1.6 高可用集群部署中手动安装这两个依赖的
Kubernetes文档中kubelet的启动参数：
--cgroup-driverstringDriverthat the kubelet uses to manipulate cgroups on the host.Possiblevalues:'cgroupfs','systemd'(default"cgroupfs")
默认值为cgroupfs，但是我们注意到yum安装kubelet,kubeadm时生成10-kubeadm.conf文件中将这个参数值改成了systemd。
查看kubelet的 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf文件，其中包含如下内容：
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
使用docker info打印docker信息：
docker info......ServerVersion:17.03.2-ce......CgroupDriver:cgroupfs
可以看出docker 17.03使用的Cgroup Driver为cgroupfs。
于是修改各节点docker的cgroup driver使其和kubelet一致，即修改或创建/etc/docker/daemon.json，加入下面的内容：
{"exec-opts":["native.cgroupdriver=systemd"]}
重启docker:
systemctl restart docker
systemctl status docker
在各节点开机启动kubelet服务：
systemctl enable kubelet.service
3.使用kubeadm init初始化集群
接下来使用kubeadm初始化集群，选择node1作为Master Node，在node1上执行下面的命令：
kubeadm init \--kubernetes-version=v1.8.0\--pod-network-cidr=10.244.0.0/16\--apiserver-advertise-address=192.168.61.11
因为我们选择flannel作为Pod网络插件，所以上面的命令指定–pod-network-cidr=10.244.0.0/16。
kubeadm init \>--kubernetes-version=v1.8.0\>--pod-network-cidr=10.244.0.0/16\>--apiserver-advertise-address=192.168.61.11[kubeadm]WARNING:kubeadmisinbeta,pleasedonotuseitforproduction clusters.[init]UsingKubernetesversion:v1.8.0[init]UsingAuthorizationmodes:[NodeRBAC][preflight]Runningpre-flight checks[preflight]WARNING:firewalldisactive,pleaseensureports[644310250]are openoryour cluster maynotfunctioncorrectly[preflight]Startingthe kubelet service[kubeadm]WARNING:startingin1.8,tokens expire after24hoursbydefault(ifyourequirea non-expiring tokenuse--token-ttl0)[certificates]Generatedca certificateandkey.[certificates]Generatedapiserver certificateandkey.[certificates]apiserver serving certissignedforDNS names[node1 kubernetes kubernetes.defaultkubernetes.default.svc kubernetes.default.svc.cluster.local]andIPs[10.96.0.1192.168.61.11][certificates]Generatedapiserver-kubelet-client certificateandkey.[certificates]Generatedsa keyandpublickey.[certificates]Generatedfront-proxy-ca certificateandkey.[certificates]Generatedfront-proxy-client certificateandkey.[certificates]Validcertificatesandkeys now existin"/etc/kubernetes/pki"[kubeconfig]WroteKubeConfigfile to disk:"admin.conf"[kubeconfig]WroteKubeConfigfile to disk:"kubelet.conf"[kubeconfig]WroteKubeConfigfile to disk:"controller-manager.conf"[kubeconfig]WroteKubeConfigfile to disk:"scheduler.conf"[controlplane]WroteStaticPodmanifestforcomponent kube-apiserver to"/etc/kubernetes/manifests/kube-apiserver.yaml"[controlplane]WroteStaticPodmanifestforcomponent kube-controller-manager to"/etc/kubernetes/manifests/kube-controller-manager.yaml"[controlplane]WroteStaticPodmanifestforcomponent kube-scheduler to"/etc/kubernetes/manifests/kube-scheduler.yaml"[etcd]WroteStaticPodmanifestforalocaletcd instance to"/etc/kubernetes/manifests/etcd.yaml"[init]Waitingforthe kubelet to boot up the control planeasStaticPodsfromdirectory"/etc/kubernetes/manifests"[init]Thisoften takes around a minute;orlongerifthe control plane images have to be pulled.[apiclient]Allcontrol plane components are healthy after28.505733seconds[uploadconfig]Storingthe configuration usedinConfigMap"kubeadm-config"inthe"kube-system"Namespace[markmaster]Willmark node node1asmasterbyadding a labelanda taint[markmaster]Masternode1 taintedandlabelledwithkey/value:node-role.kubernetes.io/master=""[bootstraptoken]Usingtoken:9e68dd.7117f03c900f9234[bootstraptoken]ConfiguredRBAC rules to allowNodeBootstraptokens to postCSRsinorderfornodes togetlongterm certificate credentials[bootstraptoken]ConfiguredRBAC rules to allow the csrapprover controller automatically approveCSRsfromaNodeBootstrapToken[bootstraptoken]Creatingthe"cluster-info"ConfigMapinthe"kube-public"namespace[addons]Appliedessential addon:kube-dns[addons]Appliedessential addon:kube-proxyYourKubernetesmaster has initialized successfully!Tostartusingyour cluster,you need to run(asa regular user):mkdir-p $HOME/.kube
sudo cp-i/etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id-u):$(id-g)$HOME/.kube/configYoushould now deploy a pod network to the cluster.Run"kubectl apply -f [podnetwork].yaml"withone of the options listed at:http://kubernetes.io/docs/admin/addons/Youcan now join any number of machinesbyrunning the following on each nodeasroot:kubeadm join--token9e68dd.7117f03c900f9234192.168.61.11:6443--discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7
上面记录了完成的初始化输出的内容。
其中由以下关键内容：
kubeadm 1.8当前还处于beta状态，还不能用于生产环境。目前来看这东西安装的etcd和apiserver都是单节点，当然不能用于生产环境。
RBAC模式已经在Kubernetes 1.8中稳定可用。kubeadm 1.8也默认启用了RBAC
接下来是生成证书和相关的kubeconfig文件，这个目前我们在Kubernetes 1.6 高可用集群部署也是这么做的，目前没看出有什么新东西
生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
另外注意kubeadm还报了starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use –token-ttl 0)的警告
下面的命令是配置常规用户如何使用kubectl访问集群：
mkdir-p $HOME/.kube
sudo cp-i/etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id-u):$(id-g)$HOME/.kube/config
最后给出了将节点加入集群的命令kubeadm join –token 9e68dd.7117f03c900f9234 192.168.61.11:6443 –discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7
查看一下集群状态：
kubectlgetcs
NAME STATUS MESSAGE ERROR
schedulerHealthyok
controller-managerHealthyok
etcd-0Healthy{"health":"true"}
确认个组件都处于healthy状态。
集群初始化如果遇到问题，可以使用下面的命令进行清理：
kubeadm reset
ifconfig cni0 down
ip linkdeletecni0
ifconfig flannel.1down
ip linkdeleteflannel.1rm-rf/var/lib/cni/
4.安装Pod Network
接下来安装flannel network add-on：
mkdir-p~/k8s/wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlkubectl apply-f kube-flannel.yml
clusterrole"flannel"created
clusterrolebinding"flannel"created
serviceaccount"flannel"created
configmap"kube-flannel-cfg"created
daemonset"kube-flannel-ds"created
这里注意kube-flannel.yml这个文件中已经包含了ServiceAccount, ClusterRole和ClusterRoleBinding，原来是在一个单独的kube-flannel-rbac.yml文件中。kube-flannel.yml这个文件里的flannel的镜像是0.9.0，quay.io/coreos/flannel:v0.9.0-amd64
如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=
......apiVersion:extensions/v1beta1
kind:DaemonSetmetadata:name:kube-flannel-ds......containers:-name:kube-flannel
image:quay.io/coreos/flannel:v0.9.0-amd64
command:["/opt/bin/flanneld","--ip-masq","--kube-subnet-mgr","--iface=eth1"]......
使用kubectl get pod –all-namespaces -o wide确保所有的Pod都处于Running状态。
kubectlgetpod--all-namespaces-o wide
5.master node参与工作负载
使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，也就是说Master Node不参与工作负载。
这里搭建的是测试环境可以使用下面的命令使Master Node参与工作负载：
kubectl taint nodes node1 node-role.kubernetes.io/master-node"node1"untainted
6.测试DNS
kubectl run curl--image=radial/busyboxplus:curl-i--ttyIfyou don't see a command prompt, try pressing enter.
[ root@curl-2716574283-xr8zd:/ ]$
进入后执行nslookup kubernetes.default确认解析正常:
nslookup kubernetes.defaultServer:10.96.0.10Address1:10.96.0.10kube-dns.kube-system.svc.cluster.localName:kubernetes.defaultAddress1:10.96.0.1kubernetes.default.svc.cluster.local
7.向Kubernetes集群添加Node
下面我们将k8s-node2这个主机添加到Kubernetes集群中，在k8s-node2上执行：
kubeadm join--token9e68dd.7117f03c900f9234192.168.61.11:6443--discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7[kubeadm]WARNING:kubeadmisinbeta,pleasedonotuseitforproduction clusters.[preflight]Runningpre-flight checks[discovery]Tryingto connect to APIServer"192.168.61.11:6443"[discovery]Createdcluster-info discovery client,requesting infofrom"https://192.168.61.11:6443"[discovery]Requestinginfofrom"https://192.168.61.11:6443"again to validate TLS against the pinnedpublickey[discovery]Clusterinfo signatureandcontents are validandTLS certificate validates against pinned roots,willuseAPIServer"192.168.61.11:6443"[discovery]Successfullyestablished connectionwithAPIServer"192.168.61.11:6443"[bootstrap]Detectedserver version:v1.8.0[bootstrap]Theserver supports theCertificatesAPI(certificates.k8s.io/v1beta1)Nodejoin complete:*Certificatesigning request sent to masterandresponse
received.*Kubeletinformed ofnewsecure connection details.Run'kubectl get nodes'on the master to seethismachine join.
这次很是顺利，下面在master节点上执行命令查看集群中的节点：
kubectlgetnodes
NAME STATUS ROLES AGE VERSION
node1Readymaster25mv1.8.0node2Ready10mv1.8.0
如何从集群中移除Node
如果需要从集群中移除node2这个Node执行下面的命令：
在master节点上执行：
kubectl drain node2--delete-local-data--force--ignore-daemonsets
kubectldeletenode node2
在node2上执行：
kubeadm reset
ifconfig cni0 down
ip linkdeletecni0
ifconfig flannel.1down
ip linkdeleteflannel.1rm-rf/var/lib/cni/
8.dashboard插件部署
注意当前dashboard的版本已经是1.7.1了。 而1.7.x版本的dashboard对安全做了增强，默认需要以https的方式访问，增加了登录的页面，同时增加了一个gcr.io/google_containers/kubernetes-dashboard-init-amd64的init容器。
另外需要注意dashboard调整了部署文件的源码目录结构：
mkdir-p~/k8s/wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yamlkubectl create-f kubernetes-dashboard.yaml
kubernetes-dashboard.yaml文件中的ServiceAccount kubernetes-dashboard只有相对较小的权限，因此我们创建一个kubernetes-dashboard-admin的ServiceAccount并授予集群admin的权限，创建kubernetes-dashboard-admin.rbac.yaml：
---apiVersion:v1
kind:ServiceAccountmetadata:labels:k8s-app:kubernetes-dashboard
name:kubernetes-dashboard-adminnamespace:kube-system---apiVersion:rbac.authorization.k8s.io/v1beta1
kind:ClusterRoleBindingmetadata:name:kubernetes-dashboard-admin
labels:k8s-app:kubernetes-dashboard
roleRef:apiGroup:rbac.authorization.k8s.io
kind:ClusterRolename:cluster-admin
subjects:-kind:ServiceAccountname:kubernetes-dashboard-adminnamespace:kube-system
kubectl create-f kubernetes-dashboard-admin.rbac.yaml
serviceaccount"kubernetes-dashboard-admin"created
clusterrolebinding"kubernetes-dashboard-admin"created
查看kubernete-dashboard-admin的token:
kubectl-n kube-systemgetsecret|grep kubernetes-dashboard-admin
kubernetes-dashboard-admin-token-pfss5 kubernetes.io/service-account-token314skubectl describe-n kube-system secret/kubernetes-dashboard-admin-token-pfss5Name:kubernetes-dashboard-admin-token-pfss5Namespace:kube-systemLabels:Annotations:kubernetes.io/service-account.name=kubernetes-dashboard-admin
kubernetes.io/service-account.uid=1029250a-ad76-11e7-9a1d-08002778b8a1Type:kubernetes.io/service-account-tokenData====ca.crt:1025bytesnamespace:11bytes
token:eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1wZnNzNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjEwMjkyNTBhLWFkNzYtMTFlNy05YTFkLTA4MDAyNzc4YjhhMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.Bs6h65aFCFkEKBO_h4muoIK3XdTcfik-pNM351VogBJD_pk5grM1PEWdsCXpR45r8zUOTpGM-h8kDwgOXwy2i8a5RjbUTzD3OQbPJXqa1wBk0ABkmqTuw-3PWMRg_Du8zuFEPdKDFQyWxiYhUi_v638G-R5RdZD_xeJAXmKyPkB3VsqWVegoIVTaNboYkw6cgvMa-4b7IjoN9T1fFlWCTZI8BFXbM8ICOoYMsOIJr3tVFf7d6oVNGYqaCk42QL_2TfB6xMKLYER9XDh753-_FDVE5ENtY5YagD3T_s44o0Ewara4P9C3hYRKdJNLxv7qDbwPl3bVFH3HXbsSxxF3TQ
在dashboard的登录窗口使用上面的token登录。
9.heapster插件部署
下面安装Heapster为集群添加使用统计和监控功能，为Dashboard添加仪表盘。 使用InfluxDB做为Heapster的后端存储，开始部署：
mkdir-p~/k8s/heapster
cd~/k8s/heapster
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yamlwget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yamlwget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yamlwget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yamlkubectl create-f./
最后确认所有的pod都处于running状态，打开Dashboard,集群的使用统计会以仪表盘的形式显示出来。
本次安装涉及到的Docker镜像：
gcr.io/google_containers/kube-apiserver-amd64:v1.8.0gcr.io/google_containers/kube-controller-manager-amd64:v1.8.0gcr.io/google_containers/kube-scheduler-amd64:v1.8.0gcr.io/google_containers/kube-proxy-amd64:v1.8.0gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5quay.io/coreos/flannel:v0.9.0-amd64
gcr.io/google_containers/etcd-amd64:3.0.17gcr.io/google_containers/pause-amd64:3.0quay.io/coreos/flannel:v0.9.0-amd64
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.0gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3gcr.io/google_containers/heapster-grafana-amd64:v4.4.3gcr.io/google_containers/heapster-amd64:v1.4.0
网友评论