Kubernetes 上安装 EMQ X 系列文章之一：Ubu

作者: EMQ | 来源:发表于2018-11-05 10:50 被阅读5次

kubeadm 介绍

kubeadm 是一个工具包，可帮助您以简单，合理安全和可扩展的方式引导最佳实践 Kubernetes 群集。它还支持为您管理 Bootstrap Tokens 并升级/降级集群。

kubeadm 的目标是建立一个通过 Kubernetes 一致性测试 Kubernetes Conformance tests 的最小可行集群，但不会安装其他功能插件。

它在设计上并未为您安装网络解决方案，需要用户自行安装第三方符合CNI的网络解决方案（如 flanal，calico，canal 等）。

kubeadm 可以在多种设备上运行，可以是 Linux 笔记本电脑，虚拟机，物理/云服务器或 Raspberry Pi。这使得kubeadm 非常适合与不同种类的配置系统（例如 Terraform，Ansible 等）集成。

kubeadm 是一种简单的方式让新用户开始尝试 Kubernetes，也可能是第一次让现有用户轻松测试他们的应用程序并缝合到一起的方式，也可以作为其他生态系统中的构建块，或者具有更大范围的安装工具。

可以在支持安装 deb 或 rpm 软件包的操作系统上非常轻松地安装 kubeadm 。SIG 集群生命周期 SIG Cluster Lifecycle kubeadm 的 SIG 相关维护者提供了预编译的这些软件包，也可以在其他操作系统上使用。

准备工作

本文所用的环境：

公有云环境：AWS EC2
操作系统：ubuntu 16.04 minimal 安装
网络规划：
- kube-node1：172.31.18.155
- kube-node2：172.31.21.171
- kube-node3：172.31.20.189

安装 docker
设置国内下载源

apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF  
apt-get update

安装 kubeadm、kubectl、kubelet

apt-get install -y kubelet kubeadm=1.12.0-00 kubectl

本次实验指定 kubeadm 的版本为 1.12.0，可以使用 apt-cache madison kubeadm 命令查看可用的 kubeadm 版本

配置 master 节点

因为国内没办法访问 Google 的镜像源，变通的方法是从其他镜像源下载后，修改 tag。执行下面这个 Shell 脚本即可。

#!/bin/bash

images=(k8s.gcr.io/kube-apiserver:v1.12.0
        k8s.gcr.io/kube-controller-manager:v1.12.0
        k8s.gcr.io/kube-scheduler:v1.12.0
        k8s.gcr.io/kube-proxy:v1.12.0
        k8s.gcr.io/pause:3.1
        k8s.gcr.io/etcd:3.2.24
        k8s.gcr.io/coredns:1.2.2)

for var in ${images[@]};do
        image=${var/k8s.gcr.io\//anjia0532\/google-containers.}
        docker pull ${image}
        docker tag ${image} ${var}
done

docker pull coredns/coredns:1.2.2
docker tag coredns/coredns:1.2.2 k8s.gcr.io/coredns:1.2.2

PS: 如果是其他的 kubeadm 的版本，可以执行 kubeadm config image list 命令查看所需要的镜像，然后下载相应的镜像即可。（该命令可能需要翻墙）

接下来执行 Master 节点的初始化，首选需要制定 kuberneter 的版本，不然 kubeadm 会去墙外的网站查询从而导致超时，接下来需要指定 apiserver 的监听地址，另外我们打算使用 flannel 网络插件，根据官方文档，需要指定--pod-network-cidr=10.244.0.0/16。（如果安装 calico 网络，则需要指定 --pod-network-cidr=192.168.0.0/16）

$ sudo kubeadm init --kubernetes-version=v1.12.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.31.18.155

[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
I0724 08:36:35.636931    3409 kernel_validator.go:81] Validating kernel version
I0724 08:36:35.637052    3409 kernel_validator.go:96] Validating kernel config
    [WARNING Hostname]: hostname "devops-101" could not be reached
    [WARNING Hostname]: hostname "devops-101" lookup devops-101 on 172.20.10.1:53: no such host
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [devops-101 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.101]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [devops-101 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [devops-101 localhost] and IPs [192.168.0.101 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 46.002877 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node devops-101 as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node devops-101 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "devops-101" as an annotation
[bootstraptoken] using token: wkj0bo.pzibll6rd9gyi5z8
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.31.18.155:6443 --token rje1gq.9zohjsh8mjxp90oj --discovery-token-ca-cert-hash sha256:54ad5605a76cf7818443d5db849e0da9a076bed7e1a367d54b4019166e930285

看到以上信息表示 Master 节点已经初始化成功了，按照提示配置 kubectl

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看一下集群状态：

$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

确认个组件都处于 healthy 状态。

集群初始化如果遇到问题，可以使用下面的命令进行清理：

$ kubeadm reset
$ ifconfig cni0 down
$ ip link delete cni0
$ ifconfig flannel.1 down
$ ip link delete flannel.1
$ rm -rf /var/lib/cni/

查看 k8s 状态

$ kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
kube-node1   NotReady   master   72s   v1.12.1 

$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE   IP          NODE         NOMINATED NODE
kube-system   coredns-576cbf47c7-b5xbb             1/1     Pending   0          28h   10.244.0.5      kube-node1   <none>
kube-system   coredns-576cbf47c7-g5s9j             1/1     Pending   0          28h   10.244.0.4      kube-node1   <none>
kube-system   etcd-kube-node1                      1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-apiserver-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-controller-manager-kube-node1   1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-proxy-skq8m                     1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-scheduler-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>

可以看到节点还没有 Ready，dns 的两个 pod 也没不正常，还需要安装网络配置。

$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ kubectl apply -f  kube-flannel.yml

如果 Node 有多个网卡的话，参考 flannel issues 39701，目前需要在 kube-flannel.yml 中使用 –iface 参数指定集群主机内网网卡的名称，否则可能会出现 dns 无法解析。需要将 kube-flannel.yml 下载到本地，flanneld 启动参数加上 –iface=\<iface-name>

......
containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=eth0
......

执行成功后，Master 并不能马上变成 Ready 状态，稍等几分钟，就可以看到所有状态都正常了。

$ kubectl get pods --all-namespaces -o wide
kube-system   coredns-576cbf47c7-b5xbb             1/1     Running   0          28h   10.244.0.5      kube-node1   <none>
kube-system   coredns-576cbf47c7-g5s9j             1/1     Running   0          28h   10.244.0.4      kube-node1   <none>
kube-system   etcd-kube-node1                      1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-apiserver-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-controller-manager-kube-node1   1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-flannel-ds-amd64-l6ccn          1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-proxy-skq8m                     1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-scheduler-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
$ kubectl get node -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
kube-node1   Ready    master   28h   v1.12.1   172.31.18.155   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1

至此，k8s 的 master 节点初始化完毕

配置 worker 节点

worker 节点参照本文前期的准备工作安装好 kubectl、kubeadm、kubelet, 并下载镜像（ worker 节点只需要下载 k8s.gcr.io/pause:3.1和 k8s.gcr.io/kube-proxy:v1.12.0 即可）。

#!/bin/bash

images=(k8s.gcr.io/kube-proxy:v1.12.0
        k8s.gcr.io/pause:3.1)

for var in ${images[@]};do
        image=${var/k8s.gcr.io\//anjia0532\/google-containers.}
        docker pull ${image}
        docker tag ${image} ${var}
done

根据 Master 节点的提示加入集群。

$sudo kubeadm join 172.31.18.155:6443 --token rje1gq.9zohjsh8mjxp90oj --discovery-token-ca-cert-hash 

sha256:54ad5605a76cf7818443d5db849e0da9a076bed7e1a367d54b4019166e930285
[preflight] running pre-flight checks
    [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

[discovery] Trying to connect to API Server "172.31.18.155:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.31.18.155:6443"
[discovery] Requesting info from "https://172.31.18.155:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.31.18.155:6443"
[discovery] Successfully established connection with API Server "172.31.18.155:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-worker" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

节点的启动也需要一点时间，稍后再到 Master 上查看状态

$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE     IP          NODE         NOMINATED NODE
kube-system   coredns-576cbf47c7-b5xbb             1/1     Running   0          28h   10.244.0.5      kube-node1   <none>
kube-system   coredns-576cbf47c7-g5s9j             1/1     Running   0          28h   10.244.0.4      kube-node1   <none>
kube-system   etcd-kube-node1                      1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-apiserver-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-controller-manager-kube-node1   1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-flannel-ds-amd64-fscrm          1/1     Running   0          28h   172.31.20.189   kube-node3   <none>
kube-system   kube-flannel-ds-amd64-hhj8b          1/1     Running   0          28h   172.31.21.171   kube-node2   <none>
kube-system   kube-flannel-ds-amd64-l6ccn          1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-proxy-79thv                     1/1     Running   0          28h   172.31.20.189   kube-node3   <none>
kube-system   kube-proxy-ckg9t                     1/1     Running   0          28h   172.31.21.171   kube-node2   <none>
kube-system   kube-proxy-skq8m                     1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-scheduler-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>

$ kubectl get node -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
kube-node1   Ready    master   28h   v1.12.1   172.31.18.155   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1
kube-node2   Ready    <none>   28h   v1.12.1   172.31.21.171   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1
kube-node3   Ready    <none>   28h   v1.12.1   172.31.20.189   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1

至此，k8s 集群创建完毕。

常见问题

在公有云上部署 flannel 网络可能会出现无法跨主机 ping flannel 网卡的情况，以 AWS 为例，需要在安全组里面开放8472的 UDP 端口，因为 flannel 使用 vxlan 技术为各节点创建网络，vxlan 使用8472端口作为 UDP 封装报文的端口，所以需要开放8472 UDP 端口。

安装 kubeadm 或 kubelet 的时候可能会遇到 kubelet : Depends: kubernetes-cni (= 0.6.0) but 0.6.0-02 is to be installed 的错误

$ sudo apt install kubelet 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
kubelet : Depends: kubernetes-cni (= 0.6.0) but 0.6.0-02 is to be installed
E: Unable to correct problems, you have held broken packages.

可以配置 /etc/apt/preferences.d/k8s.pref 作为临时的解决办法

# cat /etc/apt/preferences.d/k8s.pref
Package: kubernetes-cni
Pin: version 0.6.0-02*
Pin-Priority: 900

k8s 的网络配置除了 flannel 之外还有 Calico, Canal, Flannel, Kube-router, Romana, Weave Net 等，详细的网络列表可参考插件页面。

Kubernetes 上安装 EMQ X 系列文章之一：Ubu

kubeadm 介绍

准备工作

配置 master 节点

配置 worker 节点

常见问题

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

MQTT

Kubernetes 上安装 EMQ X 系列文章之一 ：Ubu

kubeadm 介绍

准备工作

配置 master 节点

配置 worker 节点

常见问题

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

MQTT

Kubernetes 上安装 EMQ X 系列文章之一：Ubu