Kubernetes 上安装 EMQ X 系列文章之二 ：EMQ

EMQ X 支持通过 Kubernetes 服务自动集群，在本系列文章之一的【Ubuntu 上使用 kubeadm】 中介绍了如何使用 kubeadm 搭建了一个三台节点的 k8s 集群，接下来介绍在这个集群中使用 EMQ X 的自动集群功能搭建一个 EMQ X 集群。

本文需要对 k8s 集群的资源如 pods、services 等有基本的概念，可以阅读官方文档来了解。

实验环境

• 公有云环境：AWS EC2
• 操作系统：ubuntu 16.04
• kubeadm version：v1.12.1
• Docker version：18.6.1
• 集群节点

  hostname	节点角色	IP地址
  kube-node1	master	172.31.18.155
  kube-node2	worker	172.31.21.171
  kube-node3	worker	172.31.20.189

准备工作

查看 k8s 集群状态

$ kubectl get node -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
kube-node1   Ready    master   28h   v1.12.1   172.31.18.155   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1
kube-node2   Ready    <none>   28h   v1.12.1   172.31.21.171   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1
kube-node3   Ready    <none>   28h   v1.12.1   172.31.20.189   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1

$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE     IP          NODE         NOMINATED NODE
kube-system   coredns-576cbf47c7-b5xbb             1/1     Running   0          28h   10.244.0.5      kube-node1   <none>
kube-system   coredns-576cbf47c7-g5s9j             1/1     Running   0          28h   10.244.0.4      kube-node1   <none>
kube-system   etcd-kube-node1                      1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-apiserver-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-controller-manager-kube-node1   1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-flannel-ds-amd64-fscrm          1/1     Running   0          28h   172.31.20.189   kube-node3   <none>
kube-system   kube-flannel-ds-amd64-hhj8b          1/1     Running   0          28h   172.31.21.171   kube-node2   <none>
kube-system   kube-flannel-ds-amd64-l6ccn          1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-proxy-79thv                     1/1     Running   0          28h   172.31.20.189   kube-node3   <none>
kube-system   kube-proxy-ckg9t                     1/1     Running   0          28h   172.31.21.171   kube-node2   <none>
kube-system   kube-proxy-skq8m                     1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-scheduler-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>

查看 apiserver 配置

EMQ X 自动集群功能需要用到 k8s 的 apiserver，先看一下 apiserver 的具体配置

$ kubectl describe pods kube-apiserver-kube-node1 -n kube-system  
Name:               kube-apiserver-kube-node1
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               kube-node1/172.31.18.155
Start Time:         Tue, 23 Oct 2018 02:29:37 +0000
Labels:             component=kube-apiserver
                    tier=control-plane
Annotations:        kubernetes.io/config.hash: c5ac975e628056601100307026359ba8
                    kubernetes.io/config.mirror: c5ac975e628056601100307026359ba8
                    kubernetes.io/config.seen: 2018-10-17T02:00:07.757483468Z
                    kubernetes.io/config.source: file
                    scheduler.alpha.kubernetes.io/critical-pod: 
Status:             Running
IP:                 172.31.18.155
Containers:
  kube-apiserver:
    Container ID:  docker://d753a932b41ff8a99b6a11767d463f13af7e9de7526567df6eb5bd29a101f17b
    Image:         k8s.gcr.io/kube-apiserver:v1.12.1
    Image ID:      docker-pullable://k8s.gcr.io/kube-apiserver@sha256:52b9dae126b5a99675afb56416e9ae69239e012028668f7274e30ae16112bb1f
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --authorization-mode=Node,RBAC
      --advertise-address=172.31.18.155
      --allow-privileged=true
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Running
      Started:      Tue, 23 Oct 2018 02:29:39 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Oct 2018 02:00:09 +0000
      Finished:     Thu, 18 Oct 2018 12:43:31 +0000
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:        250m
    Liveness:     http-get https://172.31.18.155:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ca-certificates from etc-ca-certificates (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  usr-local-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  etc-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:            <none>

通过 --advertise-address=172.31.18.155 获得 apiserver 的 IP 地址（该地址在使用kubeadm init 命令创建集群的时候可以通过 --apiserver-advertise-address=172.31.18.155 参数来指定)。

访问 apiserver

执行kubectl proxy --help命令可以看到kubectl proxy可以在本机和 apiserver 之间创建一个代理服务器，它允许指定规则的 http 访问。

$ kubectl proxy --accept-hosts="^.*$" --address='172.31.18.155' -p=8080 &
$ curl http://172.31.18.155:8080
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1",
    "/apis/apiregistration.k8s.io/v1beta1",
    ......

创建资源

kubectl 可以通过 kubectl create -f xxx.yml 来创建各种资源，那么只需要编写一个或多个合法的 yml 文件就可以创建 EMQ X 集群了。本文将所有资源都写在了一个 yml 文件中，在实际生产中，可以在一个目录下创建多个 yml 文件，并使用kubectl create -f 目录名来部署。
为了方便理解和学习，在本次实验中，将需要创建的资源类型和命名提前确定下来

资源类型	命名
service	emqx
deployment	emqx
pod	emqx

需要注意的是，一般来说 service、deployment 和 pod 最好命名为不同的名字以便区分，不过 EMQ X 的自动集群功能需要将它们命名为统一的名字。

Deployment && Pod

查看 EMQ X 官方文档中基于 Kubernetes 自动集群的相关配置，需要将etc/emqx.conf的配置做如下修改。

cluster.discovery = k8s

##--------------------------------------------------------------------
## Cluster with k8s

cluster.k8s.apiserver = http://10.110.111.204:8080

cluster.k8s.service_name = emqx

## Address Type: ip | dns
cluster.k8s.address_type = ip

## The Erlang application name
cluster.k8s.app_name = emqx

## Kubernates Namespace
cluster.k8s.namespace = default

查看 Github 上关于 EMQ X 镜像的文档，可以使用编辑环境变量的方式来修改etc/emqx.conf，根据上文所需的配置，我们可以指定如下环境变量

- name: EMQX_CLUSTER__DISCOVERY
    value: k8s
- name: EMQX_NAME
    value: emqx
- name: EMQX_CLUSTER__K8S__APISERVER
    value: http://172.31.19.161:8080
- name: EMQX_CLUSTER__K8S__NAMESPACE
    value: default
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
    value: emqx
- name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
    value: ip
- name: EMQX_CLUSTER__K8S__APP_NAME
    value: emqx

创建 deployment 和 pod，因为 k8s 不能直接使用 Dockerfile 去编译镜像，只能从 docker 的镜像仓库中拉取，所以 pod 使用 docker hub 的镜像。指定部署两个 EMQ X 镜像的 pod，并指定 pod 的端口和环境变量。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: emqx
  labels:
        app: emqx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:latest
        ports:
        - name: emqx-dashboard
          containerPort: 18083
        env:
        - name: EMQX_CLUSTER__DISCOVERY
          value: k8s
        - name: EMQX_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: http://172.31.19.161:8080
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
          value: ip
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        tty: true

Services

创建 Service，使用 NodePort 的方式将 emqx-dashboard 的端口暴露出来，方便访问 EMQ X 的 dashboard。

apiVersion: v1
kind: Service
metadata:
  name: emqx
spec:
  ports:
  - port: 32333
    nodePort: 32333
    targetPort:  emqx-dashboard
    protocol: TCP
  selector:
    app: emqx
  type: NodePort

部署服务

查看一下 emqx.yml 文件

cat emqx.yml

apiVersion: v1
kind: Service
metadata:
  name: emqx
spec:
  ports:
  - port: 32333
    nodePort: 32333
    targetPort:  emqx-dashboard
    protocol: TCP
  selector:
    app: emqx
  type: NodePort

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: emqx
  labels:
        app: emqx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:latest
        ports:
        - name: emqx-dashboard
          containerPort: 18083
        env:
        - name: EMQX_CLUSTER__DISCOVERY
          value: k8s
        - name: EMQX_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: http://172.31.19.161:8080
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
          value: ip
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        tty: true

部署 EMQ X

$ kubectl create -f emqx.yml 
service/emqx created
deployment.extensions/emqx created

查看部署的状态，可以看到所有的资源都成功部署了

$ kubectl get all -o wide
NAME                                  READY   STATUS        RESTARTS   AGE     IP            NODE         NOMINATED NODE
pod/emqx-7685fc45cd-qmxlq             1/1     Running       0          3m22s   10.244.2.14   kube-node3   <none>
pod/emqx-7685fc45cd-zq2cj             1/1     Running       0          3m22s   10.244.1.18   kube-node2   <none>

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE     SELECTOR
service/emqx         NodePort    10.98.146.60   <none>        32333:32333/TCP   2m49s   app=emqx
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP           6d5h    <none>

NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES             SELECTOR
deployment.apps/emqx   2         2         2            2           2m49s   emqx         emqx/emqx:latest   app=emqx

NAME                              DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES             SELECTOR
replicaset.apps/emqx-7685fc45cd   2         2         2       2m49s   emqx         emqx/emqx:latest   app=emqx,pod-template-hash=7685fc45cd

使用命令行查看集群状态，可以看到 EMQ X 已自动集群。

$ kubectl exec -it emqx-7685fc45cd-qmxlq /opt/emqx/bin/emqx_ctl cluster status
Cluster status: [{running_nodes,['emqx@10.244.1.18','emqx@10.244.2.14']}]

即使删掉一个 pod，k8s 也会自动创建出一个新的 pod 来自动集群

$ kubectl delete pod/emqx-7685fc45cd-zq2cj
pod "emqx-7685fc45cd-zq2cj" deleted

$ kubectl get pods 
NAME                    READY   STATUS    RESTARTS   AGE
emqx-7685fc45cd-nt54v   1/1     Running   0          56s
emqx-7685fc45cd-qmxlq   1/1     Running   0          6m25

$ kubectl exec -it emqx-7685fc45cd-qmxlq /opt/emqx/bin/emqx_ctl cluster status
Cluster status: [{running_nodes,['emqx@10.244.1.19','emqx@10.244.2.14']},
                 {stopped_nodes,['emqx@10.244.1.18']}]

打开浏览器，输入http://任意节点的公网IP：32333可以成功访问 EMQ X 的 dashboard 页面。