锻骨境-第9层 k8s自建Ingress Controlle

Nginx Ingress

这里使用Nginx Ingress 提供对外的服务。

ingress 的配置是针对郁域名方式的

正常情况下 ，我们用的域名都是带TLS证书 ，防止篡改的。
这里自定义一个域名叫 k8s.local . 在自己的电脑上配置下window host. 192.168.10.133 *.k8s.local
使用cfssl给自己的集群加个域名证书

• 编辑k8s-local.json

{
  "CN": "k8s.local",
  "hosts": [
    "127.0.0.1",
      "*.k8s.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shenzhen",
      "L": "shenzhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

• 生成两个 证书，*.k8s.local这个域名的,分别在kube-system 和 kubernets-dashboard 命名空间下，注意和自己的 服务 在一个空间下。请注意本层dashboard 使用的是自己颁发的证书：
  必须创建secret 名为： kubernetes-dashboard-certs 。 如果没有，dashboard 会使用自己的证书。

[root@k8s-node1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes k8s-local.json | cfssljson -bare k8s-local
2019/11/03 15:03:39 [INFO] generate received request
2019/11/03 15:03:39 [INFO] received CSR
2019/11/03 15:03:39 [INFO] generating key: rsa-2048
2019/11/03 15:03:39 [INFO] encoded CSR
2019/11/03 15:03:39 [INFO] signed certificate with serial number 65727990797100765876932917522552114529933202146
2019/11/03 15:03:39 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# ls
ca-config.json  ca-csr.json  ca.pem  etcd-2.etcd  etcd.yaml  k8s-csr.json   k8s-local.json     k8s-local.pem   kubernetes-key.pem  nohup.out
ca.csr          ca-key.pem   etcd    etcdctl      install    k8s-local.csr  k8s-local-key.pem  kubernetes.csr  kubernetes.pem      ssl
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# kubectl -n kubernetes-dashboard  create secret tls  kubernetes-dashboard-certs  --key=k8s-local-key.pem --cert=k8s-local.pem

secret/default-tls-cert created
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# 
[root@k8s-node1 etcd]# kubectl -n kube-system  create secret tls default-tls-cert --key=k8s-local-key.pem --cert=k8s-local.pem
secret/default-tls-cert created
[root@k8s-node1 etcd]# 

开始搭建Nginx Ingress Controller

官方文档：https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

下载下来之后，对于他的文档，我 修改了部署到的命名空间为kube-system 并去除了nodeSelector 节点选择器。

• 我自己的ingress 部署文件

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: kube-system

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
          
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

如果不修改镜像，会pull 失败 ,用describe命令查看pod ，此时，要么更换国内被人的镜像，要么FQ。 这里我使用代理后下载官网的镜像。

                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                 Message
  ----     ------     ----                   ----                 -------
  Normal   Scheduled  3m34s                  default-scheduler    Successfully assigned kube-system/nginx-ingress-controller-5c64dcfbf9-4gdj7 to k8s-master
  Warning  Failed     3m14s                  kubelet, k8s-master  Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     3m14s                  kubelet, k8s-master  Error: ErrImagePull
  Normal   BackOff    3m13s                  kubelet, k8s-master  Back-off pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"
  Warning  Failed     3m13s                  kubelet, k8s-master  Error: ImagePullBackOff
  Normal   Pulling    2m58s (x2 over 3m29s)  kubelet, k8s-master  pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"
[root@k8s-master k8s]# 

• pull 镜像，其实k8s 也会自己pull （可选，请自己科学上网）

[root@k8s-master etcd]# docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1 
0.26.1: Pulling from kubernetes-ingress-controller/nginx-ingress-controller
c8775c51b291: Already exists 
4fc647720de5: Downloading [============================>                      ]  9.911MB/17.21MB
4bee7d3b55eb: Downloading [==================================>                ]  538.5kB/771.2kB
85ee1a272ac8: Downloading [=>                                                 ]  3.242MB/121.9MB
6a37290ece43: Waiting 
cd42756652d1: Waiting 
ba0a46163f53: Waiting 
b15c3bdebd38: Waiting 
69a3d60cbd64: Waiting 
7b1f35e5645a: Waiting 
8c7b9cd15ac7: Waiting 
9021fc6169bc: Waiting 

• 镜像好了之后，部署service，nginx-service.yaml 如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-ingress-controller
    name: kube-system
  name: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - name: cce-service-0
    nodePort: 31304
    port: 80
    protocol: TCP
    targetPort: 80
  - name: cce-service-1
    nodePort: 31381
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

• 查看结果：

[root@k8s-master k8s]# kubectl  apply -f nginx-ingress-svc.yaml   -n  kube-system 
service/nginx-lb-svc created
[root@k8s-master k8s]# 
[root@k8s-master k8s]# 
[root@k8s-master k8s]# 
[root@k8s-master k8s]# 
[root@k8s-master k8s]# 
[root@k8s-master k8s]# kubectl  get svc  -n kube-system 
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
kube-dns       ClusterIP      10.96.0.10      <none>           53/UDP,53/TCP                23h
nginx-lb-svc   LoadBalancer   10.98.182.170   192.168.10.134   80:31304/TCP,443:31381/TCP   19s

配置dashboard 域名访问：

配置ClusterIP 类型的svc

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-internal
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  type: ClusterIP

• 配置ingress 路由规则：

注意： nginx.ingress.kubernetes.io/secure-backends: "true" 在0.18 之后失效。

• 重点是这几个注解：

    nginx.ingress.kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

• Ingress 配置规则：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    
  generation: 1
  name: dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  tls:
  - hosts:
    - admin.k8s.local
    secretName: default-tls-cert
  rules:
  - host: admin.k8s.local
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard-internal
          servicePort: 443
        path: /

• 此时可以用Chrome 查看页面：
  请现在hosts 里配置自己的域名解析

image.png

至此，完成了用Ingress Controller 的路由规则访问dashboard,且使用https 的方式。
至于自定义的域名证书怎么被浏览器认为是安全的，见：
锻骨境-号外1 k8s 里定义证书，本地计算机使用

• 其他相关知识
  • k8s https 管理 (https://my.oschina.net/u/2306127/blog/1929905)
  • 客户端管理集群管理工具： https://kubernetic.com/