K8s学习(二):集群部署

1.1 服务版本信息

服务名	版本信息
kubelet	1.24.2
kubeadm	1.24.2
kubectl	1.24.2

pod错误排查可以看下https://zhuanlan.zhihu.com/p/34332367这个博主的内容

1.2 为什么弃用docker？

Docker 是 Kubernetes 使用的第一个容器运行时。 这也是许多 Kubernetes 用户和爱好者如此熟悉Docker的原因之一。 对Docker 的支持被硬编码到 Kubernetes 中——一个被项目称为 dockershim的组件。

随着容器化成为行业标准，Kubernetes 项目增加了对其他运行时的支持。 最终实现了容器运行时接口（CRI），让系统组件（如 kubelet）以标准化的方式与容器运行时通信。 因此，dockershim 成为了 Kubernetes 项目中的一个异常现象。

对 Docker 和 dockershim 的依赖已经渗透到 CNCF 生态系统中的各种工具和项目中，这导致了代码脆弱。

通过删除 dockershim CRI，我们拥抱了 CNCF 的第一个价值： “快比慢好”。

2.添加k8s软件源-阿里云(all节点)

2.1 ubuntu

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-get add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

命令说明：
1.通过下载工具下载位于https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg的deb软件包密钥，然后通过"apt-key"命令添加密钥
2.通过cat把源deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main写入到"/etc/apt/sources.list.d/kubernetes.list"

2.2 centos

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3. 安装 kubeadm、kubelet、kubectl(all节点)

查看可选择版本

$ apt-get update
$ apt-cache madison kubeadm kubelet kubectl
#centos
$ yum list kubelet --showduplicates

安装指定版本

#ubuntu
$ apt-get install -y kubelet=1.24.2-00  kubeadm=1.24.2-00 kubectl=1.24.2-00
#centos
$ yum install -y kubelet-1.24.2 kubeadm-1.24.2  kubectl-1.24.2 

$ kubelet --version
Kubernetes v1.24.2

4. 启动kubelet服务(all节点)

# 设置开机启动
$ systemctl enable kubelet.service --now
# 查看是否启动
$ systemctl status kubelet.service

如果启动失败，可以执行命令journalctl -u kubelet -n 100看下是否有错误，也可以看下最近几条的日志tail -n 10 /var/log/messages|grep kube

启动错误：Error: failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubele>
解决：配置文件路径不存在，后面kubeadm init会自动生成

5. 初始化集群（master节点）

5.1 配置文件

a. 下载默认配置文件

# 创建目录 /usr/local/kubernetes
$ mkdir -p  /usr/local/kubernetes
# 下载默认配置
$ kubeadm config print init-defaults --component-configs KubeletConfiguration > /usr/local/kubernetes/kubeadm.yaml

b. 修改配置文件

# 修改1: advertiseAddress
localAPIEndpoint:
  advertiseAddress: 192.168.77.133  # 指定master节点内网IP
...
# 修改2：修改master节点名称
nodeRegistration:
  ...
  name: master # 修改master节点名称


# 修改3：此处新增一个 kubeproxy.config节点，并把kube-proxy模式为ipvs，默认为iptables
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
...
# 修改4: 设置imageRepository为阿里云的registry，避免因gcr被墙，无法直接拉取镜像
imageRepository: registry.aliyuncs.com/google_containers 
...

# 修改5: 指定k8s版本号，默认这里忽略了小版本号
kubernetesVersion: 1.24.2 
...

# 修改6: 确认设置kubelet的cgroupDriver为systemd
---
apiVersion: kubelet.config.k8s.io/v1beta1
...
cgroupDriver: systemd

# 修改7: 指定 pod 子网
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16  # 指定 pod 子网

5.2 拉取容器镜像

# 根据配置文件拉取镜像
$ kubeadm config images pull --config /usr/local/kubernetes/kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6

5.3 启动集群

# 根据配置文件启动镜像
$ kubeadm init --config /usr/local/kubernetes/kubeadm.yaml
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.77.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.77.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.77.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 12.012612 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.77.128:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:ab661eca8399a920e0811ff88ba4238919a588c9bd5b6d4998c35d69c5efbb12 

问题1：如果一直报：Error getting node" err="node \"master\" not found，可能是因为之前执行了init失败导致了某些信息被污染了，解决：尝试更新了证书，kubeadm certs renew all，然后查看systemctl status kubelet服务状态正常，重新执行init，如果报部分文件存在，先执行kubeadm reset再执行init

问题2：报错[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist，
解决：执行modprobe br_netfilter

问题3：报错：Failed to create sandbox for pod,此种情况如果在kubelet日志里面发现不了其他问题，可以看下containerd日志。journalctl -u containerd -f|grep error监听日志发现runc did not terminate successfully: exit status 127: runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond\\n\" runtime=io.containerd.runc.v2\ntime=\"2024-09-12T08:23:47+08:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/k8s.io/9aeffbdf60c32558fb60e1f56224f5db2e6464e970ae11da3cccd06d6c494cc7/init.pid: no such file or directory，此种情况大概率是libseccomp版本引起的，更新一下yum update libseccomp，重置后重新初始化

5.4 查看状态

$ kubectl get node
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   11m   v1.24.2

问题：执行命令报错：The connection to the server localhost:8080 was refused - did you specify the right host or port?
原因是kubectl命令需要使用kubernetes-admin来运行;
解决方法如下:将主节点中的/etc/kubernetes/admin.conf文件拷贝到从节点相同目录下，然后配置环境变量

$ echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bashrc 
$ source ~/.bashrc

6. 添加节点

6.1 加入集群(在节点执行)

# 在 `node1`和`node2`执行以下命令，加入集群
$ kubeadm join 192.168.77.133:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:b3edc1a3f9d8aba888a2b33a50b6b1e0293d21d5fb1bc76e586c3fc600f2b74112 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

此命令是在启动master的时候给出的。具体请看5.3 启动集群，如果找不到了或者卡住了（过期了）可以执行kubeadm token create --print-join-command

如果报错[WARNING FileExisting-tc]: tc not found in system path
执行yum install iproute-tc -y安装一下

查看是否成功

root@k8s-master# kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   2d1h    v1.24.2
node1    Ready    <none>          2m54s   v1.24.2
node2    Ready    <none>          83s     v1.24.2

NotReady是因为没有安装网络组件。

7.包管理器helm

7.1 什么是helm

每个成功的软件平台都有一个优秀的打包系统，比如Debian、Ubuntu的apt，Red Hat、CentOS的yum。Helm则是Kubernetes上的包管理器。

Helm到底解决了什么问题？为什么Kubernetes需要Helm？

答案是：Kubernetes能够很好地组织和编排容器，但它缺少一个更高层次的应用打包工具，而Helm就是来干这件事的。

7.2 安装

# 下载包
[root@master tmp]$ wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
# 解压
[root@master tmp]$ tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
# 移动二进制未加
[root@master tmp]$ mv linux-amd64/helm  /usr/local/bin/
# 验证
[root@master tmp]# helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

8. 使用helm部署Calico

选择calico作为k8s的Pod网络组件，下面使用helm在k8s集群中安装calico。

8.1 下载helm chart

[root@master helm] $ wget https://github.com/projectcalico/calico/releases/download/v3.23.1/tigera-operator-v3.23.1.tgz

8.2 安装

# 指定安装在命名空间为kube-system下
[root@master helm] $ helm install calico tigera-operator-v3.23.1.tgz -n kube-system
# 等待runing
$ kubectl get pod -n kube-system -w | grep tigera-operator
tigera-operator-5fb55776df-hnkbw   1/1     Running   1 (18h ago)     19h
# 所有都变成runing
[root@master helm]$ kubectl get pods -n calico-system -w
NAME                                       READY   STATUS    RESTARTS        AGE
calico-kube-controllers-68884f975d-b6vqq   1/1     Running   5 (4m37s ago)   18h
calico-node-4gknz                          1/1     Running   3 (4m54s ago)   19h
calico-node-8fx7n                          1/1     Running   1 (42m ago)     19h
calico-node-qzbh6                          1/1     Running   1 (18h ago)     19h
calico-typha-5d75c97db9-pwgnm              1/1     Running   3 (4m54s ago)   19h
calico-typha-5d75c97db9-vfpfq              1/1     Running   1 (18h ago)     19h

如果一直提示No resources found in calico-system namespace.，可能是节点有问题，此时执行下kubectl get pod -n calico-system -o wide可以看到一直在创建中

这个时候可以在主节点执行kubectl describe pod <podname> -n kube-system（推荐）或者进入到node1和node2查看具体是什么问题，执行journalctl -u kubelet -f

问题：Back-off pulling image "docker.io/calico/cni:v3.23.1"一直报某些镜像拉去不下来
解决：可以手动拉取，ctr images pull docker.io/calico/cni:v3.23.1

8.3 查看当时节点状态(全部变成Ready)

[root@master helm]$ kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   20m   v1.24.2
node1    Ready    <none>          16m   v1.24.2
node2    Ready    <none>          15m   v1.24.2

9.验证DNS是否可用

# 运行并进入验证容器
$ kubectl run curl --image=radial/busyboxplus:curl -it

# 验证
[ root@curl:/ ]$ nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

如果报错：nslookup: can't resolve 'kubernetes.default',
请参考：https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/dns-debugging-resolution/，我是停用了systemd-resolved，重启就好了

10.使用helm部署ingress-nginx

为了便于将集群中的服务暴露到集群外部，需要使用Ingress。接下来使用Helm将ingress-nginx部署到Kubernetes上。Nginx Ingress Controller被部署在Kubernetes的边缘节点上。

使用kubernetes-dashboard作为面板，默认安装完成的dashboard的访问方式是ClusterIP,进而想访问dashboard需更改为nodeport或者loadbalancer或者配置为ingress的方式才能访问dashbaord。本文将以ingress-nginx发布dashboard在外部访问

10.1 设置边缘节点

这里将node1(192.168.77.129)作为边缘节点，打上Label：

[root@master kubernetes]$ kubectl label node node1 node-role.kubernetes.io/edge=
node/node1 labeled
[root@master kubernetes]$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   2d21h   v1.24.2
node1    Ready    <none>          20h     v1.24.2
node2    Ready    edge            20h     v1.24.2

10.2 下载ingress-nginx的helm chart

$ wget https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.1.2/ingress-nginx-4.1.2.tgz

10.3 编写install-ingress.yaml

vim install-ingress.yaml

controller:
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"
  admissionWebhooks:
    enabled: false
  replicaCount: 1
  image:
    registry: docker.io
    image: unreachableg/k8s.gcr.io_ingress-nginx_controller
    tag: "v1.2.0"
    digest: sha256:314435f9465a7b2973e3aa4f2edad7465cc7bcdc8304be5d146d70e4da136e51
  hostNetwork: true
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule

nginx ingress controller的副本数replicaCount为1，将被调度到node1这个边缘节点上。这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。

10.4 安装

[root@master kubernetes]$ helm install ingress-nginx ingress-nginx-4.1.2.tgz --create-namespace -n ingress-nginx -f install-ingress.yaml
NAME: ingress-nginx
LAST DEPLOYED: Thu May  9 10:49:08 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
...
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

测试访问http://192.168.77.133返回默认的nginx 404页，则部署完成。

11. 使用helm部署DashBoard

为了提供更丰富的用户体验，Kubernetes还开发了一个基于Web的Dashboard，用户可以用Kubernetes Dashboard部署容器化的应用、监控应用的状态、执行故障排查任务以及管理Kubernetes的各种资源。

在Kubernetes Dashboard中可以查看集群中应用的运行状态，也能够创建和修改各种Kubernetes资源，比如Deployment、Job、DaemonSet等。用户可以Scale Up/Down Deployment、执行Rolling Update、重启某个Pod或者通过向导部署新的应用。Dashboard能显示集群中各种资源的状态以及日志信息。可以说，Kubernetes Dashboard提供了kubectl的绝大部分功能。

本文是通过ingress-nginx访问dashboard

11.1 部署metrics-server

Metrics-Server是集群核心监控数据的聚合器。通俗地说，它存储了集群中各节点的监控数据，并且提供了API以供分析和使用。

a. 下载配置文件

# 下载配置文件
[root@master ~]$ wget https://github.com/kubernetes-sigs/metrics-server/releases/download/metrics-server-helm-chart-3.8.2/components.yaml

b. 修改配置

vim components.yaml

....
---
apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        #修改1：添加容器启动参数,方便跳过安全tls,生产不推荐使用
        - --kubelet-insecure-tls 
        #修改2：更换镜像地址
        image: docker.io/unreachableg/k8s.gcr.io_metrics-server_metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent
...

c. 部署

# 部署资源
[root@master kubernetes]$ kubectl apply -f components.yaml
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
# 查看状态:ContainerCreating等待变成Running
[root@master kubernetes]$ kubectl get pod -n kube-system | grep metrics
metrics-server-77cffb4988-25wsk    1/1     Running   0              2m4s

11.2 添加对应的chart repo

# 添加chart
[root@master kubernetes]$ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
"kubernetes-dashboard" has been added to your repositories

# 更新包
[root@master kubernetes]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kubernetes-dashboard" chart repository
Update Complete. ⎈Happy Helming!⎈

11.3 制作ssl证书

本文因为是本地安装，如果是生产环境请使用真实的证书

# 生成证书请求的key
openssl genrsa -out dashboard.key 2048
# 生成证书请求
openssl req -new -key dashboard.key -out dashboard.csr -subj /C=CN/ST=JiangSu/L=NanJing/O=Shanhy/OU=Shanhy/CN=*.yzj-k8s.com
# 生成自签证书（证书文件 dashboard.crt 和私钥 dashboad.key）
openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt -days 3650
# 查看证书信息
openssl x509 -in dashboard.crt -text -noout

11.4 配置https证书为secret

# 创建secret到kube-system命名空间下
# 之后我们的dashboard也会创建在这个命名空间下，需要依赖这个，所以提前创建
kubectl create secret tls dashboard-tls --key dashboard.key --cert dashboard.crt -n kube-system
kubectl create secret tls dashboard-tls  -n kube-system --from-file=tls.crt=dashboard.crt --from-file=tls.key=dashboard.key

# 查看secret
kubectl get secret -n kube-system
NAME                              TYPE                DATA   AGE
dashboard-tls                     kubernetes.io/tls   2      30s

说明：
如果你使用--key --cert方式则创建的secret中data的默认2个文件名就是tls.key和tls.crt，你可以使用命令kubectl describe secret -n kubernetes-dashboard kubernetes-dashboard-certs查看。
如果你使用第二条命令的--from-file的方式，则你需要手工指定文件名称tls.crt和tls.key（看示例的写法），如果你把上面的命令直接写成--from-file=dashboard.crt，那么挂载后的文件就是dashboard.crt，这样你需要把第二步第2点中参数的tls.crt修改为dashboard.crt。
你还可以直接使用--from-file=mycert/这样直接指定一个目录，那么会把改目录下的所有文件都挂载到容器的/certs中，文件名保持不变。

11.4 定制char配置

image:
  repository: kubernetesui/dashboard
  tag: v2.5.1
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  hosts:
  - yzj-k8s.com # 上面定义的证书域名
  tls:
    - secretName: dashboard-tls # 上面添加的证书名称
      hosts:
      - yzj-k8s.com # 上面定义的证书域名
metricsScraper:
  enabled: true

11.5 安装部署

$ helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard -n kube-system -f dashboard.yaml

这里遇到了问题，Error: INSTALLATION FAILED: template: kubernetes-dashboard/templates/secrets/csrf.yaml:22:6: executing "kubernetes-dashboard/templates/secrets/csrf.yaml" at <include "kubernetes-dashboard.app.csrf.secret.value" .>: error calling include: template: kubernetes-dashboard/templates/_helpers.tpl:83:63: executing "kubernetes-dashboard.app.csrf.secret.value" at <$secret.data>: wrong type for value; expected map[string]interface {}; got interface {}
此问题一直没有找到解决办法！！！！

如果上面安装成功，忽略下面的11.6到11.8

11.6 单独安装kubernetes-dashboard

在github上选择合适的版本，本文档是k8s是1.24版本，所以安装2.6.1

• 下载配置文件

wget -O dashboard.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml
# 如果上面的下载不下来，选择直接打开网址，复制
vim dashboard.yaml 

• 修改配置文件

  
  

其中auto-generate-certificates不能注释，因为我看到过有帖子说要注释掉（这个参数不仅仅是自动证书的开关，还是总的HTTPS的开关，当我们手工配置了证书后，容器不会自动生成）。
另外两个tls参数指定的是被挂载到容器中的证书的名字，下面我们使用 tls secret 处理的证书，通过配置mountPath: /certs可以得知被挂载到容器的/certs目录中，其名字为tls.crt和tls.key（为什么叫这2个名字或者是否可以配置其他名字，请继续往下看）。

11.7 安装dashboard

$ kubectl apply -f dashboard.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

#查看安装情况
$ kubectl get service,pod,secret -n kubernetes-dashboard
NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/dashboard-metrics-scraper   ClusterIP   10.109.30.117   <none>        8000/TCP   142m
service/kubernetes-dashboard        ClusterIP   10.103.156.58   <none>        443/TCP    142m

NAME                                            READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-8c47d4b5d-g9gz9   1/1     Running   0          139m
pod/kubernetes-dashboard-f47c7f849-hbvtr        1/1     Running   0          139m

NAME                                     TYPE                DATA   AGE
secret/dashboard-tls                     kubernetes.io/tls   2      134m
secret/kubernetes-dashboard-certs        Opaque              0      142m
secret/kubernetes-dashboard-csrf         Opaque              1      142m
secret/kubernetes-dashboard-key-holder   Opaque              2      142m

问题：pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/kubernetesui/metrics-scraper:v1.0.8": failed to resolve reference "docker.io/kubernetesui/metrics-scraper:v1.0.8": failed to do request: Head "https://registry-1.docker.io/v2/kubernetesui/metrics-scraper/manifests/v1.0.8": dial tcp 202.160.128.203:443: connect: connection refused，
解决：镜像拉不下来，解决办法如下

#通过其他源拉取
$ ctr image pull docker.m.daocloud.io/kubernetesui/metrics-scraper:v1.0.8
$ ctr image list
docker.m.daocloud.io/kubernetesui/metrics-scraper:v1.0.8                                                                                  application/vnd.docker.distribution.manifest.list.v2+json sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c 18.8 MiB  linux/amd64,linux/arm,linux/arm64,linux/ppc64le,linux/s390x -
#导出镜像
$ tr image export metrics.tar docker.io/kubernetesui/metrics-scraper:v1.0.8
#导入到`crictl`,导入命令， 所有镜像都位于 http://k8s.io 命名空间下， 并且需要添加平台 --platform
$ ctr -n k8s.io images import  metrics.tar --platform linux/amd64
#查看镜像列表
$ crictl image
docker.io/kubernetesui/metrics-scraper                       v1.0.8              115053965e86b       19.7MB

11.8 配置ingress方式访问

Nginx Ingress Controller默认使用HTTP协议转发请求到后端业务容器。当您的业务容器为HTTPS协议时，可以通过使用注解nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"来使得Nginx Ingress Controller使用HTTP协议转发请求到后端业务容器。

vim ingress-dashboard.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: "nginx" # 控制器的类型为nginx
  tls:
  - hosts:
    - yzj-k8s.com   #主机名
    secretName: dashboard-tls  #这里引用创建的secrets
  rules:
  - host: yzj-k8s.com
    http:
      paths:
      - path: /
        pathType: Prefix   #起始与根都进行代理。
        backend:
          service:
            name: kubernetes-dashboard   #service名称
            port:     #后端端口
              number: 443

#加载配置文件
$ kubectl apply -f ingress-dashboard.yaml
ingress.networking.k8s.io/dashboard-ingress created
# 检查Ingress配置文件
$ kubectl describe ingress -n kubernetes-dashboard
Name:             dashboard-ingress
Labels:           <none>
Namespace:        kubernetes-dashboard
Address:          
Ingress Class:    nginx
Default backend:  <default>
TLS:
  dashboard-tls terminates yzj-k8s.com
Rules:
  Host         Path  Backends
  ----         ----  --------
  yzj-k8s.com  
               /   kubernetes-dashboard:443 (10.244.166.153:8443)
Annotations:   nginx.ingress.kubernetes.io/backend-protocol: HTTPS
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    135m  nginx-ingress-controller  Scheduled for sync

11.9 编辑本地hosts解析并访问UI

a.hosts文件添加

# 192.168.77.134是ingress-nginx的节点地址
192.168.77.134  yzj-k8s.com

b. 访问

c. 创建管理token

# 创建管理员  dashboard-admin 
$ kubectl create serviceaccount dashboard-admin -n kube-system

# 在整个集群中为管理员 dashboard-admin 授权
$ kubectl create clusterrolebinding dashboard-admin \
--clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

# 创建集群管理员登录dashboard所需token:
$ kubectl create token dashboard-admin -n kube-system --duration=87600h
eyJhbGciOiJSUzI1NiIsImtpZCI6ImxwOXhhbnJQTHZYWWlrQUx3WENFZzZZUlBhRW1kOVcxUDJsSzludERpQUEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoyMDMwOTM3MTMxLCJpYXQiOjE3MTU1NzcxMzEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkYXNoYm9hcmQtYWRtaW4iLCJ1aWQiOiIwMTQyYTZmOC1hMGVlLTQwMzctODJmYS0wOGRjM2RiN2QxMmQifX0sIm5iZiI6MTcxNTU3NzEzMSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.0ZMAG1E_PnPcd-GcpSm6HLOCkaZBMHM3_Q_6ewLbXBztcAxMEHbCenUAERSXgsWPd6AZKcuT_HtIc1f-DG9jyR6JkQ8jWiH5cAcdzKjtlNrDkqVeIOz4K-2xZewJfGTOvMLVtxtHyvOOZe8JXXGKO2bb7anYN8Fe4r0nCSEp7Sc1AZRL-S1993-Qe6HPPaNXvejiSi7-g1UeCyHZHXjGx8Ociluu9TV1MxsYso4Bmxflug0HVtpv0t3lYliT0EaJWkHjU2zkooXI8l7ltdeogTXdQBWd2KrNArtId4EfQ21Qt-B-SjwbHlEyPX8qAhnOUyGb4hD1hoGiT8Msk1q3Og

通过token登录，输入上面的token，再次访问