背景

本文主要讲解kubernetes 如何使用helm chart安装 ingress-nginx

kubernetes 使用helm chart安装 ingress-nginx

1. ingres nginx controller 和 k8s版本 兼容性要求

https://github.com/kubernetes/ingress-nginx/blob/main/README.md#supported-versions-table
Supported Ingress-NGINX version k8s supported version Alpine Version Nginx Version Helm Chart Version
🔄 v1.11.2 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.11.2
🔄 v1.11.1 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.11.1
🔄 v1.11.0 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.11.0
🔄 v1.10.4 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.10.4
🔄 v1.10.3 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.10.3

2.安装环境

注: linux amd64 替换相关镜像版本 & helm3安装包即可-安装流程同理

linux arm64: 内核版本 4.18.0-348.20.1.el7.aarch64 #1 SMP Wed Apr 13 20:57:50 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Kubernetes: v1.28.0
Docker: 26.1.4
ingress-nginx: 4.11.2

3.安装离线镜像准备

# 下载 ingress-controller依赖镜像(国内机器有墙无法拉取)
docker pull registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3
docker pull registry.k8s.io/ingress-nginx/controller:v1.11.2
# 导出为离线镜像
docker save -o kube-webhook-certgen-v1.4.3.tar registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3
docker save -o controller-v1.11.2.tar registry.k8s.io/ingress-nginx/controller:v1.11.2
# 安装k8s所有机器节点导入离线镜像
docker load -i controller-v1.11.2.tar
docker load -i kube-webhook-certgen-v1.4.3.tar

# docker images|grep ingress
registry.k8s.io/ingress-nginx/controller             v1.11.2   289a818c8d9c   2 weeks ago     294MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   v1.4.3    420193b27261   3 weeks ago     53.3MB


# 镜像打tag & push到本地仓库[可选]
#docker tag registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3 sealos.hub:5000/ingress-nginx/kube-webhook-certgen:v1.4.3
#docker push sealos.hub:5000/ingress-nginx/kube-webhook-certgen:v1.4.3
#docker tag registry.k8s.io/ingress-nginx/controller:v1.11.2 sealos.hub:5000/ingress-nginx/controller:v1.11.2
#docker push sealos.hub:5000/ingress-nginx/controller:v1.11.2

4.linux(amd64)安装 helm3

参考 https://helm.sh/zh/docs/intro/install/
https://github.com/helm/helm/releases

wget https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz
tar -xvf helm-v3.15.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
helm version

5.创建k8s拉取镜像-镜像仓库验证鉴权信息

# 创建镜像仓库验证鉴权信息: k8s拉取验证 结合 imagePullSecrets: imagePullSecrets: - name: scr 引用使用
kubectl create secret docker-registry scr \
  -n ingress-nginx \
  --docker-server=http://sealos.hub:5000 \
  --docker-username=admin \
  --docker-password=123456 \
  --docker-email=jinze@ali.com
# 删除镜像仓库验证鉴权信息
kubectl delete secret -n ingress-nginx scr

# 查看解密secret内容
kubectl get secret  -n ingress-nginx scr --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode  

4.解压安装ingress-nginx

# helmchart 安装 ingrss nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# helm列出所有版本:
helm search repo ingress-nginx/ingress-nginx -l
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
ingress-nginx/ingress-nginx     4.11.2          1.11.2          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.11.1          1.11.1          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.11.0          1.11.0          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.4          1.10.4          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.3          1.10.3          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.2          1.10.2          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.1          1.10.1          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.0          1.10.0          Ingress controller for Kubernetes using NGINX a...

# helm 下载指定版本：4.11.2 
helm fetch ingress-nginx/ingress-nginx --version 4.11.2 

# 解压ingress-nginx4.11.2 版本安装包
tar -xvf ingress-nginx-4.11.2.tgz

# 编辑 ingress-nginx 配置 values.yaml
vi ingress-nginx/values.yaml 

# 配置controller镜像
controller:
  image:
    chroot: false
    registry: registry.k8s.io
    image: ingress-nginx/controller
    tag: "v1.11.2"
    #digest: sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce
    # digest 需配置为空才能拉取 registry.k8s.io/ingress-nginx/controller:v1.11.2 镜像
    digest:

# 配置admissionWebhooks镜像
controller:
  admissionWebhooks:
    patch:
      image:
        digest: sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3
        # digest 需配置为空才能拉取 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3 镜像
        digest:

# ingress 暴露为NodePort
controller:
  service:
    #type: LoadBalancer
    type: NodePort
# k8s 拉取镜像仓库验证secret
#imagePullSecrets: []
imagePullSecrets: 
- name: scr

# helm chart 安装（存在就更新） ingress-nginx  
cd /root/ingress-nginx && helm upgrade --install ingress-nginx .  --namespace ingress-nginx --create-namespace 

# 卸载 ingress-nginx 
helm uninstall ingress-nginx -n ingress-nginx
# 查看安装 ingress 
helm list -A|grep ingress
# 验证ingress 组件状态，是否正常拉起
kubectl get svc -A |grep ingress
kubectl get pod -A |grep ingress
kubectl get deploy -n ingress-nginx               ingress-nginx-controller   -oyaml

# ingress-nginx 成功安装效果
[root@bj-arm-master ingress-nginx]# helm upgrade --install ingress-nginx .  --namespace ingress-nginx --create-namespace 
Release "ingress-nginx" has been upgraded. Happy Helming!
NAME: ingress-nginx
LAST DEPLOYED: Fri Sep  6 11:20:42 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
Get the application URL by running these commands:
  export HTTP_NODE_PORT=$(kubectl get service --namespace ingress-nginx ingress-nginx-controller --output jsonpath="{.spec.ports[0].nodePort}")
  export HTTPS_NODE_PORT=$(kubectl get service --namespace ingress-nginx ingress-nginx-controller --output jsonpath="{.spec.ports[1].nodePort}")
  export NODE_IP="$(kubectl get nodes --output jsonpath="{.items[0].status.addresses[1].address}")"

  echo "Visit http://${NODE_IP}:${HTTP_NODE_PORT} to access your application via HTTP."
  echo "Visit https://${NODE_IP}:${HTTPS_NODE_PORT} to access your application via HTTPS."

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

6.配置 ingress 转发规则:

-- ingress配置demo1： 访问 路径/ 转发到后端 namespace为default的 bte-service的8080端口
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: bte
 namespace: default
 #annotations:
   # 无论客户端请求的是哪个路径，Ingress 控制器都会将目标请求路径重写为根路径 /
   # 当用户访问 http://example.com/foo 时，NGINX Ingress Controller 会将请求重写为 http://my-service:80/。也就是说，任何通过 /foo 访问的请求都会转发到 my-service 服务，并且请求路径会被重写为根路径 /
   #nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 # 指定 Ingress Controller 的类型 为 nginx 类型：告诉 Kubernetes，这个 Ingress 由 NGINX Ingress Controller 处理
 ingressClassName: nginx
 rules:
 #- host: "*"
 - http:
     paths:
     - path: /
       pathType: Prefix
       backend:
         service:
           name: bte-service
           port:
             number: 8080

-- ingress配置demo2:  访问 路径/layout 转发到后端 namespace为default的 layout-service的8080端口
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: layout
 namespace: default
 annotations:
   # 无论客户端请求的是哪个路径，Ingress 控制器都会将目标请求路径重写为根路径 /
   # 当用户访问 http://example.com/foo 时，NGINX Ingress Controller 会将请求重写为 http://my-service:80/。也就是说，任何通过 /foo 访问的请求都会转发到 my-service 服务，并且请求路径会被重写为根路径 /
   nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 # k8s 1.18版本后 指定 Ingress Controller 的类型 为 nginx 类型配置：告诉 Kubernetes，这个 Ingress 由 NGINX Ingress Controller 处理； 
 ingressClassName: nginx  
 rules:
 #- host: "*"
 - http:
     paths:
     - path: /layout
       pathType: Prefix
       backend:
         service:
           name: layout-service
           port:
             number: 8080

3.安装过程遇到安装失败问题处理

安装遇到问题: 无法正常拉取镜像ImagePullBackOff

# kubectl get pod -A |grep ingress
ingress-nginx      ingress-nginx-admission-create-nz6hv       0/1     ImagePullBackOff   0          64s
# kubectl describe pod -n ingress-nginx      ingress-nginx-admission-create-nz6hv 

问题1: 报错: 无法正常拉取镜像 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3

image.png
image.png

此镜像版本比我们离线导入的image tag 多了 @sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3 信息

image.png

查看helm chart 源码分析问题原因

无法正常拉取镜像 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3
vi /root/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml & job-patchWebhook.yaml
digest: sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3

image.png
image.png

** fix: digest 设置为空即可 **

image.png

重启 ingress-nginx

helm upgrade --install ingress-nginx . --namespace ingress-nginx --create-namespace

问题2: 无法正常拉取镜像 registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce

# kubectl get pod -A |grep ingress
ingress-nginx      ingress-nginx-controller-5bddfb7dbf-gzjsx   0/1     ImagePullBackOff   0          49s     100.78.46.152    bj-arm-node1    <none>           <none>
# kubectl describe pod -n ingress-nginx      ingress-nginx-controller-5bddfb7dbf-gzjsx
  Failed to pull image "registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce": Error response from daemon: Get "https://registry.k8s.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

此镜像版本比我们离线导入的image tag 多了 @sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce 信息

查看helm chart 源码分析问题原因

cat /root/ingress-nginx/templates/controller-deployment.yaml

image.png

###### 检查当前目录下所有yaml文件中包含 字符串 ingress-nginx.imageDigest
grep -o "ingress-nginx.imageDigest"  ./*.*
# 输出 
/root/ingress-nginx/templates/_helpers.tpl 

image.png image.png
image.png

fix: digest 设置为空即可

image.png

重启 ingress-nginx

helm upgrade --install ingress-nginx .  --namespace ingress-nginx --create-namespace 

ingress-nginx pod正常拉起，问题fix

# kubectl get pod -A |grep ingress
ingress-nginx      ingress-nginx-controller-785fcc99b-2zdhx   1/1     Running   0          22s

image.png
image.png

-- 问题fix!

参考文档

https://kubernetes.io/docs/concepts/services-networking/ingress/ ingress文档
https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/ Ingress Controllers 文档
-- ingress-nginx相关文档
https://github.com/kubernetes/ingress-nginx/blob/main/README.md#readme ingres nginx controller github文档
https://github.com/kubernetes/ingress-nginx Ingress-nginx 文档(支持 helm chart部署) use
https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx ingress-nginx helm chart文档
https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx#values ingress-nginx helm chart values.yaml 配置说明
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/index.md
https://kubernetes.github.io/ingress-nginx/user-guide/tls/ ngress-nginx 配置文档