美文网首页
kubeadm 搭建 K8s

kubeadm 搭建 K8s

作者: Aska小强 | 来源:发表于2022-04-17 18:44 被阅读0次

    kubeadm 搭建 K8s

    本篇主要记录一下 使用 kubeadm 搭建 k8s 详细过程 ,环境使用 VirtualBox 构建的3台虚拟机

    1.环境准备

    • 操作系统:Centos7 (CentOS Linux release 7.9.2009)
    • Master 主节点: 1台 虚拟机
    • Node 计算节点: 2台 虚拟机
    • K8s version:v1.23.5 (选的较新的版本)
    • Docker version:20.10.14

    虚拟机创建我就不详细展开了

    2.虚拟机网络配置(保证能访问外网)

    虚拟机网卡 选择的是 1.网络地址转换NAT (主要用来访问外网的) 2.仅主机网络 (主要用来宿主机访问虚拟机的)

    把对应的 ip 设置如下

    kube-master 192.168.56.20

    kube-node-1 192.168.56.21

    kube-node-2 192.168.56.22

    2.1 3台虚拟机 静态ip配置

    cat /etc/sysconfig/network-scripts/ifcfg-enp0s3

    TYPE="Ethernet"
    PROXY_METHOD="none"
    BROWSER_ONLY="no"
    BOOTPROTO="none" #关闭
    DEFROUTE="yes"
    IPV4_FAILURE_FATAL="no"
    IPV6INIT="yes"
    IPV6_AUTOCONF="yes"
    IPV6_DEFROUTE="yes"
    IPV6_FAILURE_FATAL="no"
    IPV6_ADDR_GEN_MODE="stable-privacy"
    NAME="enp0s3"
    UUID="35eac2d4-234d-4859-848e-ed2a30a64a81"
    DEVICE="enp0s3"
    #添加部分 
    IPADDR="10.0.2.20"
    PREFIX="24"
    GATEWAY="10.0.2.2"
    DNS1="10.0.2.3"
    DNS2="114.114.114.114"
    ONBOOT="yes"
    

    cat /etc/sysconfig/network-scripts/ifcfg-enp0s8

    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=static #修改为 static 静态ip
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=enp0s8
    UUID=d06bac6d-294f-4bfe-bf08-5cb65c0a431f
    DEVICE=enp0s8
    #新增部分
    IPADDR=192.168.56.20
    PREFIX=24
    ONBOOT=yes
    

    2.2 虚拟机hosts 配置

    在 3台虚拟机 /etc/hosts 中配置 如下

    192.168.56.20   kube-master
    192.168.56.21   kube-node-1
    192.168.56.22   kube-node-2
    
    image-20220415170426999

    3.k8s 基础环境设置

    3.1 yum 换源(可选)

    yum -y install wget
    
    cd /etc/yum.repos.d
    mv CentOS-Base.repo CentOS-Base.repo.bak
    wget -O CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
    wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
    yum clean all
    yum makecache #重新生成缓存
    

    3.2 关闭 selinux

    setenforce 0 #临时关闭
    sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config #永久关闭
    

    3.3 关闭swap分区或禁用swap文件

    swapoff -a
    # 注释掉关于swap分区的行
    yes | cp /etc/fstab /etc/fstab_bak
    cat /etc/fstab_bak |grep -v swap > /etc/fstab
    

    3.4 允许iptables 检查桥接流量

    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
    br_netfilter
    EOF
    
    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF
    
    sudo sysctl --system
    

    3.5 关闭防火墙

    systemctl stop firewalld
    systemctl disable firewalld
    

    3.6 /proc/sys/net/ipv4/ip_forward

    echo 1 > /proc/sys/net/ipv4/ip_forward
    

    4.kubectl、kubeadm、kubelet的安装

    注意 先换源 https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.64be1b11iddYWH 阿里换源文档

    4.1 添加Kubernetes的alibaba yum源

    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    EOF
    
    yum install -y kubelet kubeadm kubectl
    
    
    

    4.2 安装 kubelet kubeadm kubectl

    yum install -y kubelet kubeadm kubectl 
    systemctl enable kubelet && systemctl start kubelet 
    

    由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y --nogpgcheck kubelet kubeadm kubectl 安装

    4.3 启动kubelet 服务

    systemctl enable kubelet && systemctl start kubelet
    

    此时执行systemctl status kubelet查看服务状态,此时肯定有错误(因为我们还未进行初始化操作),错误可使用journalctl -xe查看错误信息。

    5.Docker 安装和配置

    Docker centos 安装文档 地址 https://docs.docker.com/engine/install/centos/

    5.1 Docker 安装

    sudo yum install -y yum-utils
    
    sudo yum-config-manager \
        --add-repo \
        https://download.docker.com/linux/centos/docker-ce.repo
        
        
    sudo yum install docker-ce docker-ce-cli containerd.io 
    # 也可以安装其他版本 yum list docker-ce --showduplicates | sort -r  
    # sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io
    
    #启动docker
    sudo systemctl enable docker && systemctl start docker 
    

    5.2 Docker 镜像源更换

    sudo tee /etc/docker/daemon.json <<-'EOF'
    {
      "registry-mirrors": ["https://9xxxx1clo.mirror.aliyuncs.com"] #请更换自己的 阿里云镜像地址
    }
    EOF
    

    5.3 Docker 配置cgroup-driver

    由于 cgroup-driver ,默认docker 是 cgroupfs ,而k8s 默认是 systemd,所以需要设置 docker 的 cgroup 为 systemd

    推荐使用方式二进行修改

    #查看cgroup-driver
    
     docker info | grep -i cgroup
    # 方式一 追加 --exec-opt native.cgroupdriver=systemd 参数
     sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service
     
     systemctl daemon-reload # 重新加载服务
     systemctl enable docker # 启用docker服务(开机自起)
     systemctl restart docker # 启动docker服务
    
    # 方式二 修改docker配置文件
     vim /etc/docker/daemon.json
    {
      "exec-opts": ["native.cgroupdriver=systemd"]
    }
    

    5.4 拉取k8s所需镜像

    可以通过

    kubeadm config images list 查看 所需的镜像

    kubeadm config images list 
    
    k8s.gcr.io/kube-apiserver:v1.23.5
    k8s.gcr.io/kube-controller-manager:v1.23.5
    k8s.gcr.io/kube-scheduler:v1.23.5
    k8s.gcr.io/kube-proxy:v1.23.5
    k8s.gcr.io/pause:3.6
    k8s.gcr.io/etcd:3.5.1-0
    k8s.gcr.io/coredns/coredns:v1.8.6
    

    如果拉取不到镜像 可以通过 去 docker hub 上查找对应的 镜像

    image-20220415171557414
    docker pull k8simage/kube-apiserver:v1.23.5 
     
    docker tag k8simage/kube-apiserver:v1.23.5 k8s.gcr.io/kube-apiserver:v1.23.5 #重新tag改成上面所需的镜像即可, 其他的都按照这样来
    
    image-20220415172704185

    Tips: 完成上面操作后 你可以通过 docker save 导出镜像,以后再用的时候 可以通过 docker image load < 进行离线加载镜像

    #导出镜像 ,批量导出所有镜像
    
    docker save $(docker images --format '{{.Repository}}:{{.Tag}}') -o k8s.images.tar 
    #导入镜像 ,注意 是 < 不是 > 。。 > 后会导致tar 无法使用需要重新上传
    docker image load < ...tar/zip 
    

    6.kube-master 节点 初始化

    我这里选的是 flannel 网络插件,先来看看 flannel 的配置文件

    6.1 查看flannel 网络范围

    flannel.yml https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

    不同的 网络插件 网络范围不同

    image-20220415173212001

    6.2 kubeadm init 初始化

    kubeadm init --pod-network-cidr=10.244.0.0/16 \ #网卡插件的 网络范围  --apiserver-advertise-address=192.168.56.20 #master 的 ip 地址
    

    等一会就会有如下结果

    image-20220415173521177

    可以把 kubeadm join 命令记下来 便于后面使用 保存输出中的kubeadm join部分内容,用于添加node节点,或者使用kubeadm token listkubeadm token create --print-join-command查看

    执行后续步骤,可以看到提示

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    

    6.3 网络插件flannel 配置

    前面提到了flannel 把对应的flannel.yml文件下载到服务器中,可以使用wget

    可以看到 在没配置 flannel 的时候 coredns 一直是 pending 状态

    image-20220414184915596
    wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml #可能会下载不到,文章最后我会提供内容,直接copy过去即可
    
    #Tips: 可能需要指定一下网卡 ,在kube-flannel.yml中 - /opt/bin/flanneld args: 后面添加
    #- --iface=enp0s8 # 需要指定网卡,否则可能k8s会导致使用错误的网卡 
    
    kubectl apply -f kube-flannel.yml #应用该文件
    
    image-20220414224216803
    kubectl get pods --all-namespaces -o wide #查看是否安装成功,它会在 master 和 node 节点上都会自动部署
    
    image-20220415175527461

    最后通过 kubectl get nodes -o wide 查看 节点

    7.kube-node 节点初始化

    7.1 查看token

    可以通过该命令进行查看 token

    kubeadm token create --print-join-command
    
    #kubeadm join kube-master:6443 --token ii5xyw.mywlktar1e8f60cu --discovery-token-ca-cert-hash sha256:358f92418178342f79aa7b049e6c18a423f0fb88186189142459016fe6efb8e7
    

    7.2 node节点上 执行 kubeadmin join

    kubeadm join kube-master:6443 --token ii5xyw.mywlktar1e8f60cu --discovery-token-ca-cert-hash sha256:358f92418178342f79aa7b049e6c18a423f0fb88186189142459016fe6efb8e7
    

    7.3 在master 上查看结果

    watch kubectl get nodes -o wide #可以通过watch 实时监听 加入情况
    
    image-20220415180246749

    至此使用 kubeadm 已经成功搭建k8s 集群

    总结

    本篇主要讲解了 如何使用 kubeadm 进行搭建 k8s 集群 ,搭建的是 k8s 较新版本 v1.23.5 版本,使用kubeadm 搭建相对来说还是比较简单的 。

    k8s官网文档

    https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

    阿里 k8s 更换镜像文档

    https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.64be1b11iddYWH

    Docker hub 地址

    https://hub.docker.com/r/k8simage/

    Docker 安装地址

    https://docs.docker.com/engine/install/centos/

    flannel.yml 文件

    ---
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
      name: psp.flannel.unprivileged
      annotations:
        seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
        seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
        apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
        apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    spec:
      privileged: false
      volumes:
      - configMap
      - secret
      - emptyDir
      - hostPath
      allowedHostPaths:
      - pathPrefix: "/etc/cni/net.d"
      - pathPrefix: "/etc/kube-flannel"
      - pathPrefix: "/run/flannel"
      readOnlyRootFilesystem: false
      # Users and groups
      runAsUser:
        rule: RunAsAny
      supplementalGroups:
        rule: RunAsAny
      fsGroup:
        rule: RunAsAny
      # Privilege Escalation
      allowPrivilegeEscalation: false
      defaultAllowPrivilegeEscalation: false
      # Capabilities
      allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
      defaultAddCapabilities: []
      requiredDropCapabilities: []
      # Host namespaces
      hostPID: false
      hostIPC: false
      hostNetwork: true
      hostPorts:
      - min: 0
        max: 65535
      # SELinux
      seLinux:
        # SELinux is unused in CaaSP
        rule: 'RunAsAny'
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: flannel
    rules:
    - apiGroups: ['extensions']
      resources: ['podsecuritypolicies']
      verbs: ['use']
      resourceNames: ['psp.flannel.unprivileged']
    - apiGroups:
      - ""
      resources:
      - pods
      verbs:
      - get
    - apiGroups:
      - ""
      resources:
      - nodes
      verbs:
      - list
      - watch
    - apiGroups:
      - ""
      resources:
      - nodes/status
      verbs:
      - patch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: flannel
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: flannel
    subjects:
    - kind: ServiceAccount
      name: flannel
      namespace: kube-system
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: flannel
      namespace: kube-system
    ---
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: kube-flannel-cfg
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    data:
      cni-conf.json: |
        {
          "name": "cbr0",
          "cniVersion": "0.3.1",
          "plugins": [
            {
              "type": "flannel",
              "delegate": {
                "hairpinMode": true,
                "isDefaultGateway": true
              }
            },
            {
              "type": "portmap",
              "capabilities": {
                "portMappings": true
              }
            }
          ]
        }
      net-conf.json: |
        {
          "Network": "10.244.0.0/16",
          "Backend": {
            "Type": "vxlan"
          }
        }
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                    - linux
          hostNetwork: true
          priorityClassName: system-node-critical
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni-plugin
           #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply)
            image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1
            command:
            - cp
            args:
            - -f
            - /flannel
            - /opt/cni/bin/flannel
            volumeMounts:
            - name: cni-plugin
              mountPath: /opt/cni/bin
          - name: install-cni
           #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)
            image: rancher/mirrored-flannelcni-flannel:v0.17.0
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
           #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)
            image: rancher/mirrored-flannelcni-flannel:v0.17.0
            command:
            - /opt/bin/flanneld
            args:
            #- --iface=enp0s8 # 需要指定网卡,否则可能会导致问题
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                add: ["NET_ADMIN", "NET_RAW"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: EVENT_QUEUE_DEPTH
              value: "5000"
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
            - name: xtables-lock
              mountPath: /run/xtables.lock
          volumes:
          - name: run
            hostPath:
              path: /run/flannel
          - name: cni-plugin
            hostPath:
              path: /opt/cni/bin
          - name: cni
            hostPath:
              path: /etc/cni/net.d
          - name: flannel-cfg
            configMap:
              name: kube-flannel-cfg
          - name: xtables-lock
            hostPath:
              path: /run/xtables.lock
              type: FileOrCreate
    

    欢迎大家访问 个人博客 Johnny小屋

    相关文章

      网友评论

          本文标题:kubeadm 搭建 K8s

          本文链接:https://www.haomeiwen.com/subject/dymlertx.html