kubeedge1.8.2安装+edgemesh配置

前提

云服务器（有公网ip+内网ip）
内网服务器（只有内网ip）

环境

• ubuntu 18
• docker
  docker需要额外配置一下

cat > /etc/docker/daemon.json <<EOF
> {
>   "exec-opts": ["native.cgroupdriver=systemd"]
> }
> EOF

systemctl restart docker

修改主机名（不修改直接跳过

hostnamectl set-hostname yourhostname
vi /etc/hosts #添加127.0.0.1 yourhostname
重启

生效应该是能看到下面这样，我修改的主机名称是tx

master配置

安装kubeadm kubelet kubectl

Linux国内环境下安装kubeadm、kubelet、kubectl

apt-get update && apt-get install -y apt-transport-https curl

apt-key add apt-key.gpg( 公匙下载地址：https://packages.cloud.google.com/apt/doc/apt-key.gpg )

#k8s源
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF

apt-get update

# 安装1.21版本
apt-get install -y kubelet=1.21.0-00 kubeadm=1.21.0-00 kubectl=1.21.0-00

#锁定版本
apt-mark hold kubelet kubeadm kubectl

关于外网设备访问一个advertise-address为内网IP的内网构建的kubernetes集群的问题

已经加入集群的一个agent执行kubectl命令时，报错如下：

参考：
Unable to connect to the server: x509: certificate is valid for问题解决
访问k8s集群出现Unable to connect to the server: x509: certificate is valid for xxx, not xxx问题解决【详细步骤】

rm /etc/kubernetes/pki/apiserver.*
kubeadm init phase certs apiserver --apiserver-advertise-address 内网Ip --apiserver-cert-extra-sans 公网ip
docker ps|grep apiserver
docker restart ** **

效果：在agent里也能访问cluster的情况

kubeadm 安装k8s

如果直接kubeadm init，会遇到下面的报错：Kubernetes init 提示 [ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/

解决办法：先提前pull好镜像

docker pull registry.aliyuncs.com/google_containers/coredns:1.8.0 
docker tag registry.aliyuncs.com/google_containers/coredns:1.8.0 registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0

kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16（这是个固定值） --apiserver-advertise-address=cp(内网地址) --kubernetes-version=v1.21.0

出现下面的提示就安装成功了

image

kubectl生效

想要kubectl生效，有两种解决办法：
法1：

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

法2：

export KUBECONFIG=/etc/kubernetes/admin.conf ！#注意admin.conf的正确路径！

最后的效果：

kubectl get pods -n kube-system

image

kubeedge

克服一些网络问题

在这个网站查找github.com和raw.githubusercontent.com的ip，加入到/etc/hosts里

eg:

185.199.108.133 githubusercontent.com
185.199.109.133 githubusercontent.com
140.82.114.4 github.com

提前下载好的文件

https://github.com/kubeedge/kubeedge/releases/download/v1.8.2/keadm-v1.8.2-linux-amd64.tar.gz

tar -zxvf keadm-v1.8.2-linux-amd64.tar.gz # 解压keadm的tar.gz的包
cd keadm-v1.8.2-linux-amd64/keadm/
cp keadm /usr/sbin/ #将其配置进入环境变量，方便使用

/etc/kubeedge 下载好的：
https://raw.githubusercontent.com/kubeedge/kubeedge/release-1.7/build/tools/cloudcore.service
https://github.com/kubeedge/kubeedge/releases/download/v1.7.0/kubeedge-v1.7.0-linux-amd64.tar.gz

keadm init --advertise-address=公网ip --kubeedge-version=1.8.2

出现下图就成功了（下图是安装的1.7.0版本时截的图

image

[重要！]关闭kube-proxy

官方文档指出，kubeedge默认是排斥kube-proxy的，因为有可以代替proxy的组件edgemesh。需要关闭kube-proxy，配置edgemesh：

image-20220305100155997

如果没有关闭proxy，会在节点里看见问题1中kube-proxy的报错

官方文档也说了如果想用kube-proxy应该如何配置，但是我按照官方文档中的配置后没有成功，依然有问题1的报错，所以还是采用了edgemesh取代kube-proxy的方法

kubectl edit daemonsets.apps -n kube-system kube-proxy 添加affinity这一段

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "4"
  creationTimestamp: "2022-02-28T00:53:47Z"
  generation: 4
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "486209"
  uid: 4151fb3a-dfda-49af-b162-798bf3d63d96
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-proxy
    spec:
    -------
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: DoesNotExist
     -------
      containers:
      - command:
      ……

配置好后节点上就不会有kube-proxy相关的pod和容器了

image

edgemesh

教程

修改cloudcore

apiVersion: cloudcore.config.kubeedge.io/v1alpha1
commonConfig:
  tunnelPort: 10350
kind: CloudCore
kubeAPIConfig:
  burst: 200
  contentType: application/vnd.kubernetes.protobuf
  kubeConfig: /root/.kube/config
  master: ""
  qps: 100
modules:
  cloudHub:
    advertiseAddress:
    - 公网IP
    dnsNames:
    - ""
    edgeCertSigningDuration: 365
    enable: true
    https:
      address: 0.0.0.0
      enable: true
      port: 10002
    keepaliveInterval: 30
    nodeLimit: 1000
    quic:
      address: 0.0.0.0
      enable: false
      maxIncomingStreams: 10000
      port: 10001
    tlsCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsCAKeyFile: /etc/kubeedge/ca/rootCA.key
    tlsCertFile: /etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    tokenRefreshDuration: 12
    unixsocket:
      address: unix:///var/lib/kubeedge/kubeedge.sock
      enable: true
    websocket:
      address: 0.0.0.0
      enable: true
      port: 10000
    writeTimeout: 30
  cloudStream:
  ---
    enable: true
  ---
    streamPort: 10003
    tlsStreamCAFile: /etc/kubeedge/ca/streamCA.crt
    tlsStreamCertFile: /etc/kubeedge/certs/stream.crt
    tlsStreamPrivateKeyFile: /etc/kubeedge/certs/stream.key
    tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
    tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
    tunnelPort: 10004
  deviceController:
    buffer:
      deviceEvent: 1
      deviceModelEvent: 1
      updateDeviceStatus: 1024
    context:
      receiveModule: devicecontroller
      responseModule: cloudhub
      sendModule: cloudhub
    enable: true
    load:
      updateDeviceStatusWorkers: 1
  dynamicController:
    enable: true
  edgeController:
    buffer:
      configMapEvent: 1
      deletePod: 1024
      endpointsEvent: 1
      podEvent: 1
      queryConfigMap: 1024
      queryEndpoints: 1024
      queryNode: 1024
      queryPersistentVolume: 1024
      queryPersistentVolumeClaim: 1024
      querySecret: 1024
      queryService: 1024
      queryVolumeAttachment: 1024
      ruleEndpointsEvent: 1
      rulesEvent: 1
      secretEvent: 1
      serviceAccountToken: 1024
      serviceEvent: 1
      updateNode: 1024
      updateNodeStatus: 1024
      updatePodStatus: 1024
    context:
      receiveModule: edgecontroller
      responseModule: cloudhub
      sendModule: cloudhub
      sendRouterModule: router
    enable: true
    load:
      ServiceAccountTokenWorkers: 4
      UpdateRuleStatusWorkers: 4
      deletePodWorkers: 4
      queryConfigMapWorkers: 4
      queryEndpointsWorkers: 4
      queryNodeWorkers: 4
      queryPersistentVolumeClaimWorkers: 4
      queryPersistentVolumeWorkers: 4
      querySecretWorkers: 4
      queryServiceWorkers: 4
      queryVolumeAttachmentWorkers: 4
      updateNodeStatusWorkers: 1
      updateNodeWorkers: 4
      updatePodStatusWorkers: 1
    nodeUpdateFrequency: 10
  router:
    address: 0.0.0.0
    enable: false
    port: 9443
    restTimeout: 60
  syncController:
    enable: true

安装helm

tar -zxvf helm-v3.7.0-linux-amd64.tar.gz 
cp helm /usr/local/bin
helm version

image

helm安装kubeedge

helm install edgemesh --set server.nodeName=你的master名字 --set server.publicIP=公网IP https://raw.githubusercontent.com/kubeedge/edgemesh/main/build/helm/edgemesh.tgz

kubectl get pod -owide -A

image

查看cloud的日志

/var/log/kubeedge/cloudcore.log

agent

服务器执行

提前下载好文件，和master一样

tar -zxvf keadm-v1.8.2-linux-amd64.tar.gz 
cp keadm-v1.8.2-linux-amd64/keadm/keadm /usr/sbin/
mkdir /etc/kubeedge/
mv edgecore.service /etc/kubeedge/
mv checksum_kubeedge-v1.8.2-linux-amd64.tar.gz.txt /etc/kubeedge/
mv kubeedge-v1.8.2-linux-amd64.tar.gz /etc/kubeedge/

keadm join --cloudcore-ipport=公网IP:10000 --edgenode-name=agent
名字  --kubeedge-version=1.8.2 --token=（在maaster执行keadm gettoken获得）

agent执行kubectl需要的准备

apt-key add apt-key.gpg
cat >>/etc/apt/sources.list.d/kubernetes.list <<EOF
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF

apt-get update
apt-get install kubectl=1.21.0-00

edgemesh

还是根据教程
不同版本的kubeedge edgecore.yaml长得不一样
vi /etc/kubeedge/config/edgecore.yaml:

apiVersion: edgecore.config.kubeedge.io/v1alpha1
database:
  aliasName: default
  dataSource: /var/lib/kubeedge/edgecore.db
  driverName: sqlite3
kind: EdgeCore
modules:
  dbTest:
    enable: false
  deviceTwin:
    enable: true
  edgeHub:
    enable: true
    heartbeat: 15
    httpServer: https://公网IP:10002
    projectID: e632aba927ea4ac2b575ec1603d56f10
    quic:
      enable: false
      handshakeTimeout: 30
      readDeadline: 15
      server: 172.16.0.2:10001
      writeDeadline: 15
    rotateCertificates: true
    tlsCaFile: /etc/kubeedge/ca/rootCA.crt
    tlsCertFile: /etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    token: --
    websocket:
      enable: true
      handshakeTimeout: 30
      readDeadline: 15
      server: 公网IP:10000
      writeDeadline: 15
  edgeStream:
    enable: true
    handshakeTimeout: 30
    readDeadline: 15
    server: 公网IP:10004
    tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
    tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
    writeDeadline: 15
  edged:
    cgroupDriver: cgroupfs
    cgroupRoot: ""
    cgroupsPerQOS: true
    ---
    clusterDNS: "169.254.96.16"
    clusterDomain: "cluster.local"
    ---
    cniBinDir: /opt/cni/bin
    cniCacheDirs: /var/lib/cni/cache
    cniConfDir: /etc/cni/net.d
    concurrentConsumers: 5
    devicePluginEnabled: false
    dockerAddress: unix:///var/run/docker.sock
    edgedMemoryCapacity: 7852396000
    enable: true
    enableMetrics: true
    gpuPluginEnabled: false
    hostnameOverride: sat-200
    imageGCHighThreshold: 80
    imageGCLowThreshold: 40
    imagePullProgressDeadline: 60
    maximumDeadContainersPerPod: 1
    networkPluginMTU: 1500
    nodeIP: 172.16.0.2
    nodeStatusUpdateFrequency: 10
    podSandboxImage: kubeedge/pause:3.1
    registerNode: true
    registerNodeNamespace: default
    remoteImageEndpoint: unix:///var/run/dockershim.sock
    remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
    runtimeRequestTimeout: 2
    runtimeType: docker
    volumeStatsAggPeriod: 60000000000
  eventBus:
    enable: true
    eventBusTLS:
      enable: false
      tlsMqttCAFile: /etc/kubeedge/ca/rootCA.crt
      tlsMqttCertFile: /etc/kubeedge/certs/server.crt
      tlsMqttPrivateKeyFile: /etc/kubeedge/certs/server.key
    mqttMode: 2
    mqttQOS: 0
    mqttRetain: false
    mqttServerExternal: tcp://127.0.0.1:1883
    mqttServerInternal: tcp://127.0.0.1:1884
    mqttSessionQueueSize: 100
  metaManager:
    contextSendGroup: hub
    contextSendModule: websocket
    enable: true
    metaServer:
      debug: false
      ---
      enable: true
      ---
    podStatusSyncInterval: 60
    remoteQueryTimeout: 60
  serviceBus:
    enable: false

systemctl restart edgecore

验证local apiserver是否开启，如果没有返回结果，说明edgecore.yaml没有配置正确，10550端口没有开启

curl 127.0.0.1:10550/api/v1/services

image

能在agent里看到POD和对应的容器