Kubernetes v1.13.5高可用集群安装

作者: 济南打工人 | 来源:发表于2019-03-30 07:59 被阅读0次

Kubernetes v1.13.5高可用集群安装
CentOS 安装 Kubernetes 高可用集群
基于kubernetes的多master节点部署方案
Redis目录
搭建高可用Kubernetes集群之kubernetes集群搭建
Nginx高可用集群实战，以及原理分析
kubeadm部署kubernetes 1.11.0集群
实用教程丨使用K3s和MySQL运行Rancher 2.4
Kubernetes 高可用集群
sealos2.0使用教程，最简单kubernetesHA方案

基础环境

系统Ubuntu 16.04

主机名称	IP	备注
node01	192.168.175.61	master and etcd
node02	192.168.175.62	master and etcd
node03	192.168.175.63	master and etcd
node04	192.168.175.64	node
VIP	192.168.175.60

软件版本：

软件	版本号
docker	v18.06.1-ce
kubelet	v1.13.5
kubectl	v1.13.5
kubeadm	v1.13.5
etcd	v3.3.5

环境初始化

修改主机名

hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03
hostnamectl set-hostname node04

配置主机映射

cat <<EOF > /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.175.61 node01
192.168.175.62 node02
192.168.175.63 node03
192.168.175.64 node04
EOF

免密码登录

node01上执行ssh免密码登陆配置

ssh-keygen  #一路回车即可
ssh-copy-id  node02
ssh-copy-id  node03
ssh-copy-id  node04

基线配置

四台主机配置设置内核、K8S源、关闭Swap、配置ntp（配置完后重启一次）

swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab    #禁用swap分区

modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge

 #https://opsx.alibaba.com/mirror 阿里云地址
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main 
EOF  
apt-get update
apt-get install -y kubeadm=1.13.5-00
apt-get install -y kubelet=1.13.5-00
apt-get install -y kubectl=1.13.5-00

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
 
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

所有节点配置k8s内核

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
EOF

sysctl --system

创建etcd证书

(node01上执行即可)

设置cfssl环境

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH

创建 CA 配置文件(下面配置的IP为etc节点的IP)

mkdir /root/ssl
cd /root/ssl
cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "8760h"
},
"profiles": {
  "kubernetes-Soulmate": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
    "expiry": "8760h"
  }
}
}
}
EOF

cat >  ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
  "C": "CN",
  "ST": "shanghai",
  "L": "shanghai",
  "O": "k8s",
  "OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.175.61",
    "192.168.175.62",
    "192.168.175.63"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

node01分发etcd证书到node02、node03上面

mkdir -p /etc/etcd/ssl  #三台机器上分别执行
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node02:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node03:/etc/etcd/ssl/

安装配置etcd

(三主节点）

安装etcd

apt install etcd -y

node01的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name node01   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.168.175.61:2380   --listen-peer-urls https://192.168.175.61:2380   --listen-client-urls https://192.168.175.61:2379,http://127.0.0.1:2379   --advertise-client-urls https://192.168.175.61:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster node01=https://192.168.175.61:2380,node02=https://192.168.175.62:2380,node03=https://192.168.175.63:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node02的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name node02   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.168.175.62:2380   --listen-peer-urls https://192.168.175.62:2380   --listen-client-urls https://192.168.175.62:2379,http://127.0.0.1:2379   --advertise-client-urls https://192.168.175.62:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster node01=https://192.168.175.61:2380,node02=https://192.168.175.62:2380,node03=https://192.168.175.63:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node03的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name node03   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.168.175.63:2380   --listen-peer-urls https://192.168.175.63:2380   --listen-client-urls https://192.168.175.63:2379,http://127.0.0.1:2379   --advertise-client-urls https://192.168.175.63:2379   --initial-cluster-token etcd-cluster-0 --initial-cluster node01=https://192.168.175.61:2380,node02=https://192.168.175.62:2380,node03=https://192.168.175.63:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

添加自启动（etc集群最少2个节点才能启动，启动报错看mesages日志）

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

在三个etcd节点执行一下命令检查

etcdctl --endpoints=https://192.168.175.61:2379,https://192.168.175.62:2379,https://192.168.175.63:2379 \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health

etcd升级

（apt安装的版本为v2.2.5，kubernetes v1.10要求版本最低为3.1）

官网下载最新安装包

wget https://github.com/coreos/etcd/releases/download/v3.3.5/etcd-v3.3.5-linux-amd64.tar.gz
tar zxf etcd-v3.3.5-linux-amd64.tar.gz
cp etcd-v3.3.5-linux-amd64/etcd /usr/bin/etcd
cp etcd-v3.3.5-linux-amd64/etcdctl /usr/bin/etcdctl

在/etc/profile文件添加以下一行，重启服务器

export ETCDCTL_API=3

重启etcd服务，并产看集群状态

root@k8s-n2:~/k8s# etcdctl member list
aa76456e260f7bd1, started, node02, https://192.168.175.62:2380, https://192.168.175.62:2379
d12950b45efa96da, started, node03, https://192.168.175.63:2380, https://192.168.175.63:2379
e598ba1c84356928, started, node01, https://192.168.175.61:2380, https://192.168.175.61:2379

安装docker

curl -fsSL "https://get.docker.com/" | sh

docker配置代理

docker 添加阿里云代理，修改配置文件/lib/systemd/system/docker.service

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --registry-mirror=https://ms3cfraz.mirror.aliyuncs.com

启动docker

添加开机自启动

systemctl daemon-reload
systemctl restart docker
systemctl enable docker
systemctl status docker

配置kubeadm

所有节点修改kubelet配置文件

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

#添加这一行
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
#添加这一行
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=xxx.xxxx.xxxx/antk8s/pause:3.1"

所有节点修改完配置文件一定要重新加载配置

systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet

初始化集群

node01、node02、node03添加集群初始配置文件（集群配置文件一样）

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://192.168.175.61:2379
  - https://192.168.175.62:2379
  - https://192.168.175.63:2379
  caFile: /etc/etcd/ssl/ca.pem
  certFile: /etc/etcd/ssl/etcd.pem
  keyFile: /etc/etcd/ssl/etcd-key.pem
  dataDir: /var/lib/etcd
networking:
  podSubnet: 10.244.0.0/16
kubernetesVersion: 1.13.5
api:
  advertiseAddress: "192.168.175.61"
token: "b99a00.a144ef80536d4344"
tokenTTL: "0s"
apiServerCertSANs:
- node01
- node02
- node03
- node04
- 192.168.175.61
- 192.168.175.62
- 192.168.175.63
- 192.168.175.64
featureGates:
  CoreDNS: true
imageRepository: "images.xxxxxxx.com/antk8s"

首先node01初始化集群

配置文件定义podnetwork是10.244.0.0/16
kubeadmin init –hlep可以看出，service默认网段是10.96.0.0/12
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf默认dns地址cluster-dns=10.96.0.10

kubeadm init --config config.yaml

这时会报错,kubeadm需要切换到v1.11转换文件

root@k8s-m1:~# kubeadm init --config config.yaml 
your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha1". Please use kubeadm v1.11 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

root@k8s-m1:~# apt install kubeadm=1.11.0-00
root@k8s-m1:~# kubeadm config migrate --old-config config.yaml --new-config new.yaml
I0330 02:07:39.075699   10719 feature_gate.go:230] feature gates: &{map[]}

再次执行初始化

root@k8s-m1:~# kubeadm init --config new.yaml 
your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha2". Please use kubeadm v1.12 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

kubeadm安装v1.12进行文件转换

root@k8s-m1:~# kubeadm config migrate --old-config new.yaml --new-config new1.yaml

初始化，这时，初始化成功

root@k8s-m1:~# kubeadm init --config new1.yaml 
[init] Using Kubernetes version: v1.13.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
  ...  ...
  ...  ...
  ...  ...
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.175.61:6443 --token b99a00.a144ef80536d4344 --discovery-token-ca-cert-hash sha2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe7a

master上面执行如下命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

部署flannel网络，只需要在node01执行就行

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

执行命令

[root@node01 ~]# kubectl   get node
NAME      STATUS    ROLES     AGE       VERSION
node01    Ready     master    31m       v1.13.5

[root@node01 ~]# kubectl get -n kube-system po 
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66555b45f-grq8d          1/1     Running   0          41m
coredns-66555b45f-hrkcf          1/1     Running   0          41m
kube-apiserver-k8s-m1            1/1     Running   0          40m
kube-controller-manager-k8s-m1   1/1     Running   0          40m
kube-flannel-ds-amd64-gjgpt      1/1     Running   0          8s
kube-proxy-pvkjm                 1/1     Running   0          41m
kube-scheduler-k8s-m1            1/1     Running   0          40m

复制相关文件到其他master上

scp  /etc/kubernetes/pki/* node02:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/* node03:/etc/kubernetes/pki/
scp admin.conf node02:/etc/kubernetes/
scp admin.conf node03:/etc/kubernetes/

两个master执行以下操作，安装master需添参数--experimental-control-plane

root@k8s-n1:/etc/kubernetes# kubeadm join 192.168.175.61:6443 --token b99asssssssssss36d4344 --discovery-token-ca-cert-hash sha256:fffffffffffffffffffffffffffffffffffffffffffffxxxxxxxxxxxxxxxx52cb0d9 --experimental-control-plane
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.175.61:6443"
  ... ...
  ... ...
[mark-control-plane] Marking the node k8s-n1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

查看集群

root@k8s-m1:~# kubectl get no
NAME     STATUS   ROLES    AGE     VERSION
k8s-m1   Ready    master   17m     v1.13.5
k8s-n1   Ready    master   12m     v1.13.5
k8s-n2   Ready    master   11m     v1.13.5
k8s-n3   Ready    <none>   3m28s   v1.13.5

root@k8s-m1:~# kubectl get -n kube-system po 
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66555b45f-4rws8          1/1     Running   0          16m
coredns-66555b45f-dsf2t          1/1     Running   0          16m
kube-apiserver-k8s-m1            1/1     Running   0          15m
kube-apiserver-k8s-n1            1/1     Running   0          12m
kube-apiserver-k8s-n2            1/1     Running   0          11m
kube-controller-manager-k8s-m1   1/1     Running   0          15m
kube-controller-manager-k8s-n1   1/1     Running   0          12m
kube-controller-manager-k8s-n2   1/1     Running   0          11m
kube-flannel-ds-amd64-856gq      1/1     Running   0          12m
kube-flannel-ds-amd64-8f5mj      1/1     Running   0          11m
kube-flannel-ds-amd64-f4pj8      1/1     Running   0          16m
kube-flannel-ds-amd64-l62n6      1/1     Running   0          3m34s
kube-proxy-dj2w5                 1/1     Running   0          3m34s
kube-proxy-dz6cf                 1/1     Running   0          12m
kube-proxy-pqtg9                 1/1     Running   0          11m
kube-proxy-vt5dv                 1/1     Running   0          16m
kube-scheduler-k8s-m1            1/1     Running   0          16m
kube-scheduler-k8s-n1            1/1     Running   0          12m
kube-scheduler-k8s-n2            1/1     Running   0          11m

至此，集群安装完毕。

Kubernetes v1.13.5高可用集群安装
基础环境系统Ubuntu 16.04 软件版本：环境初始化修改主机名配置主机映射免密码登录 node01...
CentOS 安装 Kubernetes 高可用集群
Kubernetes 高可用集群需要实现： API server 组件高可用 etcd 集群高可用，需要至少3台服...
基于kubernetes的多master节点部署方案
kubeadm高可用集群部署 1、安装要求在开始之前，部署Kubernetes集群机器需要满足以下几个条件：五...
Redis目录
安装 Redis安装 RedisCluster高可用集群搭建（一） RedisCluster高可用集群搭建（二） ...
搭建高可用Kubernetes集群之kubernetes集群搭建
在搭建kubernetes集群之前，请参考我的文章搭建高可用Kubernetes集群之etcd集群搭建篇（一）完成...
Nginx高可用集群实战，以及原理分析
知识要点： Nginx高可用集群简介 Keepalived安装配置高可用集群实践 Nginx高可用集群简介今天...
kubeadm部署kubernetes 1.11.0集群
kubeadm安装高可用集群 kubeadm作为kubernetes主推的部署工具。初始化系统所有机器都需要初...
实用教程丨使用K3s和MySQL运行Rancher 2.4
本文转自Rancher Labs 简介本文将介绍在高可用K3s Kubernetes集群上安装Rancher...
Kubernetes 高可用集群
概述我们部署的 Kubernetes 是集群模式，但在实际生产中我们需要部署高可用集群，部署 Kubern...
sealos2.0使用教程，最简单kubernetesHA方案
kubernetes集群三步安装概述本文教你如何用一条命令构建k8s高可用集群且不依赖haproxy和keep...