美文网首页
Kubernetes v1.13.5高可用集群安装

Kubernetes v1.13.5高可用集群安装

作者: 济南打工人 | 来源:发表于2019-03-30 07:59 被阅读0次

    基础环境

    系统Ubuntu 16.04

    主机名称 IP 备注
    node01 192.168.175.61 master and etcd
    node02 192.168.175.62 master and etcd
    node03 192.168.175.63 master and etcd
    node04 192.168.175.64 node
    VIP 192.168.175.60

    软件版本：

    软件 版本号
    docker v18.06.1-ce
    kubelet v1.13.5
    kubectl v1.13.5
    kubeadm v1.13.5
    etcd v3.3.5

    环境初始化

    修改主机名

    hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03
hostnamectl set-hostname node04

    配置主机映射

    cat <<EOF > /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.175.61 node01
192.168.175.62 node02
192.168.175.63 node03
192.168.175.64 node04
EOF

    免密码登录

    node01上执行ssh免密码登陆配置

    ssh-keygen  #一路回车即可
ssh-copy-id  node02
ssh-copy-id  node03
ssh-copy-id  node04

    基线配置

    四台主机配置设置内核、K8S源、关闭Swap、配置ntp（配置完后重启一次）

    swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab    #禁用swap分区

modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge

 #https://opsx.alibaba.com/mirror 阿里云地址
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main 
EOF  
apt-get update
apt-get install -y kubeadm=1.13.5-00
apt-get install -y kubelet=1.13.5-00
apt-get install -y kubectl=1.13.5-00

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
 
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

    所有节点配置k8s内核

    cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
EOF

sysctl --system

    创建etcd证书

    (node01上执行即可)

    设置cfssl环境

    wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH

    创建 CA 配置文件(下面配置的IP为etc节点的IP)

    mkdir /root/ssl
cd /root/ssl
cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "8760h"
},
"profiles": {
  "kubernetes-Soulmate": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
    "expiry": "8760h"
  }
}
}
}
EOF

cat >  ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
  "C": "CN",
  "ST": "shanghai",
  "L": "shanghai",
  "O": "k8s",
  "OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.175.61",
    "192.168.175.62",
    "192.168.175.63"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

    node01分发etcd证书到node02、node03上面

    mkdir -p /etc/etcd/ssl  #三台机器上分别执行
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node02:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node03:/etc/etcd/ssl/

    安装配置etcd

    (三主节点）

    安装etcd

    apt install etcd -y

    node01的etcd.service

    cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name node01   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.168.175.61:2380   --listen-peer-urls https://192.168.175.61:2380   --listen-client-urls https://192.168.175.61:2379,http://127.0.0.1:2379   --advertise-client-urls https://192.168.175.61:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster node01=https://192.168.175.61:2380,node02=https://192.168.175.62:2380,node03=https://192.168.175.63:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

    node02的etcd.service

    cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name node02   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.168.175.62:2380   --listen-peer-urls https://192.168.175.62:2380   --listen-client-urls https://192.168.175.62:2379,http://127.0.0.1:2379   --advertise-client-urls https://192.168.175.62:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster node01=https://192.168.175.61:2380,node02=https://192.168.175.62:2380,node03=https://192.168.175.63:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

    node03的etcd.service

    cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name node03   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.168.175.63:2380   --listen-peer-urls https://192.168.175.63:2380   --listen-client-urls https://192.168.175.63:2379,http://127.0.0.1:2379   --advertise-client-urls https://192.168.175.63:2379   --initial-cluster-token etcd-cluster-0 --initial-cluster node01=https://192.168.175.61:2380,node02=https://192.168.175.62:2380,node03=https://192.168.175.63:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

    添加自启动（etc集群最少2个节点才能启动，启动报错看mesages日志）

    systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

    在三个etcd节点执行一下命令检查

    etcdctl --endpoints=https://192.168.175.61:2379,https://192.168.175.62:2379,https://192.168.175.63:2379 \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health

    etcd升级

    （apt安装的版本为v2.2.5，kubernetes v1.10要求版本最低为3.1）

    官网下载最新安装包

    wget https://github.com/coreos/etcd/releases/download/v3.3.5/etcd-v3.3.5-linux-amd64.tar.gz
tar zxf etcd-v3.3.5-linux-amd64.tar.gz
cp etcd-v3.3.5-linux-amd64/etcd /usr/bin/etcd
cp etcd-v3.3.5-linux-amd64/etcdctl /usr/bin/etcdctl

    在/etc/profile文件添加以下一行，重启服务器

    export ETCDCTL_API=3

    重启etcd服务，并产看集群状态

    root@k8s-n2:~/k8s# etcdctl member list
aa76456e260f7bd1, started, node02, https://192.168.175.62:2380, https://192.168.175.62:2379
d12950b45efa96da, started, node03, https://192.168.175.63:2380, https://192.168.175.63:2379
e598ba1c84356928, started, node01, https://192.168.175.61:2380, https://192.168.175.61:2379

    安装docker

    curl -fsSL "https://get.docker.com/" | sh

    docker配置代理

    docker 添加阿里云代理，修改配置文件/lib/systemd/system/docker.service

    ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --registry-mirror=https://ms3cfraz.mirror.aliyuncs.com

    启动docker

    添加开机自启动

    systemctl daemon-reload
systemctl restart docker
systemctl enable docker
systemctl status docker

    配置kubeadm

    所有节点修改kubelet配置文件

    /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

    #添加这一行
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
#添加这一行
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=xxx.xxxx.xxxx/antk8s/pause:3.1"

    所有节点修改完配置文件一定要重新加载配置

    systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet

    初始化集群

    node01、node02、node03添加集群初始配置文件（集群配置文件一样）

    apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://192.168.175.61:2379
  - https://192.168.175.62:2379
  - https://192.168.175.63:2379
  caFile: /etc/etcd/ssl/ca.pem
  certFile: /etc/etcd/ssl/etcd.pem
  keyFile: /etc/etcd/ssl/etcd-key.pem
  dataDir: /var/lib/etcd
networking:
  podSubnet: 10.244.0.0/16
kubernetesVersion: 1.13.5
api:
  advertiseAddress: "192.168.175.61"
token: "b99a00.a144ef80536d4344"
tokenTTL: "0s"
apiServerCertSANs:
- node01
- node02
- node03
- node04
- 192.168.175.61
- 192.168.175.62
- 192.168.175.63
- 192.168.175.64
featureGates:
  CoreDNS: true
imageRepository: "images.xxxxxxx.com/antk8s"

    首先node01初始化集群

    配置文件定义podnetwork是10.244.0.0/16
    kubeadmin init –hlep可以看出，service默认网段是10.96.0.0/12
    /etc/systemd/system/kubelet.service.d/10-kubeadm.conf默认dns地址cluster-dns=10.96.0.10

    kubeadm init --config config.yaml

    这时会报错,kubeadm需要切换到v1.11转换文件

    root@k8s-m1:~# kubeadm init --config config.yaml 
your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha1". Please use kubeadm v1.11 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

    root@k8s-m1:~# apt install kubeadm=1.11.0-00
root@k8s-m1:~# kubeadm config migrate --old-config config.yaml --new-config new.yaml
I0330 02:07:39.075699   10719 feature_gate.go:230] feature gates: &{map[]}

    再次执行初始化

    root@k8s-m1:~# kubeadm init --config new.yaml 
your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha2". Please use kubeadm v1.12 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

    kubeadm安装v1.12进行文件转换

    root@k8s-m1:~# kubeadm config migrate --old-config new.yaml --new-config new1.yaml

    初始化，这时，初始化成功

    root@k8s-m1:~# kubeadm init --config new1.yaml 
[init] Using Kubernetes version: v1.13.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
  ...  ...
  ...  ...
  ...  ...
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.175.61:6443 --token b99a00.a144ef80536d4344 --discovery-token-ca-cert-hash sha2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe7a

    master上面执行如下命令

    mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

    部署flannel网络，只需要在node01执行就行

    kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

    执行命令

    [root@node01 ~]# kubectl   get node
NAME      STATUS    ROLES     AGE       VERSION
node01    Ready     master    31m       v1.13.5

    [root@node01 ~]# kubectl get -n kube-system po 
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66555b45f-grq8d          1/1     Running   0          41m
coredns-66555b45f-hrkcf          1/1     Running   0          41m
kube-apiserver-k8s-m1            1/1     Running   0          40m
kube-controller-manager-k8s-m1   1/1     Running   0          40m
kube-flannel-ds-amd64-gjgpt      1/1     Running   0          8s
kube-proxy-pvkjm                 1/1     Running   0          41m
kube-scheduler-k8s-m1            1/1     Running   0          40m

    复制相关文件到其他master上

    scp  /etc/kubernetes/pki/* node02:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/* node03:/etc/kubernetes/pki/
scp admin.conf node02:/etc/kubernetes/
scp admin.conf node03:/etc/kubernetes/

    两个master执行以下操作，安装master需添参数--experimental-control-plane

    root@k8s-n1:/etc/kubernetes# kubeadm join 192.168.175.61:6443 --token b99asssssssssss36d4344 --discovery-token-ca-cert-hash sha256:fffffffffffffffffffffffffffffffffffffffffffffxxxxxxxxxxxxxxxx52cb0d9 --experimental-control-plane
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.175.61:6443"
  ... ...
  ... ...
[mark-control-plane] Marking the node k8s-n1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

    查看集群

    root@k8s-m1:~# kubectl get no
NAME     STATUS   ROLES    AGE     VERSION
k8s-m1   Ready    master   17m     v1.13.5
k8s-n1   Ready    master   12m     v1.13.5
k8s-n2   Ready    master   11m     v1.13.5
k8s-n3   Ready    <none>   3m28s   v1.13.5

    root@k8s-m1:~# kubectl get -n kube-system po 
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66555b45f-4rws8          1/1     Running   0          16m
coredns-66555b45f-dsf2t          1/1     Running   0          16m
kube-apiserver-k8s-m1            1/1     Running   0          15m
kube-apiserver-k8s-n1            1/1     Running   0          12m
kube-apiserver-k8s-n2            1/1     Running   0          11m
kube-controller-manager-k8s-m1   1/1     Running   0          15m
kube-controller-manager-k8s-n1   1/1     Running   0          12m
kube-controller-manager-k8s-n2   1/1     Running   0          11m
kube-flannel-ds-amd64-856gq      1/1     Running   0          12m
kube-flannel-ds-amd64-8f5mj      1/1     Running   0          11m
kube-flannel-ds-amd64-f4pj8      1/1     Running   0          16m
kube-flannel-ds-amd64-l62n6      1/1     Running   0          3m34s
kube-proxy-dj2w5                 1/1     Running   0          3m34s
kube-proxy-dz6cf                 1/1     Running   0          12m
kube-proxy-pqtg9                 1/1     Running   0          11m
kube-proxy-vt5dv                 1/1     Running   0          16m
kube-scheduler-k8s-m1            1/1     Running   0          16m
kube-scheduler-k8s-n1            1/1     Running   0          12m
kube-scheduler-k8s-n2            1/1     Running   0          11m

    至此，集群安装完毕。

    相关文章

      网友评论

          本文标题：Kubernetes v1.13.5高可用集群安装

          本文链接：https://www.haomeiwen.com/subject/znwdbqtx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|Kubernetes v1.13.5高可用集群安装|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！