本文档介绍部署一个，有三个节点高可用 etcd 集群的步骤：
- 下载和分发 etcd 二进制文件；
- 创建 etcd 集群各节点的 x509 证书，用于加密客户端 (如 etcdctl) 与 etcd 集群、etcd 集群之间的数据流；
- 创建 etcd 的 systemd unit 文件，配置服务参数；
- 检查集群工作状态
1）下载和分发 etcd 二进制文件
到 https://github.com/coreos/etcd/releases 页面下载最新版本的发布包：
wget https://github.com/coreos/etcd/releases/download/v3.3.7/etcd-v3.3.7-linux-amd64.tar.gz
tar -zxvf etcd-v3.3.7-linux-amd64.tar.gz
分发二进制文件,推送到所有节点里：
cat > magic12_distribute_etcd.sh << "EOF"
#!/bin/bash
# 分发二进制文件
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp /data/etcd-v3.3.7-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF
2）创建 etcd 证书和私钥
创建证书签名请求：
cat > /data/cert/etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"172.68.96.101",
"172.68.96.102",
"172.68.96.103"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
EOF
- hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表，这里将 etcd 集群的三个节点 IP 都列在其中
生成证书和私钥：
[root@master cert]#
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
-ca-key=/etc/kubernetes/cert/ca-key.pem \
-config=/etc/kubernetes/cert/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
[root@master cert]# ls etcd*
分发生成的证书和私钥到各 etcd 节点：
cat > magic13_distribute_certndoe.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /etc/etcd/cert && chown -R k8s /etc/etcd/cert"
scp /data/cert/etcd*.pem root@${node_ip}:/etc/etcd/cert/
done
EOF
3）创建 etcd 的 systemd unit 模板文件
source /opt/k8s/bin/environment.sh
[root@master template]# vim /data/template/etcd.service.template
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
User=root
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/opt/k8s/bin/etcd \
--data-dir=/var/lib/etcd \
--name=##NODE_NAME## \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem \
--trusted-ca-file=/etc/kubernetes/cert/ca.pem \
--peer-cert-file=/etc/etcd/cert/etcd.pem \
--peer-key-file=/etc/etcd/cert/etcd-key.pem \
--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \
--listen-peer-urls=https://##NODE_IP##:2380 \
--initial-advertise-peer-urls=https://##NODE_IP##:2380 \
--listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=master=https://172.68.96.101:2380,node01=https://172.68.96.102:2380,node02=https://172.68.96.103:2380 \
--initial-cluster-state=new
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
- User：指定以 k8或root （可选）
- WorkingDirectory、--data-dir：指定工作目录和数据目录为 /var/lib/etcd，需在启动服务前创建这个目录；
- --name：指定节点名称，当 –initial-cluster-state 值为 new 时，–name 的参数值必须位于 –initial-cluster 列表中；
- --cert-file、--key-file：etcd server 与 client 通信时使用的证书和私钥；
- --trusted-ca-file：签名 client 证书的 CA 证书，用于验证 client 证书；
- --peer-cert-file、--peer-key-file：etcd 与 peer 通信使用的证书和私钥；
- --peer-trusted-ca-file：签名 peer 证书的 CA 证书，用于验证 peer 证书；
4）将刚刚的文件初始化成节点可用
替换模板文件中的变量，为各节点创建 systemd unit 文件：
cat > magic14_init_NodeCluster.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for (( i=0; i < 110; i++ ))
do
sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" /data/template/etcd.service.template > /data/service/etcd-${NODE_IPS[i]}.service
done
EOF
[root@master service]# bash -x magic.sh
[root@master service]# ls *.service
etcd-172.68.96.101.service etcd-172.68.96.102.service etcd-172.68.96.103.service
- NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；
分发生成的 systemd unit 文件：
cat > magic15_distribute_SystemdUnit.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /var/lib/etcd && chown -R k8s /var/lib/etcd"
scp /data/service/etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
done
EOF
5）启动 etcd 服务
cat > magic16_start_etcd_Service.sh << "EOF"
#!/bin/bash
# 启动 etcd 服务
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl start etcd &"
done
EOF
- etcd 进程首次启动时会等待其它节点的 etcd 加入集群，命令 systemctl start etcd 会卡住一段时间，为正常现象
6）检查启动结果
cat > magic17_check_etcd_Service.sh << "EOF"
#!/bin/bash
# 检查启动结果
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl status etcd|grep Active"
done
EOF
结果如下，则说明正常：
[root@master scriptable]# bash magic17_check_etcd_Service.sh
>>> 172.68.96.101
Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
>>> 172.68.96.102
Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
>>> 172.68.96.103
Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
否则查看日志，确认原因：
journalctl -xu etcd
7）验证服务状态
部署完 etcd 集群后，在任一个etc节点上执行如下命令：
cat > magic18_verification_service.sh << "EOF"
#!/bin/bash
# 验证服务状态
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
--endpoints=https://${node_ip}:2379 \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/etcd/cert/etcd.pem \
--key=/etc/etcd/cert/etcd-key.pem endpoint health
done
EOF
结果如下，则说明正常：
[root@master scriptable]# bash magic18_verification_service.sh
>>> 172.68.96.101
https://172.68.96.101:2379 is healthy: successfully committed proposal: took = 1.103382ms
>>> 172.68.96.102
https://172.68.96.102:2379 is healthy: successfully committed proposal: took = 1.686551ms
>>> 172.68.96.103
https://172.68.96.103:2379 is healthy: successfully committed proposal: took = 1.413926ms
输出均为 healthy 时表示集群服务正常
网友评论