Link Start......
很早之前我就已经把K8s搭建在了我们的云服务器上面了，因为各种原因较为拖沓导致没有及时完善，接下来就是一步一步完善，打造自己的K8s集群了。目前是单master单可用区，基本部署实验环境。后面会换成同城双活，高可用的多master、etcd集群。
- 那么今天想要在K8s搭建高可用的dockerHarbor需要解决的问题是最基本的问题：容量不能伸缩。
如何解决自建K8s集群 docker目录的磁盘伸缩的问题？
[root@master1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 7.7M 0 rom
vda 253:0 0 50G 0 disk
└─vda1 253:1 0 50G 0 part /
大家可以看到我们目前的磁盘是云服务器购买的系统盘，目前我已经在上面搭建好了K8s集群。之前没有考虑好所以直接就部署上去了，当然我们今天需要对docker的/var/lib/docker进行调整成可弹性伸缩的一个方案。
- 解决方案：
-
云上自然采用云上的解决方案来进行处理：高性能云硬盘（腾讯云高性能云盘具有三副本的高可靠性的存储，高可用性高达9个9）、账号资源使用虽然是免费的但是因为要节省团队成本我这里就不采用SSD云硬盘了。
并采用逻辑卷的方式方便以后容量的提升。
前往控制台购买数据盘进行挂载。（）在购买云硬盘时并同时做好定时快照的策略，来保证数据的安全性。当数据被意外删除的时候还可以进行一个快速恢复到上一次快照的数据，腾讯云快照具有增量备份的特性，在第一次快照完成后，第二次快照只记录发生变化的数据，故数据安全和费用可以寻找到一个均衡点来配置。并且云硬盘具有可用区的特性，所以大家在购买的时候不要选错可用区了哦）
image.png
image.png
这边就不在做其他实例的挂载演示了。
- 本地IDC方案：可以整合高可用的分布式存储例如ceph,用不起分布式的可以配合RAID做集中式存储NFS也是一样的，最后结合系统层面的LVM也可以实现弹性存储嘛。
温馨提示：我们设计生产环境之前应该与架构师多沟通交流，设计出一套合理的方案，高可用、高扩展、高安全的架构，不应该和我一样搞了之后再来做头痛治头脚痛治脚。
- 接着我们应该去CVM里面去进行挂载，组逻辑卷。
[root@master1 docker]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vdb 253:16 0 100G 0 disk
sr0 11:0 1 7.7M 0 rom
vda 253:0 0 50G 0 disk
└─vda1 253:1 0 50G 0 part /
- 可以看到第二块磁盘/vdb已经出来咯（v=使用的是KVM的virtio技术所以是虚拟设备，d：disk，b:第二块设备）100G，我们现在开始挂载。
[root@master1 /]# vgcreate k8s_data_lvm /dev/vdb
Physical volume "/dev/vdb" successfully created.
Volume group "k8s_data_lvm" successfully created
[root@master1 /]# vgs
VG #PV #LV #SN Attr VSize VFree
k8s_data_lvm 1 0 0 wz--n- <100.00g <100.00g
[root@master1 /]# lvcreate -L 99.996G -n k8s_data k8s_data_lvm
Rounding up size to full physical extent <100.00 GiB
Logical volume "k8s_data" created.
为什么是99.996G请自行百度，可有意思了。
- 进行格式化开机挂载到/data目录下。（xfs、ext4文件系统对比见谷歌）
[root@master1 /]# mkfs.xfs /dev/mapper/k8s_data_lvm-k8s_data
meta-data=/dev/mapper/k8s_data_lvm-k8s_data isize=512 agcount=4, agsize=6553344 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=26213376, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=12799, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@master1 /]# mkdir /etc/rc.start
[root@master1 /]# cat >> /etc/rc.start/mount_data.sh << EOF
#!/bin/bash
> ls /dev/vdb
> if [ $? -eq 0 ]
> then
> mount /dev/mapper/k8s_data_lvm-k8s_data /data/
> exit
> fi
> # 可以发送邮件啥的告警下
> EOF
# 可以发送邮件啥的告警下
[root@master1 /]# echo '/bin/bash /etc/rc.start/mount_data.sh' >> /etc/rc.local
如果是本地第一次建的话一定要记得给rc.local权限并确认启动rc-local服务哦。
开始迁移docker数据，只有暂时停止docker和kubelet服务。
[root@master1 ~]# df -lh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 24K 2.0G 1% /dev/shm
tmpfs 2.0G 940K 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/vda1 50G 5.6G 42G 12% /
/dev/mapper/k8s_data_lvm-k8s_data 100G 33M 100G 1% /data
tmpfs 394M 0 394M 0% /run/user/0
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/deac31b6-c472-4ce3-b809-79a5d01392ef/volumes/kubernetes.io~secret/coredns-token-5qmpb
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/bec8e176-a5f4-4cc1-98eb-2ad704381264/volumes/kubernetes.io~secret/calico-kube-controllers-token-mqnnf
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/36657799-d311-4a3d-bf00-7b2386eb1e63/volumes/kubernetes.io~secret/coredns-token-5qmpb
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/9e1a1071-cc21-4d47-b269-d8960b928aaa/volumes/kubernetes.io~secret/kube-proxy-token-gctwn
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/9c5758d5-37de-42ea-b00d-435af3a561fb/volumes/kubernetes.io~secret/default-token-hbhj5
[root@master1 ~]# df -lh | grep '/var/lib'
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/deac31b6-c472-4ce3-b809-79a5d01392ef/volumes/kubernetes.io~secret/coredns-token-5qmpb
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/bec8e176-a5f4-4cc1-98eb-2ad704381264/volumes/kubernetes.io~secret/calico-kube-controllers-token-mqnnf
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/36657799-d311-4a3d-bf00-7b2386eb1e63/volumes/kubernetes.io~secret/coredns-token-5qmpb
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/9e1a1071-cc21-4d47-b269-d8960b928aaa/volumes/kubernetes.io~secret/kube-proxy-token-gctwn
tmpfs 2.0G 12K 2.0G 1% /var/lib/kubelet/pods/9c5758d5-37de-42ea-b00d-435af3a561fb/volumes/kubernetes.io~secret/default-token-hbhj5
[root@master1 ~]# for i in $(seq `df -lh | grep '/var/lib' | wc -l`); do umount $(df -lh | grep '/var/lib' | awk -F ' ' '{print $NF}' | head -1 ) ; done
[root@master1 ~]# df -lh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 24K 2.0G 1% /dev/shm
tmpfs 2.0G 936K 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/vda1 50G 5.7G 42G 12% /
/dev/mapper/k8s_data_lvm-k8s_data 100G 33M 100G 1% /data
tmpfs 394M 0 394M 0% /run/user/0
- 开始转移数据到/data路径下
[root@master1 ~]# systemctl stop docker
[root@master1 ~]# systemctl stop kubelet
[root@master1 ~]# mv /var/lib/docker /data/
[root@master1 ~]# mv /var/lib/kubelet /data/
[root@master1 ~]# ln -sv /data/docker/ /var/lib/docker/
[root@master1 ~]# ln -sv /data/kubelet/ /var/lib/kubelet/
[root@master1 ~]# systemctl start docker
[root@master1 ~]# systemctl start kubelet
[root@master1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-6dd8bc586b-5whvf 1/1 Running 1 4m10s 10.100.166.151 node1 <none> <none>
nginx-deployment-6dd8bc586b-67mkj 1/1 Running 0 4m10s 10.100.104.3 node2 <none> <none>
nginx-deployment-6dd8bc586b-ggcsw 1/1 Running 1 4m10s 10.100.166.150 node1 <none> <none>
nginx-deployment-6dd8bc586b-hg82j 1/1 Running 1 4m10s 10.100.166.152 node1 <none> <none>
nginx-deployment-6dd8bc586b-hj2n7 1/1 Running 0 4m10s 10.100.104.5 node2 <none> <none>
nginx-deployment-6dd8bc586b-kwc45 1/1 Running 1 4m10s 10.100.166.149 node1 <none> <none>
nginx-deployment-6dd8bc586b-l797z 1/1 Running 0 4m10s 10.100.104.6 node2 <none> <none>
nginx-deployment-6dd8bc586b-rtmj8 1/1 Running 1 4m10s 10.100.166.148 node1 <none> <none>
nginx-deployment-6dd8bc586b-t7dbr 1/1 Running 0 4m10s 10.100.104.2 node2 <none> <none>
nginx-deployment-6dd8bc586b-xsbvx 1/1 Running 0 4m10s 10.100.104.4 node2 <none> <none>
[root@node1 docker]# df -lh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 24K 2.0G 1% /dev/shm
tmpfs 2.0G 2.1M 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/vda1 50G 3.8G 44G 8% /
tmpfs 394M 0 394M 0% /run/user/0
/dev/mapper/k8s_data_lvm-k8s_data 100G 937M 100G 1% /data
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/dd2e0a4a-a9e9-4a60-abe7-b8acc612f8d1/volumes/kubernetes.io~secret/calico-node-token-r6kvt
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/4aaee83a-161f-4bb8-b772-e7d6b30e557f/volumes/kubernetes.io~secret/kube-proxy-token-gctwn
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/10fb5a8c-cd76-4951-ad60-85df3b7c9770/volumes/kubernetes.io~secret/nginx-ingress-token-vv5lv
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/23eab3dd-9487-4029-8dd0-e17f1e9584dc/volumes/kubernetes.io~secret/metrics-server-token-vhdll
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/b4c348f4-7d41-40e9-8a52-6efd4e1c375c/volumes/kubernetes.io~secret/default-token-c8pf2
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/e0f06916-6d57-44a7-a9e8-b18a7f77b412/volumes/kubernetes.io~secret/default-token-c8pf2
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/6c7f8193-7ba3-458e-abab-34fc61e21982/volumes/kubernetes.io~secret/default-token-c8pf2
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/0be5aca8-8eb2-44da-bed3-0d8555575370/volumes/kubernetes.io~secret/default-token-c8pf2
tmpfs 2.0G 12K 2.0G 1% /data/kubelet/pods/8061a7a5-8e45-4786-8fc6-cc6d0a549f74/volumes/kubernetes.io~secret/default-token-c8pf2
overlay 100G 937M 100G 1% /data/docker/overlay2/bf5b1af02cd0e2975d2f44218668d51d78ca0f0fad86cf405a00cbba2a02b767/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/bfa7de5ce250ff5f75dee1e3246f98ee7c16f0339ac82177650401c93aa4435c/merged
shm 64M 0 64M 0% /data/docker/containers/99ca3b88cea443d3ea670fe874734c75d6d66b6b978b307eda00b8faec7b3ab4/mounts/shm
overlay 100G 937M 100G 1% /data/docker/overlay2/7a760aba62137f289145db8fb37520fadf73bbb27a791e208b7a86b41eb3ddb1/merged
shm 64M 0 64M 0% /data/docker/containers/f199f73591b020b2b8ae1b9896a7111460954c2408f34baa8e76307f68cb6bfe/mounts/shm
shm 64M 0 64M 0% /data/docker/containers/f34927f6bba754e163a424a956abaa054d7e4cc4e065cc3f9115e976685fb80c/mounts/shm
overlay 100G 937M 100G 1% /data/docker/overlay2/91c7b134b3082bbe83e1fbe5ca5c9cda783c2d3278078b80a36c282d6a3f8fdd/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/b3553a0a6c783eaae37f31b4e7dd47e8238d63225f56f5f6fc1365c2b1e9406f/merged
shm 64M 0 64M 0% /data/docker/containers/5354d6568a45d55b6001de0afb56ba97b3b3df8231f8cdfb9bddb6e15645bbae/mounts/shm
overlay 100G 937M 100G 1% /data/docker/overlay2/45ca4d3e206e9106fa0929e2b366831e9e9a39dc451e147e1a910cb7ba897eaa/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/5c93cad388e123496b415b5e25348eefb3053cc9953529defd13f8362051ebc9/merged
shm 64M 0 64M 0% /data/docker/containers/bf49c82319f261df85d592ebb41211dd86f613f282c07ec9eec8d04d2807d8e0/mounts/shm
overlay 100G 937M 100G 1% /data/docker/overlay2/be66b8366a997df87a55d0ff37a447acbae5ab392c42f46645b727965dc08aac/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/e821311c8a094f9fb54fd8af6d7553845c53f03b944242badae298e11d5c95a7/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/186727eb4e10bb5f773eeb5da117fe15debb19e748e8a4da19efa214a2c7929e/merged
shm 64M 0 64M 0% /data/docker/containers/4bbe2e8aeba8458d5120d50aba6356ccc8d7a67e1dc1be8bd98cfba994662b15/mounts/shm
shm 64M 0 64M 0% /data/docker/containers/95181ad016d98643c26d845b3cefe2a1523f99af50a94ef271acc4cad7c0cf07/mounts/shm
shm 64M 0 64M 0% /data/docker/containers/e3bf4f7f7402c9966cc08f80f23dfe8d36b90851acf92f16ff2733c5d75e8c84/mounts/shm
overlay 100G 937M 100G 1% /data/docker/overlay2/7bfa811e56b20260fa9ea17900ce745afaf2a5c06689506add0a88b4b19addd8/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/f4d48dcb9b6949deac58c6187bd9be069b84cf6faa9fafdfbf8cacfdce3ab7bf/merged
overlay 100G 937M 100G 1% /data/docker/overlay2/2aa15a5d717fe7caa5037be21b259d4fb2eb85a02b48ae13dcc193a482696f49/merged
- 到此测试成功。
下一篇目标是调整高可用master采用双可用区三master搭建。
网友评论