背景

kubeadm部署的k8s集群etcd出现no space报错,导致不可写,集群不可用

解决

永久方案

自动compact

etcd启动参数添加

auto-compaction-retention=1
auto-compaction-mode=periodic
quota-backend-bytes=8589934592(合适的大小)

定期defrag

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key  --endpoints=:2379 defrag

临时方案

登录etcd节点

查看当前revision

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=:2379 endpoint status --write-out='json'

compact({revision}替换为上一步获取的revision)

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=:2379 compact {revision}

执行defrag

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key  --endpoints=:2379 defrag

解除告警

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key  --endpoints=:2379 alarm disarm