目录
部署一个三实例 kube-controller-manager 的集群，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。
部署策略：
- 3 节点高可用；
- 关闭非安全端口，在安全端口 10252 接收 https 请求；
- 使用 kubeconfig 访问 apiserver 的安全端口；
- 自动 approve kubelet 证书签名请求 (CSR)，证书过期后自动轮转；
- 各 controller 使用自己的 ServiceAccount 访问 apiserver；
部署软件规划
| IP | 部署软件包 |
|---|---|
| 10.40.61.116 | kube-controller-manager |
| 10.40.58.153 | kube-controller-manager |
| 10.40.58.154 | kube-controller-manager |
01.创建 kube-controller-manager 证书和私钥
创建证书签名请求：
cat > kube-controller-manager-csr.json <<EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "China",
"L": "Beijing",
"O": "Kubernetes",
"OU": "Kubernetes",
"ST": "Beijing"
}
]
}
EOF
说明：
hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。
创建 kube-controller-manager凭证与私钥:
cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
结果将产生以下几个文件：
kube-controller-manager-key.pem
kube-controller-manager.pem
02.创建kubeconfig
kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、 CA 证书和 kube-controller-manager 证书
先确定apiserver对外提供服务的地址
export KUBE_APISERVER="https://apiserver-p001.svc.gxd88.cn:6443"
-
设置集群参数
kubectl config set-cluster kubernetes \ --certificate-authority=ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-controller-manager.kubeconfig -
设置客户端认证参数
kubectl config set-credentials system:kube-controller-manager \ --client-certificate=kube-controller-manager.pem \ --client-key=kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=kube-controller-manager.kubeconfig -
设置上下文参数
kubectl config set-context default \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=kube-controller-manager.kubeconfig -
设置默认上下文
kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig
03.service-accout证书
请使用kube-apiserver文档中生成的service-account证书， 其中kube-apiserver使用公钥， kube-controller-manager使用私钥
04.使用systemd管理kube-controller-manager
tee /etc/systemd/system/kube-controller-manager.service <<-EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/srv/kubernetes/bin/kube-controller-manager \\
--kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \\
--authentication-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \\
--authorization-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \\
--bind-address=0.0.0.0 \\
--pod-eviction-timeout=3m0s \\
--node-eviction-rate=0.1 \\
--secondary-node-eviction-rate=0.1 \\
--large-cluster-size-threshold=1 \\
--allocate-node-cidrs=true \\
--service-cluster-ip-range=10.244.0.0/16 \\
--cluster-cidr=10.243.0.0/16 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/srv/kubernetes/pki/ca.pem \\
--cluster-signing-key-file=/srv/kubernetes/pki/ca-key.pem \\
--service-account-private-key-file=/srv/kubernetes/pki/service-account-key.pem \\
--use-service-account-credentials=true \\
--root-ca-file=/srv/kubernetes/pki/ca.pem \\
--leader-elect=true \\
--v=2 \\
--logtostderr=false \\
--log-dir=/srv/kubernetes/log
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
启动参数的请看参数详解
05.启动/停止 kube-controller-manager
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kube-controller-manager.service
kube-controller-manager can be started and stopped as follows:
sudo systemctl start kube-controller-manager.service
sudo systemctl stop kube-controller-manager.service
06.验证
查看进程是否正常
$ systemctl status kube-controller-manager |grep active
Active: active (running) since 六 2020-04-11 16:11:58 CST; 6 days ago
查看权限:
$ kubectl describe clusterrole system:kube-controller-manager
Name: system:kube-controller-manager
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
secrets [] [] [create delete get update]
serviceaccounts [] [] [create get update]
events [] [] [create patch update]
events.events.k8s.io [] [] [create patch update]
endpoints [] [] [create]
serviceaccounts/token [] [] [create]
tokenreviews.authentication.k8s.io [] [] [create]
subjectaccessreviews.authorization.k8s.io [] [] [create]
leases.coordination.k8s.io [] [] [create]
endpoints [] [kube-controller-manager] [get update]
leases.coordination.k8s.io [] [kube-controller-manager] [get update]
configmaps [] [] [get]
namespaces [] [] [get]
*.* [] [] [list watch]
ClusteRole system:kube-controller-manager 的权限很小，只能创建 secret、serviceaccount 等资源对象，各 controller 的权限分散到 ClusterRole system:controller:XXX 中。
当在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。
kubectl get clusterrole|grep controller
system:controller:attachdetach-controller 2020-04-11T07:37:53Z
system:controller:certificate-controller 2020-04-11T07:37:53Z
system:controller:clusterrole-aggregation-controller 2020-04-11T07:37:53Z
system:controller:cronjob-controller 2020-04-11T07:37:53Z
system:controller:daemon-set-controller 2020-04-11T07:37:53Z
system:controller:deployment-controller 2020-04-11T07:37:53Z
system:controller:disruption-controller 2020-04-11T07:37:53Z
system:controller:endpoint-controller 2020-04-11T07:37:53Z
system:controller:endpointslice-controller 2020-04-11T07:37:53Z
system:controller:expand-controller 2020-04-11T07:37:53Z
system:controller:generic-garbage-collector 2020-04-11T07:37:53Z
system:controller:horizontal-pod-autoscaler 2020-04-11T07:37:53Z
system:controller:job-controller 2020-04-11T07:37:53Z
system:controller:namespace-controller 2020-04-11T07:37:53Z
system:controller:node-controller 2020-04-11T07:37:53Z
system:controller:persistent-volume-binder 2020-04-11T07:37:53Z
system:controller:pod-garbage-collector 2020-04-11T07:37:53Z
system:controller:pv-protection-controller 2020-04-11T07:37:53Z
system:controller:pvc-protection-controller 2020-04-11T07:37:53Z
system:controller:replicaset-controller 2020-04-11T07:37:53Z
system:controller:replication-controller 2020-04-11T07:37:53Z
system:controller:resourcequota-controller 2020-04-11T07:37:53Z
system:controller:route-controller 2020-04-11T07:37:53Z
system:controller:service-account-controller 2020-04-11T07:37:53Z
system:controller:service-controller 2020-04-11T07:37:53Z
system:controller:statefulset-controller 2020-04-11T07:37:53Z
system:controller:ttl-controller 2020-04-11T07:37:53Z
system:kube-controller-manager 2020-04-11T07:37:53Z
查看其中deployment controller
kubectl describe clusterrole system:controller:deployment-co
Name: system:controller:deployment-controller
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
replicasets.apps [] [] [create delete get list patch update watch]
replicasets.extensions [] [] [create delete get list patch update watch]
events [] [] [create patch update]
events.events.k8s.io [] [] [create patch update]
pods [] [] [get list update watch]
deployments.apps [] [] [get list update watch]
deployments.extensions [] [] [get list update watch]
deployments.apps/finalizers [] [] [update]
deployments.apps/status [] [] [update]
deployments.extensions/finalizers [] [] [update]
deployments.extensions/status [] [] [update]
07.参数详解
kube-controller-manager，k8s的大脑，大部分控制器所在，大管家，配置包括:
- 驱逐相关
- 高可用选主相关
- PodIP网段
- …
启动命令
/srv/kubernetes/bin/kube-controller-manager \
--kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \
--pod-eviction-timeout=3m0s \
--node-eviction-rate=0.1 \
--secondary-node-eviction-rate=0.1 \
--large-cluster-size-threshold=1 \
--address=0.0.0.0 \
--allocate-node-cidrs=true \
--service-cluster-ip-range=10.244.0.0/16 \
--cluster-cidr=10.243.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/srv/kubernetes/pki/ca.pem \
--cluster-signing-key-file=/srv/kubernetes/pki/ca-key.pem \
--service-account-private-key-file=/srv/kubernetes/pki/service-account-key.pem \
--use-service-account-credentials=true \
--root-ca-file=/srv/kubernetes/pki/ca.pem \
--leader-elect=true \
--v=2 \
--logtostderr=false \
--log-dir=/srv/kubernetes/log
参数用途说明
--leader-elect=true
开启选举。
利用etcd的强一致性，可以用来组件的选主，kube-controller-manager就是利用这个特性实现的高可用。
高可用条件: kube-controller-manager数量大于等于2即可。
--pod-eviction-timeout=3m0s
驱逐超时，默认5分钟，配置为3分钟，从controller感知到节点挂了开始计时。
--node-eviction-rate=0.1
一级驱逐速率，这里两个0.1相当于每个节点每10秒钟驱逐一个Pod。
Number of nodes per second on which pods are deleted in case of node failure when a zone is healthy (see –unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters.
当开启多可用区时，一级驱逐速率只有在zone健康的时候生效；当非多可用区的时候，zone代表整个集群。
--secondary-node-eviction-rate=0.1
二级驱逐速率，这里两个0.1相当于每个节点每10秒钟驱逐一个Pod。
Number of nodes per second on which pods are deleted in case of node failure when a zone is unhealthy (see –unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. This value is implicitly overridden to 0 if the cluster size is smaller than –large-cluster-size-threshold.
--large-cluster-size-threshold=1
Number of nodes from which NodeController treats the cluster as large for the eviction logic purposes. –secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller.
设置一个"大"集群的阈值，多大为大集群，默认为50，当集群节点数小于该值的时候，二级驱逐速率就是0，不驱逐！
--unhealthy-zone-threshold=0.55
Fraction of Nodes in a zone which needs to be not Ready (minimum 3) for zone to be treated as unhealthy.
Zone被认为不健康的条件，大于55%的节点故障（NodeReady的节点数大于等于3）.
举例:
比如现在ZoneB有20个节点，那么当20 * 0.55 = 11个以上节点挂了后，那么这个Zone就被认为不健康了，此时一级驱逐速率也就不生效了；由于总结点数大于1，那么二级速率有效，所以整个集群会以10s/pod/node进行驱逐。
08.Q&A
Q:
configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
A:
Q:
No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
A:
需要在启动命令中添加如下配置
--authentication-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig
--authorization-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig
网友评论