K8s Node 两个组件:
- kubelet
- kube-proxy
每个节点上都运行一个 kubelet 服务进程,默认监听 10250 端口,接收并执行 master 发来的指令,管理 Pod 及 Pod 中的容器。每个 kubelet 进程会在 API Server 上注册节点自身信息,定期向 master 节点汇报节点的资源使用情况,并通过 cAdvisor 监控节点和容器的资源。
每台机器上都运行一个 kube-proxy 服务,它监听 API server 中 service 和 endpoint 的变化情况,并通过 iptables、ipvs 等来为服务配置负载均衡(仅支持 TCP 和 UDP)。
依赖的组件
- Docker
- flannel
工作原理
kubelet.pngkube-proxy.png
安装步骤
- 启用ipvs
- 下载文件
- 制作证书
- 部署kubelet组件
- 部署kube-proxy组件
- 测试示例(Nginx)
启用ipvs
#在所有的Kubernetes节点node1和node2上执行以下脚本:
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in \${ipvs_modules}; do
/sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
if [ $? -eq 0 ]; then
/sbin/modprobe \${kernel_module}
fi
done
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
下载文件,解压缩
wget https://dl.k8s.io/v1.13.6/kubernetes-node-linux-amd64.tar.gz
[root@k8s ~]# tar zxf kubernetes-node-linux-amd64.tar.gz
[root@k8s ~]# cd kubernetes/node/bin/
[root@k8s bin]# cp kube-proxy kubelet /opt/kubernetes/bin/
[root@k8s bin]# cp kubectl /usr/bin/
制作证书
从master上获取ca.pem,ca-key.pem,server.pem.server-key.pem
[root@k8s ssl]# scp *.pem root@10.0.52.14:/opt/kubernetes/ssl/
ca-key.pem 100% 1675 3.2MB/s 00:00
ca.pem 100% 1359 3.7MB/s 00:00
server-key.pem 100% 1679 3.8MB/s 00:00
server.pem 100% 1643 4.4MB/s 00:00
[root@k8s ssl]#
制作kube-proxy csr文件,并生成kube-proxy.pem,kube-proxy-key.pem证书
可以在master上生成之后scp到node节点,也可以直接在node节点生成,但node节点需要安装cfssl
cfssl安装,可以参考我之前的博客etcd 集群部署
cat << EOF | tee /opt/kubernetes/ssl/kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
部署kubelet组件
1. 创建bootstrap.kubeconfig,kube-proxy.kubeconfig文件 通过脚本实现
参数说明
BOOTSTRAP_TOKEN : 部署master时生成的token.csv 第一个参数
APISERVER : master 地址
SSL_DIR : 放置证书文件的地址
[root@k8s ~]# cat environment.sh
#!/bin/bash
BOOTSTRAP_TOKEN=7d558bb3a5206cf78f881de7d7b82ca6
APISERVER=10.0.52.13
SSL_DIR=/opt/kubernetes/ssl
# 创建kubelet bootstrapping kubeconfig
export KUBE_APISERVER="https://$APISERVER:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
# 创建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=$SSL_DIR/kube-proxy.pem \
--client-key=$SSL_DIR/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
在 /opt/kubernetes/cfg/执行environment.sh ,生成bootstrap.kubeconfig,kube-proxy.kubeconfig两个配置文件
[root@k8s ~]# cd /opt/kubernetes/cfg/
[root@k8s cfg]# sh environment.sh
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
[root@k8s cfg]# ls
bootstrap.kubeconfig environment.sh flanneld kube-proxy.kubeconfig
[root@k8s cfg]#
2. 创建kubelet.config文件
参数说明:
address : 当前节点的ip地址
clusterDNS: clusterDNS 必须于在apiserver 设置的service-cluster-ip-range=10.0.0.0/24相对应
cat << EOF | tee /opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 10.0.52.14
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- "10.0.0.2"
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
EOF
2. 创建kubelet文件
参数说明
address : 本node的ip地址
hostname-override: 本node的hostname地址,建议直接使用本机IP,也可以不用配置
pod-infra-container-image: 启动容器镜像,原来时k8s.gcr.io/pause-amd64:3.0,替换成registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
cat << EOF | tee /opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \\
--v=4 \\
--address=10.0.52.14 \\
--hostname-override=10.0.52.14 \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet.config \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF
3. 创建kubelet systemd文件
cat << EOF | tee /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
4. 在master上操作,将kubelet-bootstrap用户绑定到系统集群角色
注意这个默认连接127.0.0.1:8080端口,所以在master上操作
如果创建失败,建议先删除之后再创建,删除操作如下:
kubectl delete clusterrolebinding kubelet-bootstrap
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
4. 启动服务
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
[root@k8s cfg]# systemctl daemon-reload
[root@k8s cfg]# systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@k8s cfg]# systemctl start kubelet
[root@k8s cfg]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-05-29 15:24:30 CST; 12s ago
Main PID: 14428 (kubelet)
Memory: 17.3M
CGroup: /system.slice/kubelet.service
└─14428 /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --address=10.0.52.14 --hostname-override=10.0....
May 29 15:24:30 k8s.node1 kubelet[14428]: I0529 15:24:30.774250 14428 feature_gate.go:206] feature gates: &{map[]}
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.224772 14428 server.go:825] Using self-signed cert (/o....key)
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.228968 14428 mount_linux.go:180] Detected OS with systemd
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.229037 14428 server.go:407] Version: v1.13.6
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.229154 14428 feature_gate.go:206] feature gates: &{map[]}
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.229220 14428 feature_gate.go:206] feature gates: &{map[]}
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.229316 14428 plugins.go:103] No cloud provider specified.
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.229330 14428 server.go:523] No cloud provider specifie...e: ""
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.229355 14428 bootstrap.go:65] Using bootstrap kubeconf... file
May 29 15:24:31 k8s.node1 kubelet[14428]: I0529 15:24:31.231105 14428 bootstrap.go:96] No valid private key and...w one
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s cfg]#
5. Master接受kubelet CSR请求
可以手动或自动 approve CSR 请求。如下是手动 approve CSR请求操作方法 查看CSR列表
csr 是certificatesigningrequests 的缩写,查看缩写的方法为:kubectl api-resources
下面操作都是再master上操作
[root@k8s ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-vRVNCJT48G9I8g2k9A7vVDkCj1cmFpdrxXQnCLhrwe0 119s kubelet-bootstrap Pending
接受node
[root@k8s ~]# kubectl certificate approve node-csr-vRVNCJT48G9I8g2k9A7vVDkCj1cmFpdrxXQnCLhrwe0
certificatesigningrequest.certificates.k8s.io/node-csr-vRVNCJT48G9I8g2k9A7vVDkCj1cmFpdrxXQnCLhrwe0 approved
再查看CSR
[root@k8s ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-vRVNCJT48G9I8g2k9A7vVDkCj1cmFpdrxXQnCLhrwe0 77m kubelet-bootstrap Approved,Issued
这时候发现CONDITION 变更为:Approved,Issued
查看集群状态
[root@k8s ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.0.52.14 Ready <none> 2m57s v1.13.6
这时候10.0.52.14这个节点已经加入到集群当中了
node 上查看
这时候10.0.52.14这个节点的/opt/kubernetes/ssl自动生成了kubelet-client的证书
[root@k8s ssl]# ls
ca-key.pem kubelet-client-2019-05-29-16-41-26.pem kubelet.crt kube-proxy-key.pem server-key.pem
ca.pem kubelet-client-current.pem kubelet.key kube-proxy.pem server.pem
注意期间要是kubelet,kube-proxy配置错误,比如监听IP或者hostname错误导致node not found,需要删除kubelet-client证书,重启kubelet服务,重启认证csr即可
部署kube-proxy组件
1. 创建 kube-proxy 配置文件
参数说明:
hostname-override: 本机ip地址
cluster-cidr: 必须于在apiserver 设置的service-cluster-ip-range=10.0.0.0/24相对应
proxy-mode:代理模式,有ipvs,iptables等设置,我们选择ipvs
cat << EOF | tee /opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=10.0.52.14 \\
--cluster-cidr=10.0.0.0/24 \\
--proxy-mode=ipvs \\
--ipvs-scheduler=wrr \\
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
EOF
2. 创建kube-proxy systemd文件
cat << EOF | tee /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
3. 启动服务
[root@k8s cfg]# systemctl daemon-reload
[root@k8s cfg]# systemctl enable kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@k8s cfg]# systemctl start kube-proxy
[root@k8s cfg]# systemctl status kube-proxy
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-05-29 16:58:30 CST; 7s ago
Main PID: 21608 (kube-proxy)
Memory: 12.8M
CGroup: /system.slice/kube-proxy.service
‣ 21608 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=10.0.52.14 --cluster-ci...
May 29 16:58:30 k8s.node1 kube-proxy[21608]: I0529 16:58:30.914320 21608 iptables.go:391] running iptables-rest...ters]
May 29 16:58:30 k8s.node1 kube-proxy[21608]: I0529 16:58:30.916289 21608 proxier.go:728] syncProxyRules took 45.60983ms
May 29 16:58:31 k8s.node1 kube-proxy[21608]: I0529 16:58:31.268300 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:31 k8s.node1 kube-proxy[21608]: I0529 16:58:31.269856 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:33 k8s.node1 kube-proxy[21608]: I0529 16:58:33.274659 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:33 k8s.node1 kube-proxy[21608]: I0529 16:58:33.277684 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:35 k8s.node1 kube-proxy[21608]: I0529 16:58:35.280655 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:35 k8s.node1 kube-proxy[21608]: I0529 16:58:35.284373 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:37 k8s.node1 kube-proxy[21608]: I0529 16:58:37.287116 21608 config.go:141] Calling handler.OnEndpo...pdate
May 29 16:58:37 k8s.node1 kube-proxy[21608]: I0529 16:58:37.291050 21608 config.go:141] Calling handler.OnEndpo...pdate
Hint: Some lines were ellipsized, use -l to show in full.
4. 同样操作部署node 10.0.52.6
- 将对应的文件复制到10.0.52.6上
# 将systemd文件copy到10.0.52.6
[root@k8s cfg]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@10.0.52.6:/usr/lib/systemd/system/
root@10.0.52.6's password:
kubelet.service 100% 264 512.0KB/s 00:00
kube-proxy.service 100% 231 565.5KB/s 00:00
# 将config文件copy到10.0.52.6
[root@k8s cfg]# scp bootstrap.kubeconfig kubelet.config kube-proxy kubelet kubelet.kubeconfig kube-proxy.kubeconfig root@10.0.52.6:/opt/kubernetes/cfg/
root@10.0.52.6's password:
bootstrap.kubeconfig 100% 2164 4.0MB/s 00:00
kubelet.config 100% 264 713.2KB/s 00:00
kube-proxy 100% 184 495.6KB/s 00:00
kubelet 100% 408 1.1MB/s 00:00
kubelet.kubeconfig 100% 2293 5.6MB/s 00:00
kube-proxy.kubeconfig 100% 6266 13.3MB/s 00:00
# 将ssl文件copy到10.0.52.6
[root@k8s ssl]# scp ca-key.pem kube-proxy-key.pem server-key.pem ca.pem kube-proxy.pem server.pem root@10.0.52.6:/opt/kubernetes/ssl/
root@10.0.52.6's password:
ca-key.pem 100% 1675 2.7MB/s 00:00
kube-proxy-key.pem 100% 1675 3.5MB/s 00:00
server-key.pem 100% 1679 4.0MB/s 00:00
ca.pem 100% 1359 3.4MB/s 00:00
kube-proxy.pem 100% 1403 3.7MB/s 00:00
server.pem 100% 1643 4.4MB/s 00:00
#将二进制文件copy到10.0.52.6
[root@k8s bin]# scp kubelet kube-proxy root@10.0.52.6:/opt/kubernetes/bin/
root@10.0.52.6's password:
kubelet 100% 108MB 107.9MB/s 00:01
kube-proxy 100% 33MB 122.0MB/s 00:00
- 将修改配置文件
[root@k8s cfg]# grep 10.0.52.14 *
kubelet:--address=10.0.52.14 \
kubelet:--hostname-override=10.0.52.14 \
kubelet.config:address: 10.0.52.14
kube-proxy:--hostname-override=10.0.52.14 \
#将上述4处地方修改为10.0.52.6即可
- 启动
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl enable kube-proxy
systemctl start kube-proxy
- Master接受kubelet CSR请求
[root@k8s ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-EXa7F9nMgEwPdURbGErwkcjIH_opG_wVhvQ4p4Hru3g 2m12s kubelet-bootstrap Pending
node-csr-vRVNCJT48G9I8g2k9A7vVDkCj1cmFpdrxXQnCLhrwe0 112m kubelet-bootstrap Approved,Issued
[root@k8s ~]# kubectl certificate approve node-csr-EXa7F9nMgEwPdURbGErwkcjIH_opG_wVhvQ4p4Hru3g
certificatesigningrequest.certificates.k8s.io/node-csr-EXa7F9nMgEwPdURbGErwkcjIH_opG_wVhvQ4p4Hru3g approved
[root@k8s ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-EXa7F9nMgEwPdURbGErwkcjIH_opG_wVhvQ4p4Hru3g 2m37s kubelet-bootstrap Approved,Issued
node-csr-vRVNCJT48G9I8g2k9A7vVDkCj1cmFpdrxXQnCLhrwe0 112m kubelet-bootstrap Approved,Issued
[root@k8s ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.0.52.14 Ready <none> 35m v1.13.6
10.0.52.6 Ready <none> 13s v1.13.6
[root@k8s ~]#
至此,Node部署成功。若要扩容node,就按照部署到10.0.52.6方式操作即可!
测试示例(Nginx)
- 创建nginx.yaml
[root@k8s ~]# cat nginx.yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.10
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
labels:
run: nginx
name: nginx
namespace: default
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
selector:
app: nginx
[root@k8s ~]#
- 创建deployment,svc资源
[root@k8s ~]# kubectl create -f nginx.yaml
deployment.apps/nginx-deployment created
service/nginx created
- 查看资源
[root@k8s ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-7544fc9954-6nxvd 1/1 Running 0 9m3s
pod/nginx-deployment-7544fc9954-hhnhw 1/1 Running 0 9m3s
pod/nginx-deployment-7544fc9954-qlvzs 1/1 Running 0 9m3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 47h
service/nginx NodePort 10.0.0.207 <none> 80:37085/TCP 9m3s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 9m3s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-7544fc9954 3 3 3 9m3s
- 测试访问
node节点因为安装有flannel所以可以直接通过内部port端口访问,<cluster ip>:port 是提供给集群内部客户访问service的入口。
外部访问需要nodePort暴露的端口访问,<nodeIP>:nodePort 是提供给集群外部客户访问service的入口。
如果没有指定,会自动从30000~50000间选择一个端口。
node节点访问:
curl 10.0.0.207:80
外部访问:
http://NodeIP:37085
如本示例访问http://10.0.52.14:37085/, 出现我们熟悉的欢迎页面,证明集群正常工作
nginx欢迎页面
如果出现如下权限错误:
[root@k8s ~]# kubectl logs nginx-deployment-7544fc9954-6nxvd -f
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-deployment-7544fc9954-6nxvd)
解决办法,给用户system:anonymous绑定到cluster-admin角色当中,可以有其他角色,自行百度。
kubectl create clusterrolebinding cluster-system-anonymous \
--clusterrole=cluster-admin \
--user=system:anonymous
网友评论