CentOS 安装 Kubernetes v1.13

作者: w也不知道 | 来源:发表于2018-12-26 16:15 被阅读0次

CentOS 安装 Kubernetes v1.13
CentOS kubernetes单机部署
开始创建 Kubernetes 主机
[K8S]Kubernetes环境准备
kubernetes1.4 安装
centOS7.2使用yum安装kubernetes
Kubernetes中的Configmap和Secret
kubeadm 安装kubernetes v1.10
[k8s]基于Centos7安装kubernetes1.13.2
kubeadm v1.12.3安装

一、系统初始化配置 ( 操作系统为 CentOS 7 )

maser: 192.168.4.11
node1: 192.168.4.25
node2: 192.168.4.26

关闭防火墙，selinux

systemctl stop firewalld && systemctl disable firewalld
setenforce 0
vi /etc/selinux/config
SELINUX=disabled

关闭swap

swapoff -a
# 也可修改 /etc/fstab 文件取消挂载swap文件系统

设置内核参数

cat << EOF | tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

配置阿里云的 docker-ce 源

cd /etc/yum.repos.d && \
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装 docker-ce

yum install docker-ce -y

启动并配置 docker

systemctl start docker
cat << EOF | tee /etc/docker/daemon.json
{"registry-mirrors": ["https://registry.docker-cn.com"]}
EOF
systemctl restart docker
systemctl enable docker

安装 ipvsadm

yum install ipvsadm -y

二、安装 cfssl 并生成 kubernetes 使用的证书

安装

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

配置 etcd 和 kubernetes 的所需要证书

mkdir /opt/{etcd,kubernetes}/{bin,cfg,ssl}

创建 etcd 证书配置文件

cat << EOF | tee /opt/etcd/ssl/ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

创建 etcd ca 配置文件

cat << EOF | tee /opt/etcd/ssl/ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "HZ",
            "ST": "HZ"
        }
    ]
}
EOF

创建 etcd server 证书配置文件

cat << EOF | tee /opt/etcd/ssl/server-csr.json
{
    "CN": "etcd",
    "hosts": [
    "192.168.4.11",
    "192.168.4.25",
    "192.168.4.26"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "HZ",
            "ST": "HZ"
        }
    ]
}
EOF

生成 etcd 证书

cd /opt/etcd/ssl
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

创建 kubernetes CA 证书

cat << EOF | tee /opt/kubernetes/ssl/ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

################################################

cat << EOF | tee /opt/kubernetes/ssl/ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "HZ",
            "ST": "HZ",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

###############################################

cd /opt/kubernetes/ssl && cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

生成 API_SERVER 证书

cat << EOF | tee /opt/kubernetes/ssl/server-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.4.11",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "HZ",
            "ST": "HZ",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

##################################################

cd /opt/kubernetes/ssl && \
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

创建 kubernetes proxy 证书

cat << EOF | tee /opt/kubernetes/ssl/kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "HZ",
      "ST": "HZ",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

##################################################

cd /opt/kubernetes/ssl && \
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

创建 admin 管理证书

cat << EOF | tee /opt/kubernetes/ssl/admin-csr.json
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "HZ",
      "L": "HZ",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

######################################################

cd /opt/kubernetes/ssl && \
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin

三、配置 etcd 服务

获取 etcd 软件包

wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
tar -xf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64 && mv etcd etcdctl /opt/etcd/bin
ln -s /opt/etcd/bin/etcdctl /usr/local/bin

配置 etcd 服务主配置文件

cat << EOF | tee /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.4.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.4.11:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.4.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.4.11:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.4.11:2380,etcd02=https://192.168.4.25:2380,etcd03=https://192.168.4.26:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

配置 etcd 系统服务文件

cat << EOF | tee /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

node1 和 node2 节点 etcd 配置

scp -r /opt node1:/
scp -r /opt node2:/
scp -r /usr/lib/systemd/system/etcd.service node1:/usr/lib/systemd/system/etcd.service
scp -r /usr/lib/systemd/system/etcd.service node2:/usr/lib/systemd/system/etcd.service

node1 etcd 配置文件

$ cat /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.4.25:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.4.25:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.4.25:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.4.25:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.4.11:2380,etcd02=https://192.168.4.25:2380,etcd03=https://192.168.4.26:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node2 etcd 配置文件

cat /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.4.26:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.4.26:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.4.26:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.4.26:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.4.11:2380,etcd02=https://192.168.4.25:2380,etcd03=https://192.168.4.26:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

在 master 节点上执行以下命令查看 etcd 节点的运行情况

etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.4.11:2379,\
https://192.168.4.25:2379,\
https://192.168.4.26:2379" cluster-health

# 出现以下内容说明所有节点正常运行
member 15a4c0bca860fd43 is healthy: got healthy result from https://192.168.4.25:2379
member a4b193f4cac482a9 is healthy: got healthy result from https://192.168.4.26:2379
member addafc82e0a8cc7f is healthy: got healthy result from https://192.168.4.11:2379
cluster is healthy

三、配置 flannel

1.获取 flannel

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-arm64.tar.gz
tar -xf flannel-v0.10.0-linux-arm64.tar.gz
mv flanneld mk-docker-opts.sh /opt/kubernetes/bin/

配置 flannel

cat << EOF | tee /opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS='etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.4.11:2379,https://192.168.4.25:2379,https://192.168.4.26:2379"'

创建 flannel 的 systemctl unit 文件

cat << EOF | tee /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

修改 docker 的 systemclt unit 文件，配置 docker 启动指定子网网段

# vi /usr/lib/systemd/system/docker.service
# 主要改以下两行
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS

在 etcd 中配置一个子网网段

etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.4.11:2379,\
https://192.168.4.25:2379,\
https://192.168.4.26:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type":"vxlan"}}'

重新启动

systemctl daemon-reload
systemctl start flanneld
systemctl enable flanneld
systemctl restart docker

复制文件到 node 节点

scp -r /opt/kubernetes node1:/opt/
scp -r /opt/kubernetes node2:/opt/
scp /usr/lib/systemd/system/docker.service node1:/usr/lib/systemd/system/docker.service
scp /usr/lib/systemd/system/docker.service node2:/usr/lib/systemd/system/docker.service
scp /usr/lib/systemd/system/flanneld.service node1:/usr/lib/systemd/system/flanneld.service
scp /usr/lib/systemd/system/flanneld.service node2:/usr/lib/systemd/system/flanneld.service

查看是否生效

ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.4.11/24 brd 192.168.4.255 scope global eth0
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    inet 172.18.88.1/24 brd 172.18.88.255 scope global docker0
       valid_lft forever preferred_lft forever
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 172.18.88.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever

四、配置 kubernetes master 节点

获取 kubernetes v 1.13 二进制安装包 (获取地址)

主要有以下几个压缩包
kubernetes-client-linux-amd64.tar.gz：客户端二进制文件
kubernetes-server-linux-amd64.tar.gz：master 节点二进制文件
kubernetes-node-linux-amd64.tar.gz：node 节点二进制文件

# 复制 kubernetes 进程文件到指定目录
tar -xvf kubernetes-server-linux-amd64.tar.gz 
cd kubernetes/server/bin/
cp kube-scheduler kube-apiserver kube-controller-manager kubectl /opt/kubernetes/bin/

部署 kube-apiserver 组件

创建 TLS Bootstrapping Token

# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
ffaad759999ebdb8133a8ba18d2cd84f

# cat /opt/kubernetes/cfg/token.csv
ffaad759999ebdb8133a8ba18d2cd84f,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

创建 apiserver 配置文件

# cat /opt/kubernetes/cfg/kube-apiserver 
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=4 \
--etcd-servers=https://192.168.4.11:2379,https://192.168.4.25:2379,https://192.168.4.26:2379 \
--bind-address=192.168.4.11 \
--secure-port=6443 \
--advertise-address=192.168.4.11 \
--allow-privileged=true \
--anonymous-auth=false \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--anonymous-auth=false \
--kubelet-client-certificate=/opt/kubernetes/ssl/admin.pem \
--kubelet-client-key=/opt/kubernetes/ssl/admin-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"

创建 kube-apiserver systemd unit 文件

# cat /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动 kube-apiserver 服务

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver

部署 kube-scheduler

创建 kube-scheduler 配置文件

# cat /opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect"

#########################################################
--address：在 127.0.0.1:10251 端口接收 http /metrics 请求；kube-scheduler 目前还不支持接收 https 请求
--kubeconfig：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver
--leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态

创建 kube-scheduler systemd unit 文件

# cat /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动 kube-scheduler 服务

systemctl daemon-reload
systemctl enable kube-scheduler.service 
systemctl restart kube-scheduler.service

4 部署 kube-controller-manager

创建 kube-controller-manager 配置文件

# cat /opt/kubernetes/cfg/kube-controller-manager

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem"

创建 kube-controller-manager systemd unit 文件

# cat /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动 kube-controller-manager 服务

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager

master 节点状态

五、部署 node 节点

复制 kubelet kube-proxy 文件到所有节点

tar -xf kubernetes-node-linux-amd64.tar.gz
scp kubernetes/node/bin/kubelet node1:/opt/kubernetes/bin/
scp kubernetes/node/bin/kubelet node2:/opt/kubernetes/bin/
scp kubernetes/node/bin/kubelet master:/opt/kubernetes/bin/
scp kubernetes/node/bin/kube-proxy node1:/opt/kubernetes/bin/
scp kubernetes/node/bin/kube-proxy node2:/opt/kubernetes/bin/
scp kubernetes/node/bin/kube-proxy master:/opt/kubernetes/bin/

设置环境变量

# 添加以下内容到 /etc/profile
export BOOTSTRAP_TOKEN=ffaad759999ebdb8133a8ba18d2cd84f
export KUBE_APISERVER="https://192.168.4.11:6443"

############################################################

source /etc/profile

创建 kubelet bootstrap kubeconfig 文件

cd /opt/kubernetes/ssl

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 创建 kube-proxy kubeconfig 文件
 
kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

# 创建 admin kubeconfig
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=admin.conf
kubectl config set-credentials admin \
  --client-certificate=/opt/kubernetes/ssl/admin.pem \
  --client-key=/opt/kubernetes/ssl/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=admin.conf
kubectl config set-context default \
  --cluster=kubernetes \
  --user=admin \
  --kubeconfig=admin.conf
kubectl config use-context default --kubeconfig=admin.conf
cp admin.conf ~/.kube/config # 目录不存在的话创建目录，在master节点上配置

将 bootstrap kubeconfig kube-proxy.kubeconfig 文件拷贝到所有节点

scp bootstrap.kubeconfig kube-proxy.kubeconfig master:/opt/kubernetes/cfg
scp bootstrap.kubeconfig kube-proxy.kubeconfig node1:/opt/kubernetes/cfg
scp bootstrap.kubeconfig kube-proxy.kubeconfig node2:/opt/kubernetes/cfg

创建 kubelet 参数配置文件拷贝到所有节点

# cat /opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.4.11
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
      enabled: false
  x509:
      clientCAFile: /opt/kubernetes/ssl/ca.pem

####################################################

scp /opt/kubernetes/ssl/ca.pem node1:/opt/kubernetes/ssl/ca.pem
spc /opt/kubernetes/ssl/ca.pem node2:/opt/kubernetes/ssl/ca.pem
scp /opt/kubernetes/cfg/kubelet.config node1:/opt/kubernetes/cfg
scp /opt/kubernetes/cfg/kubelet.config node2:/opt/kubernetes/cfg

adress 参数配置为各个节点的 kubernetes 使用的网卡地址，不配置时默认监听在所有IP地址，如果node1 IP地址为 192.168.4.25，adress 就修改为 192.168.4.25

创建 kubelet 配置文件

# cat /opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=master \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

######################################################

scp /opt/kubernetes/cfg/kubelet node1:/opt/kubernetes/cfg
scp /opt/kubernetes/cfg/kubelet node2:/opt/kubernetes/cfg

在 master上通过 kubectl get node 获得的列表中，Name 显示的名称是通过客户端 kubelet 和kube-proxy配置文件中 --hostname-override 配置参数定义的，修改这2个参数为你想要的名称，并且删除kubelet.kubeconfig(这个文件是 master 认证后客户端自动生成的，如果不删除会报 node 节点 forbidden )文件，重新启动着2个服务，master 端重新 kubectl certificate approve name 名称就可以看到新名称。所有 kubernetes 节点中该选项不能重复，不设置该选项的话，默认值为主机名
修改配置文件，不删除 kubelet.kubeconfig 文件会报错误：

Unable to register node "node2" with API server: nodes "node2" is forbidden: node "xxx" cannot modify node "node2"

创建 kubelet systemd unit 文件

# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

######################################################

scp /usr/lib/systemd/system/kubelet.service node1:/usr/lib/systemd/system/kubelet.service
scp /usr/lib/systemd/system/kubelet.service node2:/usr/lib/systemd/system/kubelet.service

将 kubelet-bootstrap 用户绑定到系统集群角色

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap

approve kubelet CSR 请求

# 查看 CSR 列表
$  kubectl get csr
NAME                                                   AGE   REQUESTOR           CONDITION
node-csr-79Crh_IQuX7M17a3_glgKvuljv4VLiwvGugPK1I_I3I   72m   kubelet-bootstrap   Pending
node-csr-Kr0LXMj9_VzV53O40lQFaA1YMmByMHTxNhGx0HxfMts   73m   kubelet-bootstrap   Pending
node-csr-ayrbI9Dk9gwcawt7Z4GnUItBk3Frs7TnFKYzldg2yME   77m   kubelet-bootstrap   Pending

# 执行以下命令进行 csr 授权
$ kubectl get csr | sed 1d | awk '{print $1}' | xargs -i kubectl certificate approve {}

配置 kube-proxy 服务

kube-proxy 运行在所有 node节点上，它监听 apiserver 中 service 和 Endpoint 的变化情况，创建路由规则来进行服务负载均衡。

创建 kube-proxy 配置文件

$ cat /opt/kubernetes/cfg/kube-proxy

KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=master \
--proxy-mode=ipvs \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

--hostname-override 参数要和 kubelet 配置文件中的参数一样
创建 kube-proxy systemd unit 文件

$ cat /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy

六、部署结果验证

节点状态

使用以下 yaml 文件进行部署验证

$ cat nginx.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
  - port: 80
    nodePort: 31000
    name: nginx-port
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

七、kubectl 命令行自动补全

Linux Bash

source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)

Linux Zsh

source <(kubectl completion zsh)

CentOS 安装 Kubernetes v1.13
一、系统初始化配置 ( 操作系统为 CentOS 7 ) maser: 192.168.4.11 node1: 1...
CentOS kubernetes单机部署
CentOS kubernetes单机部署 1、关闭防火墙 2、安装etcd和kubernetes软件(会自动安装...
开始创建 Kubernetes 主机
本指南已验证 Kubernetes v1.13 上。由于易变性的 Kubernetes 版本到版本，本部分可能会...
[K8S]Kubernetes环境准备
本文演示CentOS(centos8)中安装docker，安装Kubernetes本文属于个人技术分享，若有错误或...
kubernetes1.4 安装
centos安装kubernetes，calico，nfshttp://blog.csdn.net/yarntim...
centOS7.2使用yum安装kubernetes
kubernetes学习1--centOS7.2使用yum安装kubernetes. 准备3台服务器， 172.1...
Kubernetes中的Configmap和Secret
本文的试验环境为CentOS 7.3，Kubernetes集群为1.11.2，安装步骤参见kubeadm安装kub...
kubeadm 安装kubernetes v1.10
kubeadm 安装kubernetes 安装docker用你系统对应的包管理工具来安装，centos的安装如下y...
[k8s]基于Centos7安装kubernetes1.13.2
本篇文章是基于kubernetes—CentOS7安装kubernetes1.11.2图文完整版这篇文章而写，主要...
kubeadm v1.12.3安装
使用kubeadm安装我们的kubernetes的测试环境环境：CentOS Linux release 7.4...