k8s rke 安装报错解决

官网安装失败，清除
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/cleaning-cluster-nodes/

etcd logs:2019-08-01 11:17:54.301389 I | embed: rejected connection from "192.168.3.2:60362" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")", ServerName "")

WARN[0236] Failed to create Docker container [kube-proxy] on host [192.168.1.12]: Error response from daemon: Duplicate mount point: /lib/modules


解决
docker stop $(docker ps -aq)
docker system prune -f
docker volume rm $(docker volume ls -q)

for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done

rm -rf /etc/ceph \
       /etc/cni \
       /etc/kubernetes \
       /opt/cni \
       /opt/rke \
       /run/secrets/kubernetes.io \
       /run/calico \
       /run/flannel \
       /var/lib/calico \
       /var/lib/etcd \
       /var/lib/cni \
       /var/lib/kubelet \
       /var/lib/rancher/rke/log \
       /var/log/containers \
       /var/log/pods \
       /var/run/calico

.rke remove
重启

问题
ERRO[0354] Failed to upgrade worker components on NotReady hosts, error: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [192.168.1.10]: Get http://localhost:10256/healthz: Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: E0712 10:17:35.216054    6133 node.go:124] Failed to retrieve node info: nodes "192.168.1.10" not found]

docker logs kube-proxy

Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it
解决
1. ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
2.   升级iptables 到1.6
    1. yum install gcc make libnftnl-devel libmnl-devel autoconf automake libtool bison flex libnetfilter_conntrack-devel libnetfilter_queue-devel libpcap-devel
    1. wget wget https://www.netfilter.org/projects/iptables/files/ipta bles-1.6.2.tar.bz2
    2. tar -xvf iptables-1.6.2.tar.bz2
    3. cd iptables-1.6.2
    4. ./autogen.sh
    5. ./configure
    6. make -j4
    7. make install 
    8. # 当然可以把cd /usr/local/sbin下面的iptables相关的东西打包然后分发到其它服务器
    9. cd /usr/local/sbin
    10. cp iptables /sbin
    11. cp iptables-restore /sbin/
    12. cp iptables-save /sbin/

然后直接启动 rke up，需要清理重启
注意需要关闭防火墙
systemctl stop firewalld
https://www.itocm.com/a/8B5630EF63824C43978E1482F15623A4

成功日志
INFO[0344] [addons] Successfully saved ConfigMap for addon rke-user-addon to Kubernetes
INFO[0344] [addons] Executing deploy job rke-user-addon
INFO[0355] [addons] User addons deployed successfully   
INFO[0355] Finished building Kubernetes cluster successfully