OpenShift Origin 高级安装部署指南

概述

根据官方文档中的“高级安装指南”的建议，如果想要搭建真正的满足生产环境要求的 OpenShift 集群，我们需要借助配置管理工具 Ansible 来安装 OpenShift 集群。本文所介绍的即为面向生产环境的高级安装指南。

基础设施配置清单

服务器需要三台：一台主节点服务器，两台子节点服务器，其中主节点服务器也会充当 NFS 文件共享存储目录和 etcd 数据持久化的服务器。

配置属性	master	node01	node02
CPU 逻辑内核数量	4	4	4
内存	16G	8G	8G
硬盘	300G	100G	100G
操作系统	CentOS 7	CentOS 7	CentOS 7
hostname	master.example.com	node01.example.com	node02.example.com
服务器 IP	192.168.123.155	192.168.123.156	192.168.123.158
root 用户密码	handhand	handhand	handhand
服务器角色	主控节点，NFS	计算节点，load balancer	计算节点

主要服务组件清单

组件名称	版本/分支	安装来源	安装节点	备注
docker	1.12.6	yum 软件仓库	所有节点
VMware Harbor	1.2.2	Github	N/A	由 gzregistry 服务器提供注册服务
ansible	2.4.0	yum epel	主节点
openshift-ansible	release-3.6	Github	主节点
OpenShift Origin	3.6.1	openshift-ansible / yum 软件仓库	主节点
etcd	3.1.9	openshift-ansible / yum 软件仓库	主节点

关于 etcd 的版本问题

根据安装实测，CentOS 软件仓库提供的最新版本的 etcd 3.2.7 目前存在问题，服务启动时报证书错误，如下所示：

etcd started with error: transport: remote error: tls: bad certificate; please retry

所以目前我们在 ansible 配置文件中认为限定 etcd 的安装版本为3.1.9

OpenShift 详细安装步骤

本章节将详细介绍 OpenShift Origin 集群搭建的详细过程。完整的安装指南请访问官方链接：https://docs.openshift.org/latest/install_config/install/advanced_install.html

注意：以下列示的步骤需要在所有节点上实施。

为 CentOS 配置国内镜像源

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache
yum -y update && yum -y upgrade

添加 Host 映射

192.168.123.155 master.example.com master
192.168.123.155 nfs.example.com
192.168.123.156 lb.example.com
192.168.123.156 node01.example.com node01
192.168.123.158 node02.example.com node02

☞ 注意：以下步骤仅在“主节点”上实施。

配置 SSH 免密码登陆

ssh-keygen -f ~/.ssh/id_rsa -N ''
for host in master.example.com node01.example.com node02.example.com
do
    ssh-copy-id -i ~/.ssh/id_rsa.pub $host;
done

安装基础软件包

yum -y install docker git wget net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct

设置 Docker 为开机自启动并启动 Docker 守护进程

systemctl enable docker
systemctl start docker

移动 Docker 数据目录至数据盘分区 /u01

systemctl stop docker
mkdir -p /u01/var/lib
mv /var/lib/docker /u01/var/lib
ln -s /u01/var/lib/docker /var/lib/docker
systemctl start docker

安装 Ansible

yum install -y epel-release
yum repolist
yum install -y ansible pyOpenSSL python-cryptography python-lxml

克隆仓库 openshift-ansible 并切换至 release-3.6 分支

git clone -b release-3.6 https://github.com/openshift/openshift-ansible.git /usr/share/openshift-ansible

配置 ansible 库存文件

编辑 /etc/ansible/hosts 文件并添加一下内容：

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd
lb
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root
# If ansible_ssh_user is not root, ansible_become must be set to true
#ansible_become=true
openshift_deployment_type=origin
# Specify the generic release of OpenShift to install. This is used mainly just during installation, after which we
# rely on the version running on the first master. Works best for containerized installs where we can usually
# use this to lookup the latest exact version of the container images, which is the tag actually used to configure
# the cluster. For RPM installations we just verify the version detected in your configured repos matches this
# release.
openshift_release=v3.6
# Specify an exact container image tag to install or configure.
# WARNING: This value will be used for all hosts in containerized environments, even those that have another version installed.
# This could potentially trigger an upgrade and downtime, so be careful with modifying this value after the cluster is set up.
openshift_image_tag=v3.6.1
# Specify an exact rpm version to install or configure.
# WARNING: This value will be used for all hosts in RPM based environments, even those that have another version installed.
# This could potentially trigger an upgrade and downtime, so be careful with modifying this value after the cluster is set up.
openshift_pkg_version=-3.6.1
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
# Default login account: admin / handhand
openshift_master_htpasswd_users={'admin': '$apr1$gfaL16Jf$c.5LAvg3xNDVQTkk6HpGB1'}

# Specify exact version of etcd to configure or upgrade to.
etcd_version="3.1.9"
#openshift_repos_enable_testing=true
openshift_disable_check=disk_availability,docker_storage
docker_selinux_enabled=false
openshift_docker_options=" --log-driver=journald --storage-driver=overlay --registry-mirror=http://4a0fee72.m.daocloud.io "
# OpenShift Router Options
# Router selector (optional)
# Router will only be created if nodes matching this label are present.
# Default value: 'region=infra'
openshift_hosted_router_selector='region=infra,router=true'
# default subdomain to use for exposed routes
openshift_master_default_subdomain=app.example.com
# host group for masters
[masters]
master.example.com
# host group for etcd
[etcd]
master.example.com
# Load balancers
[lb]
lb.example.com
# host group for nodes, includes region info
[nodes]
master.example.com openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'router': 'true'}"
node01.example.com openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'router': 'true'}"
node02.example.com openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'router': 'true'}"

运行 ansible 手册安装 OpenShift Origin

ansible-playbook /usr/share/openshift-ansible/playbooks/byo/config.yml

配置 NFS 服务器

安装依赖服务组件并启动服务

yum install -y nfs-utils rpcbind
rpm -qa | grep -E 'nfs-utils|rpcbind'
systemctl enable rpcbind
systemctl enable nfs
systemctl restart rpcbind
systemctl restart nfs

编辑 /etc/sysconfig/iptables 并新增一下内容：

#
# NFS client access allowd settings
#
# The following line must be commented in order to make nfs work properly
#-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 192.168.0.0/24 -p tcp --dport 2049 -j ACCEPT

保存配置并重启 iptables 服务:

service iptables save
service iptables restart

初始化 Persistent Volumes

这里借鉴 minishift 默认的 pv 初始化，即预创建100个 pv （pv0001 ~ pv0100 ) 初始化大小为100Gi
运行一下脚本进行初始化：

cd $OPENSHIFT_ADMIN/bin
./init-100-pvs.sh

在本机配置 hosts 映射

Windows 操作系统下编辑 C:\Windows\System32\drivers\etc\hosts
Linux / macOS 操作系统下编辑 /etc/hosts
添加一下内容：

192.168.123.155 master.example.com

访问 OpenShift Origin 主页

访问以下地址确保 OpenShift Origin 可以正确访问： https://master.example.com:8443/
用户账户： admin / handhand