本文讲介绍CentOS7.5下面搭建GlusterFS集群+Heketi的详细部署步骤。

安装前配置

本实例安装三台GlusterFS集群和一台Gluster client机器。

为了在第一台主机上保留yum rpm包，在其它主机离线安装，需要配置yum保留cache：

sudo sed -i 's/keepcache=0/keepcache=1/g' /etc/yum.conf

安装GlusterFS集群

Gluster的安装包在centos-release-gluster源里面，CentOS7.5默认带的是GlusterFS 3.11版本，默认版本比较低，如果要安装更高的版本需要安装centos-release-gluster源，这个源中有4，5，6版本的Gluster包。

yum search  centos-release-gluster
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
 * base: mirrors.aliyun.com
 * extras: mirror.jdcloud.com
 * updates: mirrors.nwsuaf.edu.cn
base                                                                                                                                                | 3.6 kB  00:00:00     
extras                                                                                                                                              | 3.4 kB  00:00:00     
updates                                                                                                                                             | 3.4 kB  00:00:00     
(1/4): base/7/x86_64/group_gz                                                                                                                       | 166 kB  00:00:00     
(2/4): extras/7/x86_64/primary_db                                                                                                                   | 200 kB  00:00:00     
(3/4): base/7/x86_64/primary_db                                                                                                                     | 6.0 MB  00:00:01     
(4/4): updates/7/x86_64/primary_db                                                                                                                  | 5.0 MB  00:02:07     
=================================================================== N/S matched: centos-release-gluster ===================================================================
centos-release-gluster-legacy.noarch : Disable unmaintained Gluster repositories from the CentOS Storage SIG
centos-release-gluster40.x86_64 : Gluster 4.0 (Short Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster41.noarch : Gluster 4.1 (Long Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster5.noarch : Gluster 5 packages from the CentOS Storage SIG repository
centos-release-gluster6.noarch : Gluster 6 packages from the CentOS Storage SIG repository

  Name and summary matches only, use "search all" for everything.

Gluster版本如何选型

这个需要查看GusterFS的发布计划：
https://www.gluster.org/community/release-schedule

image.png

从这个计划中我们可以看出，EOL是EndOfLife的意思，就是不在维护和支持了，出现bug也不会修改了，所以一般不需要选择EOL的版本，当前已经发布并且维护的版本是4.1和5，其中4.1已经发布并且较长时间，比较稳定，所以本次选择4.1版本安装。

sudo yum install centos-release-gluster41.noarch -y

然后在第一台机器上安装gluster服务端，注意，要使用我们安装的源需要添加选项--enablerepo=centos-gluster*-test

sudo yum --enablerepo=centos-gluster*-test install glusterfs-server glusterfs-cli glusterfs-geo-replication -y

关闭防火墙

sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service

启动Gluster服务

sudo systemctl start glusterd.service
sudo systemctl enable glusterd.service

第一台Gluster服务器就安装完成，复制yum cache到其它节点：

scp -r /var/cache/yum/ root@centos3:/var/cache/

在其它节点选择从cache安装并启动

sudo yum -C -y install centos-release-gluster41.noarch
sudo yum -C -y --enablerepo=centos-gluster*-test install glusterfs-server glusterfs-cli glusterfs-geo-replication

sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service

sudo systemctl start glusterd.service
sudo systemctl enable glusterd.service

把三个节点的Gluster添加为一个集群

HOST_NAME_LIST="centos1,centos2,centos3"
awk 'BEGIN{split('"\"$HOST_NAME_LIST\""',arr,",");for(i in arr){cmd="sudo gluster peer probe "arr[i];print(cmd);system(cmd)}}'
sudo gluster peer probe centos1
peer probe: success. Probe on localhost not needed
sudo gluster peer probe centos2
peer probe: success. 
sudo gluster peer probe centos3
peer probe: success. 
[root@centos1 cent]# gluster peer status
Number of Peers: 2

Hostname: centos2
Uuid: 62f6ecbf-fec4-4004-aa5b-c69d2b792a19
State: Peer in Cluster (Connected)

Hostname: centos3
Uuid: dd1e84c4-e9c2-4b0a-8496-55941abb305a
State: Peer in Cluster (Connected)

安装Heketi

Heketi的安装包也是随着GlusterFS发布的，所以必须选择一致的Heketi进行安装，同样，我们从centos-gluster41的源中安装Heketi。

sudo yum --enablerepo=centos-gluster*-test install heketi heketi-client -y

配置Heketi配置文件，配置文件的含义可以去下面链接查看
https://www.cnblogs.com/netonline/p/10288219.html

## /etc/heketi/heketi.json
sudo echo '
{
  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": true,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "a123"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "a123"
    }
  },

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/var/lib/heketi/id_rsa",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab"
    },

    "_kubeexec_comment": "Kubernetes configuration",
    "kubeexec": {
      "host" :"https://kubernetes.host:8443",
      "cert" : "/path/to/crt.file",
      "insecure": false,
      "user": "kubernetes username",
      "password": "password for kubernetes user",
      "namespace": "OpenShift project or Kubernetes namespace",
      "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
    },

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",

    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    ],
    "_log": "logging to /var/log/message",
    "loglevel" : "warning"
  }
}
' | sudo tee /etc/heketi/heketi.json

定义每个节点上的存储结构topology

sudo echo '
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "centos1"
                            ],
                            "storage": [
                                "centos1"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "centos2"
                            ],
                            "storage": [
                                "centos2"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "centos3"
                            ],
                            "storage": [
                                "centos3"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb"
                    ]
                }       
            ]
        }
    ]
}
' | sudo tee /etc/heketi/topology-cluster-1.json

配置SSH免密登录

sudo ssh-keygen -t rsa -q -f /var/lib/heketi/id_rsa -N ''
sudo chown heketi:heketi /var/lib/heketi/id_rsa*
sudo ssh-copy-id -i /var/lib/heketi/id_rsa root@centos1
sudo ssh-copy-id -i /var/lib/heketi/id_rsa root@centos2
sudo ssh-copy-id -i /var/lib/heketi/id_rsa root@centos3

启动Heketi服务

sudo systemctl start heketi.service
sudo systemctl enable heketi.service
sudo systemctl status heketi.service

通过Heketi命令初始化GlusterFS各节点的存储

heketi-cli --user admin --secret a123 --server http://127.0.0.1:8080 topology load --json /etc/heketi/topology-cluster-1.json
Creating cluster ... ID: 919157039f0aef6126dad5934d3fdb66
        Allowing file volumes on cluster.
        Allowing block volumes on cluster.
        Creating node 192.168.43.88 ... ID: 61b13f9ebce7c382ef9bd43215017374
                Adding device /dev/sdb ... OK
        Creating node 192.168.43.152 ... ID: 0ff9b39ca465aefb777ad2b196a116a5
                Adding device /dev/sdb ... OK

这里会生成cluster id，这个后面要用到。
创建一个2G的3副本的复制卷

alias heketi-cli='heketi-cli --server "http://127.0.0.1:8080" --user admin --secret a123'
heketi-cli volume create --size=2  --replica=3 --clusters=919157039f0aef6126dad5934d3fdb66

        Name: vol_a34cffc6832123b9ffd795744c3eb2df
        Size: 1
        Volume Id: a34cffc6832123b9ffd795744c3eb2df
        Cluster Id: 919157039f0aef6126dad5934d3fdb66
        Mount: 192.168.43.152:vol_a34cffc6832123b9ffd795744c3eb2df
        Mount Options: backup-volfile-servers=192.168.43.88
        Block: false
        Free Size: 0
        Reserved Size: 0
        Block Hosting Restriction: (none)
        Block Volumes: []
        Durability Type: replicate
        Distribute Count: 1
        Replica Count: 3

Heketi会为每个卷起一个uuid的名字，可以通过heketi-cli volume list查看。
使用gluster命令也可以create volume，但是默认没有start，需要用gluster start命令启动volume.

使用Gluster FS存储

如果需要通过native方式使用GlusterFS，则需要在客户端机器上安装glusterfs-client包

sudo yum install centos-release-gluster41.noarch -y
sudo yum --enablerepo=centos-gluster*-test install glusterfs-client -y

通过mount命令就可以挂在GlusterFS的存储卷

sudo mount -t glusterfs centos1:vol_a34cffc6832123b9ffd795744c3eb2df /gfsmnt/