Ceph在CentOS7下的安装

作者: IT界的蜗牛 | 来源:发表于2018-11-26 22:35 被阅读0次

Ceph在CentOS7下的安装
Ceph搭建部署
Ceph Object Gateway
使用Ceph作为OpenStack的统一存储解决方案
centos 7 安装 nodejs
CephFS搭建与部署
macrobase安装过程中的问题
Linux安装-FAQ
用ceph-deploy简单部署一套环境
安装CentOS7双系统

1. 安装Esxi server和vCenter

2. 准备工作

deploy
node1
node2
node3
osd1
osd2
osd3
postgres

2.1 安装Centos 节点

image.png

问题
1. VM 不能自动获取IP

cd etc/sysconfig/network-scripts
vi ifcfg-ethxxx, # 修改ONBOOT="no" to "yes"

#重启服务
systemctl restart network

2.2 创建deploy用户并赋予sudo权限

useradd -d /home/deploy -m deploy
passwd deploy
cat /etc/group
# 确保所有节点上新创建的用户都有 sudo 权限
echo "iflyceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/deploy
sudo chmod 0440 /etc/sudoers.d/deploy

2.3 修改hostname名

deploy@deploy ~]$ hostname
deploy
[deploy@deploy ~]$ sudo hostnamectl --pretty

[deploy@deploy ~]$ sudo hostnamectl --static
deploy
[deploy@deploy ~]$ sudo hostnamectl --transient
deploy
[deploy@deploy ~]$ hostnamectl set-hostname deploy  # "deploy" is the hostname of the VM

2.4 ssh免密登陆

修改 /etc/hosts文件

10.110.125.90 deploy
10.110.125.77 node1
10.110.125.55 node2
10.110.125.54 rgw
10.110.125.83 osd1
10.110.125.52 osd2
10.110.125.53 osd3

#127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

生成ssh 公钥文件，并分发给其他节点

# 用authorized_keys来保存各节点的公钥
deploy@node3 .ssh]$ vi authorized_keys
[deploy@node3 .ssh]$ ssh-keygen
[deploy@node3 .ssh]$ ls
authorized_keys  id_rsa  id_rsa.pub
# 添加id_rsa.pub到authorized_keys
[deploy@node3 .ssh]$ cat id_rsa.pub >> authorized_keys

# 复制authorized_keys到其他节点
[deploy@node3 .ssh]$ scp -r authorized_keys deploy@deploy:/home/deploy/.ssh
The authenticity of host 'deploy (10.110.125.90)' can't be established.
ECDSA key fingerprint is SHA256:d1zplvj7CshS6Ub+8t1iPo/RlLTYoiDRkx0v7LHkd1w.
ECDSA key fingerprint is MD5:83:79:03:8d:d6:aa:a0:b9:1e:35:c0:65:6a:c3:79:c3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'deploy,10.110.125.90' (ECDSA) to the list of known hosts.
deploy@deploy's password:
authorized_keys                                                             100% 3148    84.5KB/s   00:00

2.5 打开必须端口

Monitor默认使用6789端口，OSDs默认使用6800:7300之间的端口

[deploy@node1 ~]$ sudo /sbin/iptables -I INPUT -p tcp --dport 6789 -j ACCEPT
[deploy@node1 ~]$ sudo /sbin/iptables -I INPUT -p tcp --dport 6800:7300 -j ACCEPT

3. 在vCenter下安装Ceph

3.1 安装ceph-deploy

# Install and enable the Extra Packages for Enterprise Linux (EPEL) repository
[deploy@deploy ~]$ sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Loaded plugins: fastestmirror
epel-release-latest-7.noarch.rpm        |  15 kB     00:00
Examining /var/tmp/yum-root-4rZs4w/epel-release-latest-7.noarch.rpm: epel-release-7-11.noarch

# Add the Ceph repository to your yum configuration file at /etc/yum.repos.d/ceph-deploy.repo with the following command
[deploy@deploy ~]$ cat /etc/yum.repos.d/ceph-deploy.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-mimic/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

# Update your repository and install ceph-deploy
[deploy@deploy ~]$ sudo yum update
[deploy@deploy ~]$ sudo yum install ceph-deploy

# Install pip and ntp
sudo yum install ntp ntp-date ntp-doc
sudo yum install python-pip

# Add ntp server's address to configuration file 
[deploy@deploy ~]$ vi /etc/ntp.conf
#server 3.centos.pool.ntp.org iburst
server 10.110.160.1
server 10.110.160.2

# Start ntpd server
sudo systemctl start ntpd

3.2 安装ceph 集群

创建deploy集群文件夹

deploy@deploy ~]$ cd
[deploy@deploy ~]$ mkdir my-cluster
[deploy@deploy ~]$ cd my-cluster
[deploy@deploy my-cluster]$

创建集群

deploy@deploy my-cluster]$ ceph-deploy new node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/deploy/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new node1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f99e49cdd70>
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f99e41473b0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['node1']
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[node1][DEBUG ] connected to host: deploy
[node1][INFO  ] Running command: ssh -CT -o BatchMode=yes node1
[ceph_deploy.new][WARNIN] could not connect via SSH
[ceph_deploy.new][INFO  ] will connect again with password prompt
deploy@node1's password:
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.new][INFO  ] adding public keys to authorized_keys
[node1][DEBUG ] append contents to file
deploy@node1's password:
[node1][DEBUG ] connection detected need for sudo
deploy@node1's password:
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[node1][INFO  ] Running command: sudo /usr/sbin/ip link show
[node1][INFO  ] Running command: sudo /usr/sbin/ip addr show
[node1][DEBUG ] IP addresses found: [u'10.110.125.77', u'fc00:10:110:127:4a0c:15e:3cec:afe8']
[ceph_deploy.new][DEBUG ] Resolving host node1
[ceph_deploy.new][DEBUG ] Monitor node1 at 10.110.125.77
[ceph_deploy.new][DEBUG ] Monitor initial members are ['node1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.110.125.77']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[deploy@deploy my-cluster]$

更新ceph.conf

[deploy@deploy my-cluster]$ ls
ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring

[deploy@deploy my-cluster]$ cat ceph.conf
[global]
fsid = 340734b0-331b-4037-8a8c-60e3069e41af
mon_initial_members = node1
mon_host = 10.110.125.77
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
# 添加以下三行
public network = 10.110.125.0/22
rgw_swift_account_in_url = true
rgw_enable_usage_log = true
[deploy@deploy my-cluster]$

安装ceph节点 (每个节点都要安装)

ceph-deploy install node1 node2 node3 --release mimic
ceph-deploy install osd1 osd2 osd3 --release mimic

设置MON，ADMIN demean

ceph-deploy mon create-initial

ceph-deploy admin node1 node2 node3

ceph-deploy mgr create node1

Steps to add MON

# How to add MON
# ceph-deploy mon add node2
#
 [deploy@deploy my-cluster]$ ceph-deploy mon add node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/deploy/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mon add node2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : add
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fd88b7d5bd8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['node2']
[ceph_deploy.cli][INFO  ]  func                          : <functionmon at 0x7fd88b7bb398>
[ceph_deploy.cli][INFO  ]  address                       : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mon][INFO  ] ensuring configuration of new mon host: node2
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
deploy@node2's password:
[node2][DEBUG ] connection detected need for sudo
deploy@node2's password:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.mon][DEBUG ] Adding mon to cluster ceph, host node2
[ceph_deploy.mon][DEBUG ] using mon address by resolving host: 10.110.125.55
[ceph_deploy.mon][DEBUG ] detecting platform for host node2 ...
deploy@node2's password:
[node2][DEBUG ] connection detected need for sudo
deploy@node2's password:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO  ] distro info: CentOS Linux 7.5.1804 Core
[node2][DEBUG ] determining if provided host has same hostname in remote
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] adding mon to node2
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][DEBUG ] create the mon path if it does not exist
[node2][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node2/done
[node2][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node2/done
[node2][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-node2.mon.keyring
[node2][DEBUG ] create the monitor keyring file
[node2][INFO  ] Running command: sudo ceph --cluster ceph mon getmap-o /var/lib/ceph/tmp/ceph.node2.monmap
[node2][WARNIN] got monmap epoch 1
[node2][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs-i node2 --monmap /var/lib/ceph/tmp/ceph.node2.monmap --keyring /var/lib/ceph/tmp/ceph-node2.mon.keyring --setuser 167 --setgroup 167
[node2][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-node2.mon.keyring
[node2][DEBUG ] create a done file to avoid re-doing the mon deployment
[node2][DEBUG ] create the init path if it does not exist
[node2][INFO  ] Running command: sudo systemctl enable ceph.target
[node2][INFO  ] Running command: sudo systemctl enable ceph-mon@node2
[node2][WARNIN] Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@node2.service to /usr/lib/systemd/system/ceph-mon@.service.
[node2][INFO  ] Running command: sudo systemctl start ceph-mon@node2
[node2][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
[node2][WARNIN] node2 is not defined in `mon initial members`
[node2][WARNIN] monitor node2 does not exist in monmap
[node2][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
[node2][DEBUG ] ********************************************************************************
[node2][DEBUG ] status for monitor: mon.node2
[node2][DEBUG ] {
[node2][DEBUG ]   "election_epoch": 0,
[node2][DEBUG ]   "extra_probe_peers": [],
[node2][DEBUG ]   "feature_map": {
[node2][DEBUG ]     "mon": [
[node2][DEBUG ]       {
[node2][DEBUG ]         "features": "0x3ffddff8ffa4fffb",
[node2][DEBUG ]         "num": 1,
[node2][DEBUG ]         "release": "luminous"
[node2][DEBUG ]       }
[node2][DEBUG ]     ]
[node2][DEBUG ]   },
[node2][DEBUG ]   "features": {
[node2][DEBUG ]     "quorum_con": "0",
[node2][DEBUG ]     "quorum_mon": [],
[node2][DEBUG ]     "required_con": "144115188346404864",
[node2][DEBUG ]     "required_mon": [
[node2][DEBUG ]       "kraken",
[node2][DEBUG ]       "luminous",
[node2][DEBUG ]       "mimic",
[node2][DEBUG ]       "osdmap-prune"
[node2][DEBUG ]     ]
[node2][DEBUG ]   },
[node2][DEBUG ]   "monmap": {
[node2][DEBUG ]     "created": "2018-11-26 01:00:08.121877",
[node2][DEBUG ]     "epoch": 1,
[node2][DEBUG ]     "features": {
[node2][DEBUG ]       "optional": [],
[node2][DEBUG ]       "persistent": [
[node2][DEBUG ]         "kraken",
[node2][DEBUG ]         "luminous",
[node2][DEBUG ]         "mimic",
[node2][DEBUG ]         "osdmap-prune"
[node2][DEBUG ]       ]
[node2][DEBUG ]     },
[node2][DEBUG ]     "fsid": "340734b0-331b-4037-8a8c-60e3069e41af",
[node2][DEBUG ]     "modified": "2018-11-26 01:00:08.121877",
[node2][DEBUG ]     "mons": [
[node2][DEBUG ]       {
[node2][DEBUG ]         "addr": "10.110.125.77:6789/0",
[node2][DEBUG ]         "name": "node1",
[node2][DEBUG ]         "public_addr": "10.110.125.77:6789/0",
[node2][DEBUG ]         "rank": 0
[node2][DEBUG ]       }
[node2][DEBUG ]     ]
[node2][DEBUG ]   },
[node2][DEBUG ]   "name": "node2",
[node2][DEBUG ]   "outside_quorum": [],
[node2][DEBUG ]   "quorum": [],
[node2][DEBUG ]   "rank": -1,
[node2][DEBUG ]   "state": "probing",
[node2][DEBUG ]   "sync_provider": []
[node2][DEBUG ] }
[node2][DEBUG ] ********************************************************************************
[node2][INFO  ] monitor: mon.node2 is currently at the state of probing
[deploy@deploy my-cluster]$

Steps to check ceph status

# ceph -s
#
[deploy@deploy my-cluster]$ ssh node1 sudo ceph -s
deploy@node1's password:
  cluster:
    id:     340734b0-331b-4037-8a8c-60e3069e41af
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node3,node2,node1
    mgr: node1(active), standbys: node2, node3
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

3.2 安装OSDs
check fdisk status
sdb, sdc, sdd of node osd1 would be used as osd

deploy@osd1 ~]$ lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part
  ├─centos-root 253:0    0   47G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   50G  0 disk
sdc               8:32   0   50G  0 disk
sdd               8:48   0   50G  0 disk
sr0              11:0    1 1024M  0 rom

[deploy@deploy my-cluster]$ ceph-deploy osd create --data /dev/sdb osd1
[ceph_deploy.osd][DEBUG ] Host osd1 is now ready for osd use.

# continue add sdc, sdd

continue add three osds for osd2, osd3

查看OSD数量

[deploy@deploy ~]$ ssh node1 sudo ceph -s
deploy@node1's password:
  cluster:
    id:     340734b0-331b-4037-8a8c-60e3069e41af
    health: HEALTH_WARN
            1 osds down
            8 slow ops, oldest one blocked for 5150 sec, mon.node1 has slow ops

  services:
    mon: 3 daemons, quorum node3,node2,node1
    mgr: node1(active), standbys: node2, node3
    osd: 9 osds: 5 up, 6 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

3.3 安装rgw
RGW默认使用Civetweb作为其Web Sevice，而Civetweb默认使用端口7480提供服务，如果想修改端口（如80端口），就需要修改Ceph的配置文件。在配置文件中增加一个section[client.rgw.xxx]

[deploy@deploy my-cluster]$ vi ceph.conf
# Content
[client.rgw.node1] //[client.rgw.{gatewayNode}
rgw_frontends = "civetweb port=7480 num_threads=512 request_timeout_ms=30000"
rgw_enable_usage_log = true
rgw_thread_pool_size = 512
rgw_override_bucket_index_max_shards = 20
rgw_max_chunk_size = 1048576
rgw_cache_lru_size = 1000000
rgw_op_thread_timeout = 6000
rgw_num_rados_handles = 8
rgw_cache_enabled = true
objecter_inflight_ops = 10240
objecter_inflight_op_bytes = 1048576000

# Push configuration to other rgw node ( node1, node2, node3)
ceph-deploy --overwrite-conf config push node3

创建rgw网关

ceph-deploy install --rgw node1 node2 node3

创建rgw实例

ceph-deploy rgw create node1

问题列表

WARN related with MON creation

[node2][INFO  ] Running command: sudo ceph --cluster ceph mon getmap-o /var/lib/ceph/tmp/ceph.node2.monmap
[node2][WARNIN] got monmap epoch 1

[node2][WARNIN] node2 is not defined in `mon initial members`
[node2][WARNIN] monitor node2 does not exist in monmap

常用命令

CentOS 查看磁盘分区情况

# lsblk 查看分区和磁盘
[deploy@osd1 ~]$ lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part
  ├─centos-root 253:0    0   47G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   50G  0 disk
sdc               8:32   0   50G  0 disk
sdd               8:48   0   50G  0 disk
sr0              11:0    1 1024M  0 rom

# df -h 查看空间使用情况
[deploy@osd1 /]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   47G  1.4G   46G   3% /
devtmpfs                 908M     0  908M   0% /dev
tmpfs                    920M     0  920M   0% /dev/shm
tmpfs                    920M  8.8M  911M   1% /run
tmpfs                    920M     0  920M   0% /sys/fs/cgroup
/dev/sda1               1014M  142M  873M  14% /boot
tmpfs                    184M     0  184M   0% /run/user/1000

# fdisk -l 分区工具查看分区信息
deploy@osd1 /]$ sudo fdisk -l

Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00038faf

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM

Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
...

# du -sh ./* 统计当前目录各文件夹大小
[deploy@osd1 ~]$ du -sh
32K

Ceph在CentOS7下的安装
1. 安装Esxi server和vCenter 2. 准备工作 deploynode1node2node3osd...
Ceph搭建部署
修改yum源添加ceph的repo源文件整合软件仓库安装ceph软件在ceph管理节点安装ceph-dep...
Ceph Object Gateway
Environment 关于ceph的安装可参考ceph luminous安装 1.Installing ceph...
使用Ceph作为OpenStack的统一存储解决方案
Ceph官方安装文档前文列表 OpenStack Train版安装② Ceph版本选择与安装③ Ceph—RBD...
centos 7 安装 nodejs
在安装node的过程中总结了下centos7安装nginx，现在安装完了node，又来总结下centos7安装no...
CephFS搭建与部署
0. 前提已搭建好的ceph cluster，详细步骤可参见Ceph 部署（Centos7 + Luminous...
macrobase安装过程中的问题
macrobase需要安装postgresql数据库环境本次安装环境centos7，在centos7环境下安装po...
Linux安装-FAQ
在Mac-Parallels下安装CentOS7 Mac利用PD虚拟机安装Centos7 CentOS 7 在 P...
用ceph-deploy简单部署一套环境
本次实验全在CENTOS7下完成，如果对DEBIAN有强制需求，修改相应的命令也行。先添加CEPH源方法一： ...
安装CentOS7双系统
今天因为实验需要，尝试着在windows10下安装CentOS7双系统，总体感觉安装CentOS7要比安装Ub...