MongoDB Replica Set 详细安装文档

1. 准备工作

• 获取CentOS 7 ISO 文件

  CentOS-7-x86_64-Everything-1503-01.iso

• 安装VirtualBox

  VirtualBox

• 安装CentOS 7

  安装中，需要注意的几个步骤

  

• 连接网络

  • 检查网卡设置

  • 搞通网关连通

  • 设置dns等即可

  参考

2. Install MongoDB

• Install net-tools

确保ifconfig, netstat, route 等命令可以使用

  sudo yum install net-tools

• Install wget

  下面会用到wget命令，CentOS 7 最小安装默认没带wget命令

  sudo yum install wget

• Install epel

epel，企业Linux附加软件包。epel的软件包通常不会与企业版Linux
官方源中的软件包发生冲突，或者相互替换。

  wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
  sudo rpm -ivh epel-release-7-5.noarch.rpm

  yum --enablerepo=epel info mongodb
    Name        : mongodb
    Arch        : x86_64
    Version     : 2.6.11
    Release     : 1.el7
    Size        : 43 M
    Repo        : epel/x86_64

在epel源中显示的版本为 2.6.11 版本，不符合预期。需安装最新的3.0.7版本

参考官方的安装指南
install-mongodb-on-red-hat

  sudo touch /etc/yum.repos.d/mongodb-org-3.0.repo

  sudo vim /etc/yum.repos.d/mongodb-org-3.0.repo

在 /etc/yum.repos.d/mongodb-org-3.0.repo 中粘贴如下内容

  [mongodb-org-3.0]
   name=MongoDB Repository
   baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
   gpgcheck=0
   enabled=1

执行

  sudo yum install -y mongodb-org

提示安装成功

  Installed:
  mongodb-org.x86_64 0:3.0.7-1.el7

安装之后，使用下面命令测试安装是否成功

  sudo service mongod start  开启MongoDB

  sudo service mongod stop   关闭MongoDB

3. 统一目录

创建数据存储目录

  cd /data/
  sudo mkdir db37017
  sudo mkdir db47017

统一配置文件

  cd /etc/
  sudo cp mongod.conf mongod_37017.conf
  sudo cp mongod.conf mongod_47017.conf

授予当前操作者操作目录的权限

  sudo chown -R deploy /var/log/mongodb
  sudo chown -R deploy /data/
  sudo chown -R deploy /var/run/mongodb/

192.168.1.100:37017 配置如下

配置文件位置：/etc/mongod37017.conf

  # mongod.conf
  # for documentation of all options, see:
  #   http://docs.mongodb.org/manual/reference/configuration-options/

  systemLog:
    destination: file
    logAppend: true
    path: /var/log/mongodb/mongod37017.log

  storage:
    dbPath: /data/wt_db37017
    journal:
      enabled: true
    engine: wiredTiger

  processManagement:
    fork: true
    pidFilePath: /var/run/mongodb/mongod37017.pid

  net:
    port: 37017

  replication:
    replSetName: kt_rs

192.168.1.100:47017 配置如下

配置文件位置：/etc/mongod47017.conf

  # mongod.conf
  # for documentation of all options, see:
  #   http://docs.mongodb.org/manual/reference/configuration-options/

  systemLog:
    destination: file
    logAppend: true
    path: /var/log/mongodb/mongod47017.log

  storage:
    dbPath: /data/wt_db47017
    journal:
      enabled: true
    engine: wiredTiger

  processManagement:
    fork: true
    pidFilePath: /var/run/mongodb/mongod47017.pid

  net:
    port: 47017

  replication:
    replSetName: kt_rs

192.168.1.101:27017 配置如下

配置文件位置：/etc/mongod27017.conf

  # mongod.conf
  # for documentation of all options, see:
  #   http://docs.mongodb.org/manual/reference/configuration-options/

  systemLog:
    destination: file
    logAppend: true
    path: /var/log/mongodb/mongod27017.log

  storage:
    dbPath: /data/wt_db27017
    journal:
      enabled: true
    engine: wiredTiger

  processManagement:
    fork: true
    pidFilePath: /var/run/mongodb/mongod27017.pid

  net:
    port: 27017

  replication:
   replSetName: kt_rs

注意：MongoDB 的默认引擎为 mmapv1 ，若之前已经使用mmapv1
有生成的数据目录，再使用同一个数据目录，但引擎变更为 wiredTiger
时，无法启动。需另外创建一块数据目录，为wiredTiger 单独配置。
详细参考下面链接:

upgrade-a-replica-set
default-mongodb-port

4. Replica Set Deployment

• bind_ip

  MongoDB 默认为 127.0.0.1
  要配置 Replica Set 就需要注释掉bind_ip

• telnet

  
    yum install telnet
  
  

确保以下命令在各服务器上务必都成功

```

  telnet 192.168.1.101 27017
  telnet 192.168.1.100 37017
  telnet 192.168.1.100 47017

```

• firewall

  如果在内网，可以关闭防火墙

  
    sudo systemctl stop firewalld
  
  

• configration

以 192.168.1.101:27017 为主节点配置

  [deploy@dev01 ~]$ mongo 192.168.1.101:27017

  config = {
    "_id": "kt_rs",
    "members": [
      {
        "_id": 0,
        "host": "192.168.1.101:27017"
      }
    ]
  }

  rs.initiate(config)
  rs.status()

添加其他节点

  rs.add("192.168.1.100:37017")
  rs.add("192.168.1.100:47017")

禁止链式复制，设置各节点优先级

 cfg = rs.config()
 cfg["settings"]["chainingAllowed"] = false
 cfg["members"][3].priority = 5  ## 设置当前配置最好的机器优先级稍微高一些
 rs.reconfig(cfg, { "force": true })

Replica Set 之后状态如下：

kt_rs:SECONDARY> rs.status()
 {
  "set" : "kt_rs",
  "date" : ISODate("2015-11-06T10:02:40.831Z"),
  "myState" : 2,
  "syncingTo" : "192.168.1.101:27017",
  "members" : [
      {
          "_id" : 1,
          "name" : "192.168.1.100:37017",
          "health" : 1,
          "state" : 2,
          "stateStr" : "SECONDARY",
          "uptime" : 7937,
          "optime" : Timestamp(14467.2962, 2),
          "optimeDate" : ISODate("2015-11-06T09:42:42Z"),
          "syncingTo" : "192.168.1.101:27017",
          "configVersion" : 33761,
          "self" : true
      },
      {
          "_id" : 2,
          "name" : "192.168.1.100:47017",
          "health" : 1,
          "state" : 2,
          "stateStr" : "SECONDARY",
          "uptime" : 7316,
          "optime" : Timestamp(14467.2962, 2),
          "optimeDate" : ISODate("2015-11-06T09:42:42Z"),
          "lastHeartbeat" : ISODate("2015-11-06T10:02:40.245Z"),
          "lastHeartbeatRecv" : ISODate("2015-11-06T10:02:40.245Z"),
          "pingMs" : 0,
          "syncingTo" : "192.168.1.100:37017",
          "configVersion" : 33761
      },
      {
          "_id" : 3,
          "name" : "192.168.1.101:27017",
          "health" : 1,
          "state" : 1,
          "stateStr" : "PRIMARY",
          "uptime" : 7937,
          "optime" : Timestamp(14467.2962, 2),
          "optimeDate" : ISODate("2015-11-06T09:42:42Z"),
          "lastHeartbeat" : ISODate("2015-11-06T10:02:38.917Z"),
          "lastHeartbeatRecv" : ISODate("2015-11-06T10:02:39.667Z"),
          "pingMs" : 0,
          "electionTime" : Timestamp(1446796155, 1),
          "electionDate" : ISODate("2015-11-06T07:49:15Z"),
          "configVersion" : 33761
      }
  ],
  "ok" : 1
 }

执行以下命令启动：

  mongod -f /etc/mongod27017.conf
  mongod -f /etc/mongod37017.conf
  mongod -f /etc/mongod47017.conf

5. 设置monit监控MongoDB

• 安装monit

  安装monit 之前，确保epel 安装成功

  sudo yum install monit
  ## 设置开机自动启动
  sudo systemctl enable monit.service
  sudo service monit start

任务	旧指令	新指令
使某服务自动启动	chkconfig --level 3 httpd on	systemctl enable httpd.service
启动某服务	service httpd start	systemctl start httpd.service
重启某服务	service httpd restart	systemctl restart httpd.service

CentOS 7.x
systemd

monit 配置

配置文件位置： /etc/monitrc

  set daemon  30
  set logfile syslog

  set mailserver smtp.126.com username "wan***" password "****"

  set mail-format {
    from: wanghao293@126.com
    subject: monit alert --  $EVENT 192.168.1.100
    message: $EVENT Service $SERVICE
                  Date:        $DATE
                  Action:      $ACTION
                  Host:        $HOST
                  Description: $DESCRIPTION

             Your faithful employee,
             Monit
  }

  set alert wanghao@kaitongamc.com

  set httpd port 27.2 and
     allow admin:aPIFuc/3

  include /etc/monit.d/*

• 设置monit 监控MongoDB

192.168.1.101 监控配置

配置文件位置：/etc/monit.d/m_mongo27017.conf

  check process mongodb with pidfile /var/run/mongodb/mongod27017.pid
    group database
    start program = "/usr/bin/mongod -f /etc/mongod27017.conf"
    stop program  = "/usr/bin/mongod -f /etc/mongod27017.conf --shutdown"
    if failed host 127.0.0.1 port 27017 then restart
    if failed host 127.0.0.1 port 27017 then alert
    if 5 restarts within 5 cycles then timeout
    if 5 restarts within 5 cycles then alert

192.168.1.100 监控配置，由于启动两个 MongoDB instance 所以分别监控

配置文件位置：/etc/monit.d/m_mongo37017.conf

 check process mongodb37017 with pidfile /var/run/mongodb/mongod37017.pid
   group database
   start program = "/usr/bin/mongod -f /etc/mongod37017.conf"
   stop program  = "/usr/bin/mongod -f /etc/mongod37017.conf --shutdown"
   if failed host 127.0.0.1 port 37017 then restart
   if failed host 127.0.0.1 port 37017 then alert
   if 5 restarts within 5 cycles then timeout
   if 5 restarts within 5 cycles then alert

配置文件位置：/etc/monit.d/m_mongo47017.conf

 check process mongodb47017 with pidfile /var/run/mongodb/mongod47017.pid
   group database
   start program = "/usr/bin/mongod -f /etc/mongod47017.conf"
   stop program  = "/usr/bin/mongod -f /etc/mongod47017.conf --shutdown"
   if failed host 127.0.0.1 port 47017 then restart
   if failed host 127.0.0.1 port 47017 then alert
   if 5 restarts within 5 cycles then timeout
   if 5 restarts within 5 cycles then alert

在192.168.1.101 和 192.168.1.100 分别执行 sudo monit reload 使配置生效。

你还可以

sudo monit start all 启动所有监控应用

sudo monit start mongodb37017 启动端口为37017 的MongoDB instance

sudo monit start mongodb47017 启动端口为47017 的MongoDB instance

sudo monit stop all 停止所有监控应用

sudo monit restart all 重启所有监控应用

sudo monit unmonitor all 停止监控所有应用

6. 数据文件备份

• clone 备份脚本

  git clone https://github.com/micahwedemeyer/automongobackup.git

  sudo chown deploy /var/backups/mongodb/

• 编写定时任务

  crontab -e

  30 23 * * * /bin/bash /home/deploy/automongobackup/src/automongobackup.sh

• 备份生成的文件结构如下

  [deploy@dev01 ~]$ tree /var/backups/mongodb/
  /var/backups/mongodb/
  ├── daily
  │   ├── 2015-11-04_23h30m.Wednesday.tgz
  │   └── 2015-11-05_23h30m.Thursday.tgz
  ├── latest
  │   └── 2015-11-05_23h30m.Thursday.tgz
  ├── monthly
  └── weekly

  4 directories, 3 files

7 集群设置注意点

Replica Set 有效节点不足最小值(Math.floor(4/2 + 1))
该4个节点的集群中，有效节点不足3个，整个集群便不可用。

8 MongoDB 常见的错误

出现错误，首先查看日志 /var/log/mongodb/mongod.log

8.1 mongodb exception in initAndListen: 12596 old lock file, terminating

• 删除data目录中的.lock 文件

• 重新启动mongod

8.2 exception in initAndListen: 15926 Insufficient free space for journals, terminating

  [initandlisten] Insufficient free space for journal files
  [initandlisten] Please make at least 3379MB available in /var/lib/mongo/journal or use --smallfiles
  [initandlisten]
  [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
  [initandlisten] now exiting
  [initandlisten] shutdown: going to close listening sockets...
  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
  [initandlisten] shutdown: going to flush diaglog...
  [initandlisten] shutdown: going to close sockets...
  [initandlisten] shutdown: waiting for fs preallocator...
  [initandlisten] shutdown: final commit...
  [initandlisten] shutdown: closing all files...
  [initandlisten] closeAllFiles() finished
  [initandlisten] dbexit:  rc: 100

解决办法，

第一种，添加 --smallfiles 启动
mongod --smallfiles --port 27017 -f /etc/mongod.conf --replSet 20151101 --fork

Sets MongoDB to use a smaller default file size. The --smallfiles option reduces the initial size for data files and limits the maximum size to 512 megabytes. --smallfiles also reduces the size of each journal file from 1 gigabyte to 128 megabytes. Use --smallfiles if you have a large number of databases that each holds a small quantity of data.

The --smallfiles option can lead the mongod instance to create a large number of files, which can affect performance for larger databases.

--smallfiles

第二种，增加 /var/lib/mongo 容量

无损增加容量

https://www.centos.org/docs/5/html/5.2/Deployment_Guide/s2-disk-storage-parted-resize-part.html
http://serverfault.com/questions/644127/centos-6-3-increase-disk-size-on

8.3 No route to host

  [root@localhost kaitong4]# telnet 10.132.1.199 27017
    Trying 10.132.1.199...
    telnet: connect to address 10.132.1.199: No route to host

检查防火墙

在 10.132.1.199 执行

   systemctl stop firewalld

telnet 连接远程的服务器

 [root@localhost kaitong4]# telnet 10.132.1.199 27017
   Trying 10.132.1.199...
   Connected to 10.132.1.199.

注意：保证MongoDB的服务器能相互联通，是设置Replica Set 的基础

8.4 exception in initAndListen: 29 Data directory /var/lib/mongo not found

直接创建生成目录即可


```

  mkdir -p /var/lib/mongo


```

8.5 Permission denied, terminating

[deploy@dev02 wt_db47017]$ tail -f -n 100 /var/log/mongodb/mongod37017.log

2015-11-11T05:29:54.147-0500 I CONTROL  ***** SERVER RESTARTED *****
2015-11-11T05:29:54.194-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-11-11T05:29:54.259-0500 E STORAGE  [initandlisten] WiredTiger (13) [1447237794:259729][17432:0x7f94a378ec80]: /data/wt_db37017/journal/WiredTigerLog.0000000005: Permission denied
2015-11-11T05:29:54.264-0500 I -        [initandlisten] Assertion: 28595:13: Permission denied
2015-11-11T05:29:54.264-0500 I STORAGE  [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2015-11-11T05:29:54.264-0500 I CONTROL  [initandlisten] dbexit:  rc: 100
^C

**解决方式**
[deploy@dev02 wt_db47017]$ sudo chown -R deploy /data/wt_db37017/
[deploy@dev02 wt_db47017]$ /usr/bin/mongod -f /etc/mongod37017.conf
about to fork child process, waiting until server is ready for connections.
forked process: 17502
child process started successfully, parent exiting

此方法也适用于 ** ERROR: Cannot write pid file to /var/run/mongodb/mongod27017.pid: Permission denied**
的这种方式。

8.6 重新设置replica set时，使用force，强制执行

rs.initiate(ctf)
{
"info" : "try querying local.system.replset to see current configuration",
"ok" : 0,
"errmsg" : "already initialized",
"code" : 23
}
rs.reconfig(ctf)
{
"ok" : 0,
"errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the "force" argument to override",
"code" : 10107
}
rs.reconfig(ctf, force: true)
2016-02-23T16:23:40.837+0800 E QUERY SyntaxError: Unexpected token :
rs.reconfig({ctf, force: true})
2016-02-23T16:23:48.546+0800 E QUERY SyntaxError: Unexpected token ,
rs.reconfig(ctf, { force: true})
{ "ok" : 1 }

9 线上部署步骤流程图

其中 A 节点是已经运行，线上服务器
B、C是新增的备份节点。

参考

mongodb 官方文档