mongodb.png

Kubernetes部署Mongodb高可用集群

Mongo介绍

Mongodb是时下流行的NoSql数据库，它的存储方式是文档式存储，并不是Key-Value形式。关于Mongodb的特点，这里就不多介绍了，大家查看官方说明。

Mongodb高可用集群方案介绍

Replica set 模式

mongo_replica_set.png

如上图所示是mongodb 的 Replica Set集群模式，主备节点存储数据，仲裁节点不存储数据。客户端同时连接主节点与备节点，不连接仲裁节点。

默认设置下，主节点提供所有增删查改服务，备节点不提供任何服务。但是可以通过设置使备节点提供查询服务，这样就可以减少主节点的压力，当客户端进行数据查询时，请求自动转到备节点上。这个设置叫做Read Preference Modes，同时客户端提供了简单的配置方式，可以不必直接对数据库进行操作。

仲裁节点是一种特殊的节点，它本身并不存储数据，主要的作用是决定哪一个备节点在主节点挂掉之后提升为主节点，所以客户端不需要连接此节点。这里虽然只有一个备节点，但是仍然需要一个仲裁节点来提升备节点级别。

Mongodb的Replica Set即副本集方式主要有两个目的，一个是数据冗余做故障恢复使用，当发生硬件故障或者其它原因造成的宕机时，可以使用副本进行恢复。另一个是做读写分离，读的请求分流到副本上，减轻主（Primary）的读压力。

Master-Slaver 模式

这个是最简答的集群搭建，不过准确说也不能算是集群，只能说是主备。并且官方已经不推荐这种方式。

Sharding 模式

sharding模式最为完备，和Replica Set类似，都需要一个仲裁节点，但是Sharding还需要配置节点和路由节点。就三种集群搭建方式来说，这种是最复杂的。

Mongodb Kubernetes 部署(Replica set 模式)

注： 部署MongoDB主要用到的是mongo-db-sidecar。来设置replica set集群模式。

创建headless service

apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    app: mongo
spec:
  ports:
    - name: mongo
      port: 27017
      targetPort: 27017
  clusterIP: None
  selector:
    app: mongo

创建StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mongodb-data
provisioner: fuseim.pri/ifs

创建Statefulset

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  selector:
    matchLabels:
      app: mongo
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        app: mongo
        role: mongo
        environment: prod
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: 10.19.85.58:6666/mongo:latest
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: 10.19.85.58:6666/mongo-k8s-sidecar:latest
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=prod"
          resources:
            limits:
              memory: "{{ .ContainerMemory }}Gi"
              cpu: "{{ .ContainerCPU }}"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: mongodb-data
      resources:
        requests:
          storage: 10Gi

创建并查看以上服务

查看service服务

# kubectl get service  -n mongo
NAME            CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
mongo           None            <none>        27017/TCP         2m

查看mongo服务

# kubectl get pod -n mongo
NAME        READY     STATUS    RESTARTS   AGE
mongo-0     2/2       Running   0          2m
mongo-1     2/2       Running   2          2m
mongo-2     2/2       Running   0          2m 

查看集群状态

#kubectl exec -it mongo-0  -- mongo -n mongo
2020-04-24T11:17:33.030+0000 I CONTROL  [initandlisten] 
2020-04-24T11:17:33.030+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-04-24T11:17:33.030+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2020-04-24T11:17:33.030+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2020-04-24T11:17:33.030+0000 I CONTROL  [initandlisten] 
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] 
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] 
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2020-04-24T11:17:34.040+0000 I CONTROL  [initandlisten] 
> rs.status()
{
        "info" : "run rs.initiate(...) if not yet done for the set",
        "ok" : 0,
        "errmsg" : "no replset config has been received",
        "code" : 94,
        "codeName" : "NotYetInitialized"
}

结果显示集群状态不可以，没有组建成集群。

查看日志结果如下：

#kubectl logs mongo-0 mongo-sidecar -n mongo
...
Error in workloop { [Error: [object Object]]
message:
{ kind: 'Status',
    apiVersion: 'v1',
    metadata: {},
    status: 'Failure',
    message:
    'pods is forbidden: User "system:serviceaccount:mongo:default" cannot list resource "pods" in API group "" at the cluster scope',
    reason: 'Forbidden',
    details: { kind: 'pods' },
    code: 403 },
statusCode: 403 }

日志信息显示默认分配的sa账号没有list此namespace下pods的权限。

namespace 权限问题解决方案

对默认的clusterrole view权限进行赋权，我们可以使用clusterrole对sa特定namespace进行赋权，如下ClusterRoleBinding yaml方案：

apiVersion: v1
kind: ClusterRoleBinding
metadata:
  name: view
roleRef:
  name: view
subjects:
- kind: ServiceAccount
  name: default
  namespace: mongo
userNames:
- system:serviceaccount:mongo:default

添加完该namespace权限之后，再次进行查看。

kubectl exec -it mongo-0  -- mongo -n mongo
...
rs0:PRIMARY> rs.status()
{
        "set" : "rs0",
        "date" : ISODate("2020-04-23T03:30:31.502Z"),
        "myState" : 1,
        "term" : NumberLong(3),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 2,
        "writeMajorityCount" : 2,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1587612621, 1),
                        "t" : NumberLong(3)
                },
                "lastCommittedWallTime" : ISODate("2020-04-23T03:30:21.699Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1587612621, 1),
                        "t" : NumberLong(3)
                },
                "readConcernMajorityWallTime" : ISODate("2020-04-23T03:30:21.699Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1587612621, 1),
                        "t" : NumberLong(3)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1587612621, 1),
                        "t" : NumberLong(3)
                },
                "lastAppliedWallTime" : ISODate("2020-04-23T03:30:21.699Z"),
                "lastDurableWallTime" : ISODate("2020-04-23T03:30:21.699Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1587612591, 1),
        "lastStableCheckpointTimestamp" : Timestamp(1587612591, 1),
        "electionCandidateMetrics" : {
                "lastElectionReason" : "electionTimeout",
                "lastElectionDate" : ISODate("2020-04-22T10:42:49.199Z"),
                "electionTerm" : NumberLong(3),
                "lastCommittedOpTimeAtElection" : {
                        "ts" : Timestamp(1587552113, 1),
                        "t" : NumberLong(1)
                },
                "lastSeenOpTimeAtElection" : {
                        "ts" : Timestamp(1587552113, 1),
                        "t" : NumberLong(1)
                },
                "numVotesNeeded" : 2,
                "priorityAtElection" : 1,
                "electionTimeoutMillis" : NumberLong(10000),
                "numCatchUpOps" : NumberLong(0),
                "newTermStartDate" : ISODate("2020-04-22T10:42:49.910Z"),
                "wMajorityWriteAvailabilityDate" : ISODate("2020-04-22T10:42:50.777Z")
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "171.248.3.105:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 2012,
                        "optime" : {
                                "ts" : Timestamp(1587612621, 1),
                                "t" : NumberLong(3)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1587612621, 1),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2020-04-23T03:30:21Z"),
                        "optimeDurableDate" : ISODate("2020-04-23T03:30:21Z"),
                        "lastHeartbeat" : ISODate("2020-04-23T03:30:29.622Z"),
                        "lastHeartbeatRecv" : ISODate("2020-04-23T03:30:29.622Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "171.248.1.86:27017",
                        "syncSourceHost" : "171.248.1.86:27017",
                        "syncSourceId" : 2,
                        "infoMessage" : "",
                        "configVersion" : 143974
                },
                {
                        "_id" : 1,
                        "name" : "171.248.2.3:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 60495,
                        "optime" : {
                                "ts" : Timestamp(1587612621, 1),
                                "t" : NumberLong(3)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1587612621, 1),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2020-04-23T03:30:21Z"),
                        "optimeDurableDate" : ISODate("2020-04-23T03:30:21Z"),
                        "lastHeartbeat" : ISODate("2020-04-23T03:30:31.410Z"),
                        "lastHeartbeatRecv" : ISODate("2020-04-23T03:30:31.408Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "171.248.1.86:27017",
                        "syncSourceHost" : "171.248.1.86:27017",
                        "syncSourceId" : 2,
                        "infoMessage" : "",
                        "configVersion" : 143974
                },
                {
                        "_id" : 2,
                        "name" : "171.248.1.86:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 60503,
                        "optime" : {
                                "ts" : Timestamp(1587612621, 1),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2020-04-23T03:30:21Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1587552169, 1),
                        "electionDate" : ISODate("2020-04-22T10:42:49Z"),
                        "configVersion" : 143974,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1587612621, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1587612621, 1)
}

此时集群状态显示正常节点同步信息也正常。