美文网首页
Hazelcast4.2.2 集群测试百万数据下的故障转移

Hazelcast4.2.2 集群测试百万数据下的故障转移

作者: 冬天里的懒喵 | 来源:发表于2021-11-01 11:09 被阅读0次

    本文测试在Hazelcast集群中，写入百万级别的数据，对其故障转移过程进行测试。

    1.环境准备

    Hazelcast4.2.2 在springboot下配置3节点集群并测试对map的操作的基础上，
    搭建3节点的Hazelcast集群。
    springboot启动的端口为 8081、8082、8083.

    2.写入数据

    写入数据代码:

    package com.dhb.hazelcast.demo.client;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.core.HazelcastInstance;
import lombok.extern.slf4j.Slf4j;
import org.springframework.util.StopWatch;

import java.util.Map;

@Slf4j
public class BuildTestData {
    public static final int count = 1000000;

    public static void main(String[] args) {
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.setClusterName("hazelcast-cluster");
        HazelcastInstance instance = HazelcastClient.newHazelcastClient(clientConfig);
        Map<Integer, String> clusterMap = instance.getMap("map");
        StopWatch stopWatch = new StopWatch();
        log.info("begin put data ...");
        int k = 0;
        stopWatch.start(k + "");
        for (int i = 0; i < count; i++) {
            clusterMap.put(i, i + "");
            int j = i / 10000;
            if (j > 0 && j > k) {
                k = j;
                stopWatch.stop();
                stopWatch.start(k + "");
                log.info("已插入数据 {} 万条。", k);
            }
        }
        stopWatch.stop();
        log.info("end put data ... {}", stopWatch.prettyPrint());
        instance.shutdown();
    }
}

    然后执行上述脚本:

    10:06:47.471 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - begin put data ...
10:06:49.432 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 1 万条。
10:06:50.704 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 2 万条。
10:06:52.278 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 3 万条。
10:06:54.255 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 4 万条。
10:06:55.782 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 5 万条。
10:06:57.412 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 6 万条。
10:06:59.107 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 7 万条。
... ...
10:09:19.057 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 93 万条。
10:09:21.204 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 94 万条。
10:09:22.856 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 95 万条。
10:09:24.406 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 96 万条。
10:09:25.958 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 97 万条。
10:09:27.434 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 98 万条。
10:09:28.905 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - 已插入数据 99 万条。
10:09:30.377 [main] INFO com.dhb.hazelcast.demo.client.BuildTestData - end put data ... StopWatch '': running time = 161431786040 ns

    数据插入完毕。

    3.关闭一个节点

    将8083节点关闭。
    节点8081和8082 检测到心跳丢失，不能连接到8003节点，集群将8003节点移除。

    2021-11-01 10:24:32.832 DEBUG 5792 --- [Server.thread-1] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:7 is already in progress
2021-11-01 10:24:32.832 DEBUG 5792 --- [Server.thread-4] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:11 is already in progress
2021-11-01 10:24:32.832 DEBUG 5792 --- [Server.thread-3] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:10 is already in progress
2021-11-01 10:24:33.032  INFO 5792 --- [cached.thread-5] c.h.i.server.tcp.TcpServerConnector      : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Could not connect to: /192.168.87.1:5703. Reason: IOException[Connection refused: no further information to address /192.168.87.1:5703]
2021-11-01 10:24:33.134 DEBUG 5792 --- [cached.thread-5] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:0 is not yet progress
2021-11-01 10:24:38.340 DEBUG 5792 --- [Server.thread-4] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:63 is already in progress
2021-11-01 10:24:38.364 DEBUG 5792 --- [ration.thread-0] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Setting members [Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab, Member [192.168.87.1]:5702 - 4a3e736b-2bd6-462f-b42d-e83549fdfd2e this], version: 4
2021-11-01 10:24:38.364 DEBUG 5792 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Removing Member [192.168.87.1]:5703 - 125bb821-4624-4968-94ff-79baba552a2d
2021-11-01 10:24:38.368  INFO 5792 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] 

Members {size:2, ver:4} [
    Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab
    Member [192.168.87.1]:5702 - 4a3e736b-2bd6-462f-b42d-e83549fdfd2e this
]

    移除之后，将原有副本中的副本分片数据，提升为主分片，在8082节点中，有52个分片数据需要提升为主分片。

    2021-11-01 10:24:38.375  INFO 5792 --- [cached.thread-3] c.h.t.TransactionManagerService          : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Committing/rolling-back live transactions of [192.168.87.1]:5703, UUID: 125bb821-4624-4968-94ff-79baba552a2d
2021-11-01 10:24:38.403 DEBUG 5792 --- [ration.thread-0] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Submitting BeforePromotionOperations for 51 promotions. Promotion partition state stamp: 4888381698555613112, current partition state stamp: -7459311775531378332
2021-11-01 10:24:38.404 DEBUG 5792 --- [ration.thread-3] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applied partition state update with stamp: 4593659232416938281, Local stamp is: 4593659232416938281
2021-11-01 10:24:38.404 DEBUG 5792 --- [ration.thread-3] c.h.i.p.o.PartitionStateOperation        : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applied new partition state. Stamp: 4593659232416938281, caller: [192.168.87.1]:5701
2021-11-01 10:24:38.408 DEBUG 5792 --- [ration.thread-1] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] All before promotion tasks are completed. Starting finalize promotion tasks...
2021-11-01 10:24:38.410 DEBUG 5792 --- [ration.thread-1] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applied partition state update with stamp: 4888381698555613112, Local stamp is: 4888381698555613112
2021-11-01 10:24:38.410 DEBUG 5792 --- [ration.thread-1] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Submitting FinalizePromotionOperations for 51 promotions. Result: true. Promotion partition state stamp: 4888381698555613112, current partition state stamp: 4888381698555613112
2021-11-01 10:24:38.412 DEBUG 5792 --- [ration.thread-1] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] All finalize promotion tasks are completed.

    之后由于主分片数据没有副本，现在建立新的副本：

    2021-11-01 10:24:39.068 DEBUG 5792 --- [ration.thread-1] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applied completed migrations with partition state stamp: 8666519595082550993
2021-11-01 10:24:39.071 DEBUG 5792 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applying completed migration MigrationInfo{uuid=4493a55c-672b-4a62-aa3c-363550b9e485, partitionId=269, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab, destinationCurrentReplicaIndex=2, destinationNewReplicaIndex=1, master=[192.168.87.1]:5701, initialPartitionVersion=6, partitionVersionIncrement=2, status=SUCCESS}
2021-11-01 10:24:39.071 DEBUG 5792 --- [ration.thread-0] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Scheduled finalization of MigrationInfo{uuid=4493a55c-672b-4a62-aa3c-363550b9e485, partitionId=269, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab, destinationCurrentReplicaIndex=2, destinationNewReplicaIndex=1, master=[192.168.87.1]:5701, initialPartitionVersion=6, partitionVersionIncrement=2, status=SUCCESS}
2021-11-01 10:24:39.071 DEBUG 5792 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applying completed migration MigrationInfo{uuid=8563b5cb-7127-4e5f-a366-c3965915e61d, partitionId=263, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab, destinationCurrentReplicaIndex=2, destinationNewReplicaIndex=1, master=[192.168.87.1]:5701, initialPartitionVersion=6, partitionVersionIncrement=2, status=SUCCESS}
2021-11-01 10:24:39.071 DEBUG 5792 --- [ration.thread-0] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Scheduled finalization of MigrationInfo{uuid=8563b5cb-7127-4e5f-a366-c3965915e61d, partitionId=263, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab, destinationCurrentReplicaIndex=2, destinationNewReplicaIndex=1, master=[192.168.87.1]:5701, initialPartitionVersion=6, partitionVersionIncrement=2, status=SUCCESS}
2021-11-01 10:24:39.071 DEBUG 5792 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applied completed migrations with partition state stamp: -802415193368779066
2021-11-01 10:24:39.244 DEBUG 5792 --- [Server.thread-1] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:65 is already in progress
2021-11-01 10:24:39.244 DEBUG 5792 --- [Server.thread-2] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:63 is already in progress
2021-11-01 10:24:39.246 DEBUG 5792 --- [Server.thread-4] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5703 streamId:69 is already in progress
2021-11-01 10:24:42.076 DEBUG 5792 --- [ration.thread-0] c.h.i.c.impl.ClusterHeartbeatManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Received heartbeat from Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab (now: 2021-11-01 10:24:42.075, timestamp: 2021-11-01 10:24:42.076)

    这样两个节点构成的集群运行成功：

    2021-11-01 10:33:57.076 DEBUG 5792 --- [ration.thread-0] c.h.i.c.impl.ClusterHeartbeatManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Received heartbeat from Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab (now: 2021-11-01 10:33:57.074, timestamp: 2021-11-01 10:33:57.075)
2021-11-01 10:33:57.076 DEBUG 5792 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Setting cluster time diff to -1ms.
2021-11-01 10:34:00.021 DEBUG 5792 --- [cached.thread-1] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Cannot publish member list to cluster. Is-master: false, joined: true , mastership claim in progress: false
2021-11-01 10:34:02.076 DEBUG 5792 --- [ration.thread-0] c.h.i.c.impl.ClusterHeartbeatManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Received heartbeat from Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab (now: 2021-11-01 10:34:02.075, timestamp: 2021-11-01 10:34:02.076)
2021-11-01 10:34:02.076 DEBUG 5792 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Setting cluster time diff to 0ms.
2021-11-01 10:34:07.077 DEBUG 5792 --- [ration.thread-0] c.h.i.c.impl.ClusterHeartbeatManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Received heartbeat from Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab (now: 2021-11-01 10:34:07.077, timestamp: 2021-11-01 10:34:07.076)
2021-11-01 10:34:07.077 DEBUG 5792 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Setting cluster time diff to -1ms.

    节点8081与上述过程类似：
    现在通过客户端测试，数据有没有丢失：

    10:34:45.851 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - DynamicClusterConfig{instance=com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl@461ad730}
10:34:45.853 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - Map Value:IMap{name='map'}
10:34:45.856 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - Map Size :1000000

    可以看到，map中的数据仍然是100万。
    测试脚本如下：

    package com.dhb.hazelcast.demo.client;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.core.HazelcastInstance;
import lombok.extern.slf4j.Slf4j;

import java.util.Map;

@Slf4j
public class HazelcastGetStartClient {

    public static void main(String[] args) {
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.setClusterName("hazelcast-cluster");
        HazelcastInstance instance = HazelcastClient.newHazelcastClient(clientConfig);
        Map<Integer, String> clusterMap = instance.getMap("map");
//      Queue<String> clusterQueue = instance.getQueue("MyQueue");

        log.info(instance.getConfig().toString());
        log.info("Map Value:" + clusterMap.toString());
        log.info("Map Size :" + clusterMap.size());
        instance.shutdown();
    }
}

    4.关闭第二个节点

    现在将8082节点关闭：
    节点8081检测到8082心跳丢失:

    DEBUG 12524 --- [Server.thread-4] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5702 streamId:47 is already in progress
2021-11-01 10:37:26.208  INFO 12524 --- [cached.thread-2] c.h.i.server.tcp.TcpServerConnector      : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Could not connect to: /192.168.87.1:5702. Reason: IOException[Connection refused: no further information to address /192.168.87.1:5702]
2021-11-01 10:37:26.209  WARN 12524 --- [cached.thread-2] .h.i.s.t.TcpServerConnectionErrorHandler : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Removing connection to endpoint [192.168.87.1]:5702 Cause => java.io.IOException {Connection refused: no further information to address /192.168.87.1:5702}, Error-Count: 5
2021-11-01 10:37:26.209  INFO 12524 --- [cached.thread-2] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Removing Member [192.168.87.1]:5702 - 4a3e736b-2bd6-462f-b42d-e83549fdfd2e
2021-11-01 10:37:26.209 DEBUG 12524 --- [cached.thread-2] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Setting members [Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab this], version: 5
2021-11-01 10:37:26.209 DEBUG 12524 --- [cached.thread-2] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Member [192.168.87.1]:5702 - 4a3e736b-2bd6-462f-b42d-e83549fdfd2e is removed. Publishing new member list.
2021-11-01 10:37:26.209 DEBUG 12524 --- [cached.thread-2] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Sending member list to the non-master nodes: 

Members {size:1, ver:5} [
    Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab this
]

2021-11-01 10:37:26.209 DEBUG 12524 --- [cached.thread-2] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Removing Member [192.168.87.1]:5702 - 4a3e736b-2bd6-462f-b42d-e83549fdfd2e
2021-11-01 10:37:26.209  INFO 12524 --- [cached.thread-2] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] 

Members {size:1, ver:5} [
    Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab this
]

2021-11-01 10:37:26.209  INFO 12524 --- [cached.thread-1] c.h.t.TransactionManagerService          : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Committing/rolling-back live transactions of [192.168.87.1]:5702, UUID: 4a3e736b-2bd6-462f-b42d-e83549fdfd2e

    由于只有一个节点，因此只需要将之前的副本分片直接升级为主分片即可，不用在保持副本。

    2021-11-01 10:37:26.222 DEBUG 12524 --- [ustin.migration] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Submitting BeforePromotionOperations for 141 promotions. Promotion partition state stamp: -2004286217543412478, current partition state stamp: 949008774052952716
2021-11-01 10:37:26.224 DEBUG 12524 --- [ration.thread-3] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] All before promotion tasks are completed. Starting finalize promotion tasks...
2021-11-01 10:37:26.240 DEBUG 12524 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Applied partition state update with stamp: -2004286217543412478, Local stamp is: -2004286217543412478
2021-11-01 10:37:26.241 DEBUG 12524 --- [ration.thread-0] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Submitting FinalizePromotionOperations for 141 promotions. Result: true. Promotion partition state stamp: -2004286217543412478, current partition state stamp: -2004286217543412478
2021-11-01 10:37:26.245 DEBUG 12524 --- [ration.thread-3] c.h.i.p.o.PromotionCommitOperation       : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] All finalize promotion tasks are completed.
2021-11-01 10:37:26.246 DEBUG 12524 --- [ustin.migration] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Publishing partition state, stamp: -2004286217543412478
2021-11-01 10:37:26.249 DEBUG 12524 --- [ustin.migration] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Publishing partition state, stamp: -2004286217543412478
2021-11-01 10:37:26.249  INFO 12524 --- [ustin.migration] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Partition balance is ok, no need to repartition.
2021-11-01 10:37:26.400 DEBUG 12524 --- [Server.thread-2] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5702 streamId:46 is not yet progress
2021-11-01 10:37:26.400 DEBUG 12524 --- [Server.thread-1] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5702 streamId:48 is already in progress
2021-11-01 10:37:26.400 DEBUG 12524 --- [Server.thread-2] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5702 streamId:53 is already in progress
2021-11-01 10:37:26.400 DEBUG 12524 --- [Server.thread-2] c.h.i.s.tcp.TcpServerConnectionManager   : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Connection to: [192.168.87.1]:5702 streamId:47 is already in progress

    这样也不需要再发送心跳：

    2021-11-01 10:39:47.074 DEBUG 12524 --- [cached.thread-9] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Checking partition state, stamp: -2004286217543412478
2021-11-01 10:40:02.073 DEBUG 12524 --- [cached.thread-1] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Checking partition state, stamp: -2004286217543412478
2021-11-01 10:40:17.073 DEBUG 12524 --- [cached.thread-6] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Checking partition state, stamp: -2004286217543412478
2021-11-01 10:40:17.076 DEBUG 12524 --- [cached.thread-9] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Sending member list to the non-master nodes: 

Members {size:1, ver:5} [
    Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab this
]

2021-11-01 10:40:17.076 DEBUG 12524 --- [MulticastThread] c.h.i.c.impl.NodeMulticastListener       : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Dropped: SplitBrainJoinMessage{packetVersion=4, buildNumber=20210811, memberVersion=4.2.2, clusterVersion=4.2, address=[192.168.87.1]:5701, uuid='3ae8b8f8-2f18-4648-ae05-026a98ca2dab', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=5}

    我们再用测试脚本检测，数据是否为100万：

    10:41:23.584 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - DynamicClusterConfig{instance=com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl@24959ca4}
10:41:23.587 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - Map Value:IMap{name='map'}
10:41:23.594 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - Map Size :1000000

    可见，对于hazelcast集群，具有非常良好的高可用性。当节点宕机之后，会通过副本的形式，确保数据不会丢失。

    5.恢复集群节点

    现在将8082节点恢复：

    2021-11-01 10:44:04.811 DEBUG 13644 --- [ration.thread-0] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Local member list join version is set to 6
2021-11-01 10:44:04.812 DEBUG 13644 --- [ration.thread-0] c.h.i.cluster.impl.MembershipManager     : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Setting members [Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab, Member [192.168.87.1]:5702 - 6e4e036a-e12c-46a3-b79e-dc54593ef68e this], version: 6
2021-11-01 10:44:04.814 DEBUG 13644 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Adding Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab
2021-11-01 10:44:04.818  INFO 13644 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] 

Members {size:2, ver:6} [
    Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab
    Member [192.168.87.1]:5702 - 6e4e036a-e12c-46a3-b79e-dc54593ef68e this
]

2021-11-01 10:44:04.843 DEBUG 13644 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Applied partition state update with stamp: -2004286217543412478, Local stamp is: -2004286217543412478
2021-11-01 10:44:05.041 DEBUG 13644 --- [ration.thread-3] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5702 [hazelcast-cluster] [4.2.2] Already applied partition state update with stamp: -2004286217543412478, Local stamp is: -2004286217543412478

    分片同步完成之后，集群节点启动成功。之后节点之间正常进行心跳。

    2021-11-01 10:44:05.999  INFO 13644 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2021-11-01 10:44:06.316  INFO 13644 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2021-11-01 10:44:06.351  INFO 13644 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8082 (http) with context path ''
2021-11-01 10:44:06.364  INFO 13644 --- [           main] c.d.h.demo.HazelcastNode2Starter         : Started HazelcastNode2Starter in 10.335 seconds (JVM running for 10.82)
2021-11-01 10:44:06.666  INFO 13644 --- [(5)-10.38.1.240] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2021-11-01 10:44:06.666  INFO 13644 --- [(5)-10.38.1.240] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2021-11-01 10:44:06.670  INFO 13644 --- [(5)-10.38.1.240] o.s.web.servlet.DispatcherServlet        : Completed initialization in 4 ms

    现在回复第二个集群节点8083：
    同样，该节点也会发生和8082节点相同的操作:

    2021-11-01 10:49:50.091  INFO 3224 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] 

Members {size:3, ver:7} [
    Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab
    Member [192.168.87.1]:5702 - 6e4e036a-e12c-46a3-b79e-dc54593ef68e
    Member [192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188 this
]

2021-11-01 10:49:50.342 DEBUG 3224 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Already applied partition state update with stamp: 3753040894131712953, Local stamp is: 3753040894131712953
2021-11-01 10:49:50.342 DEBUG 3224 --- [ration.thread-0] c.h.i.p.o.PartitionStateOperation        : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Applied new partition state. Stamp: 3753040894131712953, caller: [192.168.87.1]:5701
2021-11-01 10:49:50.446 DEBUG 3224 --- [ration.thread-0] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Finalizing MigrationInfo{uuid=33d68ed9-fd27-458d-877e-94b3350f0f41, partitionId=4, source=[192.168.87.1]:5702 - 6e4e036a-e12c-46a3-b79e-dc54593ef68e, sourceCurrentReplicaIndex=0, sourceNewReplicaIndex=2, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=0, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=2, status=SUCCESS}
2021-11-01 10:49:50.446 DEBUG 3224 --- [ration.thread-0] c.h.i.p.InternalPartitionService         : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Committed MigrationInfo{uuid=33d68ed9-fd27-458d-877e-94b3350f0f41, partitionId=4, source=[192.168.87.1]:5702 - 6e4e036a-e12c-46a3-b79e-dc54593ef68e, sourceCurrentReplicaIndex=0, sourceNewReplicaIndex=2, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=0, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=2, status=SUCCESS} on destination with partition version: 11
2021-11-01 10:49:50.446 DEBUG 3224 --- [ration.thread-0] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Finalizing MigrationInfo{uuid=08ba199c-59b5-47c2-95a6-096e89d0e9e2, partitionId=2, source=[192.168.87.1]:5702 - 6e4e036a-e12c-46a3-b79e-dc54593ef68e, sourceCurrentReplicaIndex=0, sourceNewReplicaIndex=2, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=0, master=[192.168.87.1]:5701, initialPartitionVersion=7, partitionVersionIncrement=2, status=SUCCESS}
... ... 

2021-11-01 10:49:52.111  INFO 3224 --- [11)-10.38.1.240] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2021-11-01 10:49:52.111  INFO 3224 --- [11)-10.38.1.240] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2021-11-01 10:49:52.116  INFO 3224 --- [11)-10.38.1.240] o.s.web.servlet.DispatcherServlet        : Completed initialization in 5 ms
2021-11-01 10:49:57.077 DEBUG 3224 --- [ration.thread-0] c.h.i.c.impl.ClusterHeartbeatManager     : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Received heartbeat from Member [192.168.87.1]:5701 - 3ae8b8f8-2f18-4648-ae05-026a98ca2dab (now: 2021-11-01 10:49:57.076, timestamp: 2021-11-01 10:49:57.076)
2021-11-01 10:49:57.077 DEBUG 3224 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [192.168.87.1]:5703 [hazelcast-cluster] [4.2.2] Setting cluster time diff to -1ms.

    而节点8081会将其上的主分片副本降级，逐出：

    2021-11-01 10:49:51.251 DEBUG 12524 --- [onPool-worker-1] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Evicting 13 completed migrations.
2021-11-01 10:49:51.251 DEBUG 12524 --- [onPool-worker-3] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] AsyncMigrationTask completed: MigrationInfo{uuid=859ab1fb-0d13-4174-ab3b-97608e8438d9, partitionId=117, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=SUCCESS}
2021-11-01 10:49:51.251 DEBUG 12524 --- [onPool-worker-3] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Evicting 14 completed migrations.
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-7] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] AsyncMigrationTask completed: MigrationInfo{uuid=192f600b-ba2d-4bf0-a95b-dfd0d2a3afa6, partitionId=120, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=11, partitionVersionIncrement=1, status=SUCCESS}
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-6] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Evicting 16 completed migrations.
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-5] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Migration operation response received -> MigrationInfo{uuid=ae691b9c-7046-4c42-8c15-f832fc7fe0f6, partitionId=123, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=ACTIVE}, success: true, failure: null
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-0] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Evicting 15 completed migrations.
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-5] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Finished Migration: MigrationInfo{uuid=ae691b9c-7046-4c42-8c15-f832fc7fe0f6, partitionId=123, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=ACTIVE}
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-7] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Migration commit result true from [192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188 for MigrationInfo{uuid=3625796a-4482-46bf-b3e6-6da2f21e9fc7, partitionId=119, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=SUCCESS}
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-7] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] AsyncMigrationTask completed: MigrationInfo{uuid=3625796a-4482-46bf-b3e6-6da2f21e9fc7, partitionId=119, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=SUCCESS}
2021-11-01 10:49:51.253 DEBUG 12524 --- [onPool-worker-4] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] AsyncMigrationTask completed: MigrationInfo{uuid=026b564c-4dc3-4e1f-b521-27aee8bd0178, partitionId=122, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=11, partitionVersionIncrement=1, status=SUCCESS}
2021-11-01 10:49:51.254 DEBUG 12524 --- [ustin.migration] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Starting Migration: MigrationInfo{uuid=dd9afc64-a01b-4a33-b1b1-d1d09b90338e, partitionId=125, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=ACTIVE}
2021-11-01 10:49:51.254 DEBUG 12524 --- [ustin.migration] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Starting Migration: MigrationInfo{uuid=e4e14c35-d0ed-4f47-b770-5b10f2947a24, partitionId=126, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=7, partitionVersionIncrement=1, status=ACTIVE}
2021-11-01 10:49:51.255 DEBUG 12524 --- [onPool-worker-3] c.h.i.partition.impl.MigrationManager    : [192.168.87.1]:5701 [hazelcast-cluster] [4.2.2] Migration commit result true from [192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188 for MigrationInfo{uuid=ae691b9c-7046-4c42-8c15-f832fc7fe0f6, partitionId=123, source=null, sourceCurrentReplicaIndex=-1, sourceNewReplicaIndex=-1, destination=[192.168.87.1]:5703 - f0aed68c-e4be-4c7f-9437-30d5c5396188, destinationCurrentReplicaIndex=-1, destinationNewReplicaIndex=2, master=[192.168.87.1]:5701, initialPartitionVersion=9, partitionVersionIncrement=1, status=SUCCESS}

    在8082恢复之后，启动8083，在8082之上也会发生相同的类似操作。
    上述操作完成之后，集群将恢复。
    现在再查看其数据：

    10:55:45.983 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - DynamicClusterConfig{instance=com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl@7068e664}
10:55:45.985 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - Map Value:IMap{name='map'}
10:55:45.997 [main] INFO com.dhb.hazelcast.demo.client.HazelcastGetStartClient - Map Size :1000000

    数据仍然是100万。

    相关文章

      网友评论

          本文标题：Hazelcast4.2.2 集群测试百万数据下的故障转移

          本文链接：https://www.haomeiwen.com/subject/tujzaltx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|Hazelcast4.2.2 集群测试百万数据下的故障转移|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！