linux 下搭建rokectMQ集群

作者: 逃离魔爪 | 来源:发表于2017-12-23 15:55 被阅读64次

linux 下搭建rokectMQ集群
ActiveMQ集群
Linux系统，Zookeeper3.5多实例设置开机启动
Linux-dubbo集群搭建
Linux下搭建MySQL集群
Linux下搭建consul集群
redis 5.0.5 搭建集群
[K8S系列一]基于VirtualBox和Vagrant的Lin
hadoop集群环境搭建之伪分布式集群环境搭建（一）
2018-09-16;集群搭建;

官网地址：https://github.com/alibaba/RocketMQ/releases/tag/v3.4.6

用三台机器搭建集群，3 namesrv、3 master、3 slave：

ip1    nameSrv1     broker-a      broker-c-s
ip2    nameSrv2     broker-a-s    broker-b
ip3    nameSrv3     broker-b-s    broker-c

或者三台机器搭建，3 namesrv、2 master、2 slave：

ip1    namesrv1   broker-a
ip2    namesrv2   broker-a-s   broker-b-s
ip3    namesrv3   broker-b     rocketmq-console

这样保证高可用，各自的从机放在另一台机子上，如果一台机子宕机了，那master是会自动切换，另一个slave还可以消费宕机的那台的master的消息。

主配置文件 conf/2m-2s-async/：

#broker 名称

brokerName=broker-a #改成对应的名称

#broker IDs

brokerId=0 #主为0，从为比0大的数

#监听端口

listenPort=10910

#指定本节点rmq使用的ip，强制使用万兆网卡的ip

#brokerIPI=10.249.130.134
#如果ip乱窜就配置IP地址
brokerIP1=192.168.206.130

#消息存放地址

storePathRootDir=/root/app/rocketmq-3.4.6/rocketmq-log-data/store

storePathCommitLog=/root/app/rocketmq-3.4.6/rocketmq-log-data/commitlog

brokerRole=ASYNC_MASTER

# namesrv  地址 namesrv1:port;namesrv2:portnamesrv3:port

namesrvAddr=192.168.206.128:9876;192.168.206.129:9876;192.168.206.130:9876

#broker刷盘策略   同步刷盘SYNC_MASTER 异步刷盘ASYNC_MASTER

#清除磁盘策略  同步清除刷盘 SYNC_FLUSH  异步清除刷盘 ASYNC_FLUSH

flushDiskType=ASYNC_FLUSH

#文件保留时间，14天

fileReservedTime=336

####################################基本不变配置######################################

#是否自动创建TOPIC

autoCreateTopicEnable=true

#是否自动创建订阅组

autoCreateSubscriptionGroup=true

#生产则默认最大队列数 默认8

defaultTopicQueueNums=16

#ConsumeQueue每个文件大小 默认存储30W条消息

mapedFileSizeConsumeQueue=50000000

#发消息线程池数量 （默认：16 + Runtime.getRuntime().availableProcessors() * 4）

sendMessageThreadPoolNums=128

#拉消息线程池数量 （默认：16 + Runtime.getRuntime().availableProcessors() * 4）

pullMessageThreadPoolNums=128

从配置文件 conf/2m-2s-async/：

#broker 名称

brokerName=broker-a #改成对应的名称

#broker IDs

brokerId=1  #主为0，从为比0大的数

#监听端口

listenPort=20910

#指定本节点rmq使用的ip，强制使用万兆网卡的ip

#brokerIPI=10.249.130.134

#如果ip乱窜就配置IP地址
brokerIP1=192.168.206.130

#消息存放地址

storePathRootDir=/root/app/rocketmq-3.4.6/rocketmq-log-data-s/store

storePathCommitLog=/root/app/rocketmq-3.4.6/rocketmq-log-data-s/commitlog

brokerRole=SLAVE

# namesrv  地址 namesrv1:port;namesrv2:portnamesrv3:port

namesrvAddr=192.168.206.128:9876;192.168.206.129:9876;192.168.206.130:9876

#broker刷盘策略   同步刷盘SYNC_MASTER 异步刷盘ASYNC_MASTER

#清除磁盘策略  同步清除刷盘 SYNC_FLUSH  异步清除刷盘 ASYNC_FLUSH

flushDiskType=ASYNC_FLUSH

#文件保留时间，14天

fileReservedTime=336

####################################基本不变配置######################################

#是否自动创建TOPIC

autoCreateTopicEnable=true

#是否自动创建订阅组

autoCreateSubscriptionGroup=true

#生产则默认最大队列数 默认8

defaultTopicQueueNums=16

#ConsumeQueue每个文件大小 默认存储30W条消息

mapedFileSizeConsumeQueue=50000000

#发消息线程池数量 （默认：16 + Runtime.getRuntime().availableProcessors() * 4）

sendMessageThreadPoolNums=128

#拉消息线程池数量 （默认：16 + Runtime.getRuntime().availableProcessors() * 4）

pullMessageThreadPoolNums=128

启动时如果报内存的相关的错误，修改runserver.sh、runbroker.sh 文件，更改内存大小：

启动脚本 runRMQ-pda.sh：

jps |grep BrokerStartup |kill -9 `awk '{print $1}'`
jps |grep NamesrvStartup |kill -9 `awk '{print $1}'`
jps |grep Bootstrap |kill -9 `awk '{print $1}'

nohup sh /root/app/rocketmq-3.4.6/alibaba-rocketmq/bin/mqnamesrv &

nohup sh /root/app/rocketmq-3.4.6/alibaba-rocketmq/bin/mqbroker -c /root/app/rocketmq-3.4.6/alibaba-rocketmq/conf/2m-2s-async/broker-a.properties &


nohup sh /root/app/rocketmq-3.4.6/alibaba-rocketmq-s/bin/mqbroker -c /root/app/rocketmq-3.4.6/alibaba-rocketmq-s/conf/2m-2s-async/broker-a-s.properties &

脚本中路径更改成自己的安装路径。
另外日志文件的输出路径也可更改：logback_broker.xml、logback_filtersrv.xml、logback_namesrv.xml、logback_tools.xml。

如 logback_broker.xml 文件配置：

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="DefaultAppender"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/broker_default.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/broker_default.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>5</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
            class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>


    <appender name="RocketmqBrokerAppender"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/broker.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/broker.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>30</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
            class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>

    <appender name="RocketmqStoreAppender"
              class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/store.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/store.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>30</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
                class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>

    <appender name="RocketmqRemotingAppender"
              class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/remoting.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/remoting.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>30</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
                class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>

    <appender name="RocketmqStoreErrorAppender"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/storeerror.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/storeerror.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>30</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
            class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>


    <appender name="RocketmqTransactionAppender"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/transaction.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/transaction.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>5</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
            class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>


    <appender name="RocketmqRebalanceLockAppender"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/lock.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/lock.%i.log
            </fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>5</maxIndex>
        </rollingPolicy>
        <triggeringPolicy
            class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>100MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>

    <appender name="RocketmqStatsAppender"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/stats.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/root/app/rocketmq-3.4.6/rocketmq-log-data/logs/rocketmqlogs/otherdays/stats-%d{yyyy-MM-dd}.%i.log
            </fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy
                class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>500MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <MaxHistory>10</MaxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <append>true</append>
        <encoder>
            <pattern>%d{yyy-MM-dd HH\:mm\:ss,GMT+8} %p %t - %m%n</pattern>
            <charset class="java.nio.charset.Charset">UTF-8</charset>
        </encoder>
    </appender>

    <logger name="RocketmqBroker" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqBrokerAppender" />
    </logger>

    <logger name="RocketmqCommon" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqBrokerAppender" />
    </logger>

    <logger name="RocketmqStore" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqStoreAppender" />
    </logger>

    <logger name="RocketmqStoreError" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqStoreErrorAppender" />
    </logger>

    <logger name="RocketmqTransaction" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqTransactionAppender" />
    </logger>

    <logger name="RocketmqRebalanceLock" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqRebalanceLockAppender" />
    </logger>

    <logger name="RocketmqRemoting" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqRemotingAppender" />
    </logger>

    <logger name="RocketmqStats" additivity="false">
        <level value="INFO" />
        <appender-ref ref="RocketmqStatsAppender" />
    </logger>

    <root>
        <level value="INFO" />
        <appender-ref ref="DefaultAppender" />
    </root>
</configuration>

linux 下搭建rokectMQ集群
官网地址：https://github.com/alibaba/RocketMQ/releases/tag/v3....
ActiveMQ集群
一、安装JDK 请参考：Linux下快速安装JDK 二、搭建zookeeper集群请参考：zookeeper集群...
Linux系统，Zookeeper3.5多实例设置开机启动
Linux系统下，搭建ZooKeeper集群[https://www.jianshu.com/p/8d945069...
Linux-dubbo集群搭建
上一篇文章Linux-zookeeper集群已经搭建成功，现在以此为基础继续搭建Linux-dubbo集群上一篇文...
Linux下搭建MySQL集群
一、MySQL集群简介 1、什么是MySQL集群 MySQL集群是一个无共享的(shared-nothing)、分...
Linux下搭建consul集群
伏法朝朝忧闷，强梁夜夜欢歌，损人利己骑马骡，正直公停挨饿，修桥补路瞎眼，杀人放火儿多，我到西天问我佛，佛说……我也...
redis 5.0.5 搭建集群
# redis 5.0.5 搭建集群环境：CentOS Linux release 7.7.1908 (Core...
[K8S系列一]基于VirtualBox和Vagrant的Lin
为搭建K8S集群，首先要准备Linux集群环境，这里是在Windows 10环境下，基于VirtualBox和Va...
hadoop集群环境搭建之伪分布式集群环境搭建（一）
hadoop集群环境搭建之伪分布式集群环境搭建（一） 1、Linux基本环境配置 1.1 虚拟机网络模式选择NAT...
2018-09-16;集群搭建;
1.Mysql主从复制集群搭建： 1.先在Linux下安装好Mysql数据库; 2.然后输入...