美文网首页Druid
apache druid(时序数据库)集群搭建

apache druid(时序数据库)集群搭建

作者: 不给起这个名字 | 来源:发表于2020-07-24 11:51 被阅读0次

    准备环境

    mysql、zookeeper、hadoop(HDFS)环境
    jdk1.8(每台druid服务器都需要安装)

    druid的服务器列表

    服务器地址 服务器角色 备注
    10.32.10.240 druid-query Broker、Router
    10.32.10.241 druid-query Broker、Router
    10.32.10.242 druid-master coordinator、overlord
    10.32.10.243 druid-master coordinator、overlord
    10.32.10.244 druid-data Historical、MiddleManager
    10.32.10.245 druid-data Historical、MiddleManager

    开始搭建druid集群

    下载apache-druid-0.18.0-bin.tar.gz并上传到服务器(/mnt/druid)

    所有服务器都需要操作

    进入/mnt/druid，解压tar包

      tar -zxvf apache-druid-0.18.0-bin.tar.gz

    进行druid的集群配置共公目录/mnt/druid/apache-druid-0.18.0/conf/druid/cluster/_common

      cd apache-druid-0.18.0/conf/druid/cluster/_common/

    配置common.runtime.properties

      vim common.runtime.properties

    修改common.runtime.properties配置文件
    druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "mysql->metadata-storage", "druid-hdfs-storage", "druid-kafka-indexing-service"]
druid.host=10.32.10.240  #本机服务器地址
### zookeeper配置
druid.zk.service.host=192.168.11.111:2181,192.168.11.112:2181,192.168.11.113:2181
druid.zk.paths.base=/druid018
# Metadata storage
# For MySQL (make sure to include the MySQL JDBC driver on the classpath):
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://127.0.0.1:3306/druid018?characterEncoding=UTF8
druid.metadata.storage.connector.user=root
druid.metadata.storage.connector.password=root
# For HDFS:
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://10.32.10.16:8020/druid018/segments
#
# Indexing service logs
#
# For HDFS:
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=hdfs://10.32.10.16:8020/druid018/indexing-logs
    修改完成后，创建druid日志目录

      mkdir -p /mnt/druid/apache-druid-0.18.0/var/sv/

    引入以下mysql包到apache-druid-0.18.0/extensions/mysql-metadata-storage目录

    mysql-connector-java-5.1.48.jar、password-connector-java-5.1.48.jar

    配置master coordinator-overlord（242 243 节点同步操作)

    进行配置文件目录
      cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/master/coordinator-overlord
    jvm.config配置

    -server
-Xms15g
-Xmx15g
-XX:+ExitOnOutOfMemoryError
-XX:+UseG1GC
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Dderby.stream.error.file=var/druid/derby.log

    runtime.properties配置

    druid.service=druid/coordinator
druid.plaintextPort=8081
druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

    启动druid的master节点
    /apache-druid-0.18.0/bin/start-cluster-master-no-zk-server &

    配置 data节点 MiddleManager Historical （244 245节点同时操作）

    MiddleManager配置：
    进行配置文件目录
      cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/data/middleManager
    jvm.config配置

    -server
-Xms1g
-Xmx1g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

    runtime.properties配置

    druid.service=druid/middleManager
druid.plaintextPort=8091
# Number of tasks per middleManager
druid.worker.capacity=4
# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g ->Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError ->Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task
# HTTP server threads
druid.server.http.numThreads=60
# Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000
druid.indexer.fork.property.druid.processing.numThreads=1
# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

    Historical配置：
    进行配置文件目录
      cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/data/historical
    jvm.config配置

    -server
-Xms8g
-Xmx8g
-XX:MaxDirectMemorySize=13g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
runtime.properties配置
druid.service=druid/historical
druid.plaintextPort=8083
# HTTP server threads
druid.server.http.numThreads=60
# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=4     #1:4 cpu核数
druid.processing.numThreads=15      # cpu核数-1
druid.processing.tmpDir=var/druid/processing
# Segment storage
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":300000000000}]
druid.server.maxSize=300000000000
# Query cache
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=256000000

    启动命令：
    /apache-druid-0.18.0/bin/start-cluster-data-server &

    配置 query节点 broker & router(240 241节点同时操作）

    borker配置：
    进行配置文件目录
      cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/query/broker
    jvm.config配置

    -server
-Xms12g
-Xmx12g
-XX:MaxDirectMemorySize=6g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
runtime.properties配置
druid.service=druid/broker
druid.plaintextPort=8082
# HTTP server settings
druid.server.http.numThreads=60
# HTTP client settings
druid.broker.http.numConnections=50
druid.broker.http.maxQueuedBytes=10000000
# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=6
druid.processing.numThreads=1
druid.processing.tmpDir=var/druid/processing
# Query cache disabled -- push down caching and merging instead
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false

    borker配置：
    进行配置文件目录
      cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/query/router
    jvm.config配置

    -server
-Xms1g
-Xmx1g
-XX:+UseG1GC
-XX:MaxDirectMemorySize=128m
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
runtime.properties配置
druid.service=druid/router
druid.plaintextPort=8888
# HTTP proxy
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.server.http.numThreads=100
# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
# Management proxy to coordinator / overlord: required for unified web console.
druid.router.managementProxy.enabled=true

    启动命令：
    /apache-druid-0.18.0/bin/start-cluster-query-server &

    配置完成后，由druid的query节点的8888端口提供可视化页面，可以通过nginx来反向代理10.32.10.240:8888和10.32.10.241:8888

    注意：启动完成后要通过exit退出服务器连接，不能直接关闭窗口或断开连接，不然druid的后台进程也会被关闭

    相关文章

      网友评论

        本文标题：apache druid(时序数据库)集群搭建

        本文链接：https://www.haomeiwen.com/subject/mhftlktx.html

        延伸阅读

        深度阅读

        • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

        栏目导航

        热点阅读

        Druid

        关于我们|服务条款|联系我们|apache druid(时序数据库)集群搭建|投稿指南|网站地图|RSS订阅|排版工具|手机版

        提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

        Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

        备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

        本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

        所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！