dolphinscheduler-集群安装

作者: wudl | 来源:发表于2021-08-01 00:38 被阅读0次

1. 整体的架构

1.1 官网

https://dolphinscheduler.apache.org/zh-cn/index.html

1.2 架构图

ds 架构图.png

1.3 源码下载：本文下载的1.3.6 版本

https://github.com/apache/dolphinscheduler/releases

1.4 下载好解压：

unzip  dolphinscheduler-1.3.6.zip

2.修改源码和大数据的版本pom 文件

主要修改hdfs  和hive 的版本对应 ambari  或者cdh  的里面的hive  和 hdfs  的版本
进入 dolphinscheduler-1.3.6
修改：

<hadoop.version>3.1.1</hadoop.version>
<hive.jdbc.version>3.1.0</hive.jdbc.version>

2.1 进行编译：

mvn -U clean package -Prelease -Dmaven.test.skip=true
[INFO] Building tar: /opt/module/dolphinscheduler/dolphinscheduler-1.3.6/dolphinscheduler-dist/target/apache-dolphinscheduler-1.3.6-bin.tar.gz
...............................................
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (src) @ dolphinscheduler-dist ---
[INFO] Reading assembly descriptor: src/main/assembly/dolphinscheduler-src.xml
[INFO] Building tar: /opt/module/dolphinscheduler/dolphinscheduler-1.3.6/dolphinscheduler-dist/target/apache-dolphinscheduler-1.3.6-src.tar.gz
[INFO] 
[INFO] --- maven-javadoc-plugin:2.10.3:jar (attach-javadocs) @ dolphinscheduler-dist ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable package
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for dolphinscheduler 1.3.6:
[INFO] 
[INFO] dolphinscheduler ................................... SUCCESS [  1.360 s]
[INFO] dolphinscheduler-ui ................................ SUCCESS [ 42.789 s]
[INFO] dolphinscheduler-plugin-api ........................ SUCCESS [  1.258 s]
[INFO] dolphinscheduler-common ............................ SUCCESS [01:16 min]
[INFO] dolphinscheduler-dao ............................... SUCCESS [ 45.908 s]
[INFO] dolphinscheduler-remote ............................ SUCCESS [ 12.852 s]
[INFO] dolphinscheduler-service ........................... SUCCESS [ 21.570 s]
[INFO] dolphinscheduler-alert ............................. SUCCESS [ 11.064 s]
[INFO] dolphinscheduler-server ............................ SUCCESS [  5.140 s]
[INFO] dolphinscheduler-api ............................... SUCCESS [01:04 min]
[INFO] dolphinscheduler-skywalking ........................ SUCCESS [01:33 min]
[INFO] dolphinscheduler-dist .............................. SUCCESS [05:47 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12:03 min
[INFO] Finished at: 2021-07-31T18:47:53+08:00
[INFO] ------------------------------------------------------------------------
[root@node01 dolphinscheduler-1.3.6]#

正常编译完后，会在 dolphinscheduler-dist/target ⽬目录下⽣生成如下安装包：

[root@node01 target]# cd /opt/module/dolphinscheduler/dolphinscheduler-1.3.6/dolphinscheduler-dist/target
[root@node01 target]# ll
总用量 230828
-rw-r--r-- 1 root root 234705278 7月  31 18:47 apache-dolphinscheduler-1.3.6-bin.tar.gz
-rw-r--r-- 1 root root   1657273 7月  31 18:47 apache-dolphinscheduler-1.3.6-src.tar.gz
drwxr-xr-x 2 root root         6 7月  31 18:42 archive-tmp
drwxr-xr-x 3 root root        22 7月  31 18:42 maven-shared-archive-resources
[root@node01 target]# pwd
/opt/module/dolphinscheduler/dolphinscheduler-1.3.6/dolphinscheduler-dist/target
[root@node01 target]#

4. 部署规划

服务名	noe01	noe02	noe03
master	✔	✔
worker/logServer		✔	✔
alertServer		✔
apiServer	✔
ui			✔
mysql	✔

5. 安装

[root@node01 dolphinscheduler]# ll
总用量 234428
-rw-r--r--  1 root root 234705278 7月  31 18:57 apache-dolphinscheduler-1.3.6-bin.tar.gz
drwxr-xr-x 22 root root      4096 7月  31 17:59 apache-dolphinscheduler-1.3.6-src
-rw-r--r--  1 root root   1718499 7月  31 17:59 apache-dolphinscheduler-1.3.6-src.tar.gz
drwxr-xr-x 24 root root      4096 7月  31 18:35 dolphinscheduler-1.3.6
-rw-r--r--  1 root root   3610157 7月  31 18:03 dolphinscheduler-1.3.6.zip
drwxr-xr-x 27 root root      4096 7月  31 14:43 incubator-dolphinscheduler
[root@node01 dolphinscheduler]# tar -zxvf apache-dolphinscheduler-1.3.6-bin.tar.gz

5.1. 赋值权限

[root@node01 dolphinscheduler]# cd apache-dolphinscheduler-1.3.6-bin
[root@node01 apache-dolphinscheduler-1.3.6-bin]# ll
总用量 156
drwxr-xr-x 2 root root    79 7月  31 18:58 bin
drwxr-xr-x 5 root root  4096 7月  31 18:58 conf
drwxr-xr-x 3 root root    30 7月  31 18:58 ext
-rwxr-xr-x 1 root root  4663 4月  25 11:13 install.sh
drwxr-xr-x 2 root root 12288 7月  31 18:58 lib
-rw-r--r-- 1 root root 38823 4月  25 11:13 LICENSE
drwxr-xr-x 3 root root 12288 7月  31 18:58 licenses
-rw-r--r-- 1 root root 70929 4月  25 11:13 NOTICE
drwxr-xr-x 2 root root   218 7月  31 18:58 script
drwxr-xr-x 4 root root   125 7月  31 18:58 sql
drwxr-xr-x 8 root root   113 7月  31 18:58 ui
[root@node01 apache-dolphinscheduler-1.3.6-bin]# chmod ugo+x bin/*
[root@node01 apache-dolphinscheduler-1.3.6-bin]# chmod ugo+x script/*
[root@node01 apache-dolphinscheduler-1.3.6-bin]# chmod ugo+x install.sh
[root@node01 apache-dolphinscheduler-1.3.6-bin]# chmod ugo+x conf/env/dolphinscheduler_env.sh 
[root@node01 apache-dolphinscheduler-1.3.6-bin]# pwd
/opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin

5.2 数据库配置

[root@node01 apache-dolphinscheduler-1.3.6-bin]# vi conf/datasource.properties 
#
# mysql
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://192.168.1.161:3306/dolphinscheduler1.3.6?useUnicode=true&characterEncoding=UTF-8
spring.datasource.username=root
spring.datasource.password=123456

5.3 添加数据库驱动

[root@node01 lib]# pwd
/opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin/lib
[root@node01 lib]# ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar
[root@node01 lib]#
[root@node01 apache-dolphinscheduler-1.3.6-bin]# pwd
/opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin
[root@node01 apache-dolphinscheduler-1.3.6-bin]# ll
总用量 156
drwxr-xr-x 2 root root    79 7月  31 18:58 bin
drwxr-xr-x 5 root root  4096 7月  31 19:07 conf
drwxr-xr-x 3 root root    30 7月  31 18:58 ext
-rwxr-xr-x 1 root root  4663 4月  25 11:13 install.sh
drwxr-xr-x 2 root root 12288 7月  31 19:09 lib
-rw-r--r-- 1 root root 38823 4月  25 11:13 LICENSE
drwxr-xr-x 3 root root 12288 7月  31 18:58 licenses
-rw-r--r-- 1 root root 70929 4月  25 11:13 NOTICE
drwxr-xr-x 2 root root   218 7月  31 18:58 script
drwxr-xr-x 4 root root   125 7月  31 18:58 sql
drwxr-xr-x 8 root root   113 7月  31 18:58 ui
[root@node01 apache-dolphinscheduler-1.3.6-bin]# sh ./script/create-dolphinscheduler.sh 
y_id` varchar(200);     END IF; END;  
19:18:08.674 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: CALL uc_dolphin_T_qrtz_fired_triggers_R_entry_id 
19:18:08.714 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: DROP PROCEDURE uc_dolphin_T_qrtz_fired_triggers_R_entry_id 
19:18:08.717 [main] INFO org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - sqlSQLFilePath/opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin/sql/upgrade/1.3.6_schema/mysql/dolphinscheduler_dml.sql
19:18:08.726 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - upgrade DolphinScheduler finished
19:18:08.727 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - create DolphinScheduler success
[root@node01 apache-dolphinscheduler-1.3.6-bin]#

5.4 配置大数据的环境变量

[root@node01 apache-dolphinscheduler-1.3.6-bin]# vi conf/env/dolphinscheduler_env.sh 

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#


export HADOOP_HOME=/usr/hdp/current/hadoop-client
export HADOOP_CONF_DIR=etc/hadoop/conf
#export SPARK_HOME1=/opt/soft/spark1
export SPARK_HOME2=/opt/module/spark-2.4.3-bin-hadoop2.7/bin
export PYTHON_HOME=/usr/bin/python
export JAVA_HOME=/opt/module/jdk1.8.0_231
export HIVE_HOME=/usr/hdp/current/hive-client
export PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH

5.5 修改配置集群的参数

[root@node01 apache-dolphinscheduler-1.3.6-bin]# vi  conf/config/install_config.conf 



# NOTICE :  If the following config has special characters in the variable `.*[]^${}\+?|()@#&`, Please escape, for example, `[` escape to `\[`
# postgresql or mysql
dbtype="mysql"

# db config
# db address and port
dbhost="192.168.1.162:3306"

# db username
username="root"

# database name
dbname="dolphinscheduler1.3.6"

# db passwprd
# NOTICE: if there are special characters, please use the \ to escape, for example, `[` escape to `\[`
password="123456"

# zk cluster
zkQuorum="192.168.1.161:2181,192.168.1.162:2181,192.168.1.163:2181"

# Note: the target installation path for dolphinscheduler, please not config as the same as the current path (pwd)
installPath="/ds1.3.6"

# deployment user
# Note: the deployment user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled, the root directory needs to be created by itself
deployUser="root"

# 邮件协议
mailProtocol="SMTP"

# alert config

# mail server port
# note: Different protocols and encryption methods correspond to different ports, when SSL/TLS is enabled, make sure the port is correct.
mailServerPort="25"

# sender
mailSender="460030103@qq.com"

# user
mailUser="460030103@qq.com"

# sender password
# note: The mail.passwd is email service authorization code, not the email login password.
mailPassword="xxxxxx"

# TLS mail protocol support
starttlsEnable="true"

# SSL mail protocol support
# only one of TLS and SSL can be in the true state.
sslEnable="false"

#note: sslTrust is the same as mailServerHost
sslTrust="smtp.qq.com"


# resource storage type: HDFS, S3, NONE
resourceStorageType="HDFS"

# if resourceStorageType is HDFS，defaultFS write namenode address，HA you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，s3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://node01.com:8020"

# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"

# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty
yarnHaIps="192.168.xx.xx,192.168.xx.xx"

# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname
singleYarnIp="yarnIp1"

# resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"

# who have permissions to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"

# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username
keytabUserName="hdfs-mycluster@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"


# api server port
apiServerPort="12345"


# install hosts
# Note: install the scheduled hostname list. If it is pseudo-distributed, just write a pseudo-distributed hostname
ips="node01.com,node02.com,node03.com"

# ssh port, default 22
# Note: if ssh port is not default, modify here
sshPort="22"

# run master machine
# Note: list of hosts hostname for deploying master
masters="node01.com,node02.com"

# run worker machine
# note: need to write the worker group name of each worker, the default value is "default"
workers="node02.com,node03.com"

# run alert machine
# note: list of machine hostnames for deploying alert server
alertServer="node02.com"

# run api machine
# note: list of machine hostnames for deploying api server
apiServers="node03.com"
-- INSERT --

5.6 修改安装文件：

[root@node01 apache-dolphinscheduler-1.3.6-bin]# vi install.sh 

#!/bin/sh
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

workDir=`dirname $0`
workDir=`cd ${workDir};pwd`

source ${workDir}/conf/config/install_config.conf

# 1.replace file
echo "1.replace file"

txt=""
if [[ "$OSTYPE" == "darwin"* ]]; then
    # Mac OSX
    txt="''"
fi

datasourceDriverClassname="com.mysql.jdbc.Driver"
if [ $dbtype == "postgresql" ];then
  datasourceDriverClassname="org.postgresql.Driver"
fi
sed -i ${txt} "s#spring.datasource.driver-class-name.*#spring.datasource.driver-class-name=${datasourceDriverClassname}#g" conf/datasource.properties
sed -i ${txt} "s#spring.datasource.url.*#spring.datasource.url=jdbc:${dbtype}://${dbhost}/${dbname}?characterEncoding=UTF-8\&allowMultiQueries=true#g" conf/datasource.properties
sed -i ${txt} "s#spring.datasource.username.*#spring.datasource.username=${username}#g" conf/datasource.properties
sed -i ${txt} "s#spring.datasource.password.*#spring.datasource.password=${password}#g" conf/datasource.properties

sed -i ${txt} "s#fs.defaultFS.*#fs.defaultFS=${defaultFS}#g" conf/common.properties
sed -i ${txt} "s#fs.s3a.endpoint.*#fs.s3a.endpoint=${s3Endpoint}#g" conf/common.properties
sed -i ${txt} "s#fs.s3a.access.key.*#fs.s3a.access.key=${s3AccessKey}#g" conf/common.properties
sed -i ${txt} "s#fs.s3a.secret.key.*#fs.s3a.secret.key=${s3SecretKey}#g" conf/common.properties
sed -i ${txt} "s#yarn.resourcemanager.ha.rm.ids.*#yarn.resourcemanager.ha.rm.ids=${yarnHaIps}#g" conf/common.properties
sed -i ${txt} "s#yarn.application.status.address.*#yarn.application.status.address=http://${singleYarnIp}:8088/ws/v1/cluster/apps/%s#g" conf/common.properties
sed -i ${txt} "s#hdfs.root.user.*#hdfs.root.user=${hdfsRootUser}#g" conf/common.properties
sed -i ${txt} "s#resource.upload.path.*#resource.upload.path=${resourceUploadPath}#g" conf/common.properties
sed -i ${txt} "s#resource.storage.type.*#resource.storage.type=${resourceStorageType}#g" conf/common.properties
sed -i ${txt} "s#hadoop.security.authentication.startup.state.*#hadoop.security.authentication.startup.state=${kerberosStartUp}#g" conf/common.properties
sed -i ${txt} "s#java.security.krb5.conf.path.*#java.security.krb5.conf.path=${krb5ConfPath}#g" conf/common.properties
sed -i ${txt} "s#login.user.keytab.username.*#login.user.keytab.username=${keytabUserName}#g" conf/common.properties
sed -i ${txt} "s#login.user.keytab.path.*#login.user.keytab.path=${keytabPath}#g" conf/common.properties
sed -i ${txt} "s#zookeeper.quorum.*#zookeeper.quorum=${zkQuorum}#g" conf/zookeeper.properties
sed -i ${txt} "s#server.port.*#server.port=${apiServerPort}#g" conf/application-api.properties
sed -i ${txt} "s#mail.server.host.*#mail.server.host=${mailServerHost}#g" conf/alert.properties
sed -i ${txt} "s#mail.server.port.*#mail.server.port=${mailServerPort}#g" conf/alert.properties
sed -i ${txt} "s#mail.sender.*#mail.sender=${mailSender}#g" conf/alert.properties
sed -i ${txt} "s#mail.user.*#mail.user=${mailUser}#g" conf/alert.properties
sed -i ${txt} "s#mail.passwd.*#mail.passwd=${mailPassword}#g" conf/alert.properties
sed -i ${txt} "s#mail.smtp.starttls.enable.*#mail.smtp.starttls.enable=${starttlsEnable}#g" conf/alert.properties
sed -i ${txt} "s#mail.smtp.ssl.trust.*#mail.smtp.ssl.trust=${sslTrust}#g" conf/alert.properties
sed -i ${txt} "s#mail.smtp.ssl.enable.*#mail.smtp.ssl.enable=${sslEnable}#g" conf/alert.properties

# 2.create directory
echo "2.create directory"

if [ ! -d $installPath ];then
  sudo mkdir -p $installPath
  sudo chown -R $deployUser:$deployUser $installPath
fi

hostsArr=(${ips//,/ })
for host in ${hostsArr[@]}
do
# create if programPath does not exist
if ! ssh $host test -e $programPath; then
ssh $host "sudo mkdir -p $programPath;sudo chown -R $deployUser:$deployUser
$programPath"
fi
# create if downloadPath does not exist
if ! ssh $host test -e $downloadPath; then
ssh $host "sudo mkdir -p $downloadPath;sudo chown -R $deployUser:$deployUser
$downloadPath"
fi
# create if execPath does not exist
if ! ssh $host test -e $execPath; then
ssh $host "sudo mkdir -p $execPath; sudo chown -R $deployUser:$deployUser
$execPath"
fi
# create if xlsFilePath does not exist
if ! ssh $host test -e $xlsFilePath; then
ssh $host "sudo mkdir -p $xlsFilePath; sudo chown -R $deployUser:$deployUser
$xlsFilePath"
fi
done



# 3.scp resources
echo "3.scp resources"
sh ${workDir}/script/scp-hosts.sh
if [ $? -eq 0 ]
then
        echo 'scp copy completed'
else
        echo 'scp copy failed to exit'
        exit 1
fi


# 4.stop server
echo "4.stop server"
sh ${workDir}/script/stop-all.sh


# 5.delete zk node
echo "5.delete zk node"

sh ${workDir}/script/remove-zk-node.sh $zkRoot


# 6.startup
echo "6.startup"
sh ${workDir}/script/start-all.sh
-- INSERT --

刷新
source /opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin/conf/env/dolphinscheduler_env.sh

5.7 加hdfs 配置文件

[root@node01 apache-dolphinscheduler-1.3.6-bin]# scp /etc/hadoop/conf/core-site.xml /opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin/conf/
[root@node01 apache-dolphinscheduler-1.3.6-bin]# scp /etc/hadoop/conf/hdfs-site.xml /opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin/conf/
[root@node01 apache-dolphinscheduler-1.3.6-bin]#

5.8 修改 java 的环境变量

两个文件最前面添加   export JAVA_HOME=/opt/module/jdk1.8.0_231

[root@node01 apache-dolphinscheduler-1.3.6-bin]# vi bin/dolphinscheduler-daemon.sh 
[root@node01 apache-dolphinscheduler-1.3.6-bin]# vi script/dolphinscheduler-daemon.sh 
[root@node01 apache-dolphinscheduler-1.3.6-bin]# pwd
/opt/module/dolphinscheduler/apache-dolphinscheduler-1.3.6-bin
[root@node01 apache-dolphinscheduler-1.3.6-bin]#

5.9 一键部署 ./install.sh

命令： ./install.sh

nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false -Xms2g -Xmx2g -Xmn1g -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.worker.WorkerServer > /ds1.3.6/logs/dolphinscheduler-worker-server-node03.com.out 2>&1 &
End start worker-server.
Begin start logger-server......
Info: Skywalking not enabled.
starting logger-server, logging to /ds1.3.6/logs/dolphinscheduler-logger-server-node03.com.out
nohup /opt/module/jdk1.8.0_231/bin/java  -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.log.LoggerServer > /ds1.3.6/logs/dolphinscheduler-logger-server-node03.com.out 2>&1 &
End start logger-server.
node02.com worker server is starting
Begin start worker-server......
Info: Skywalking not enabled.
starting worker-server, logging to /ds1.3.6/logs/dolphinscheduler-worker-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false -Xms2g -Xmx2g -Xmn1g -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.worker.WorkerServer > /ds1.3.6/logs/dolphinscheduler-worker-server-node02.com.out 2>&1 &
End start worker-server.
Begin start logger-server......
Info: Skywalking not enabled.
starting logger-server, logging to /ds1.3.6/logs/dolphinscheduler-logger-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java  -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.log.LoggerServer > /ds1.3.6/logs/dolphinscheduler-logger-server-node02.com.out 2>&1 &
End start logger-server.
Begin start alert-server......
Info: Skywalking not enabled.
starting alert-server, logging to /ds1.3.6/logs/dolphinscheduler-alert-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogback.configurationFile=conf/logback-alert.xml -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.alert.AlertServer > /ds1.3.6/logs/dolphinscheduler-alert-server-node02.com.out 2>&1 &
End start alert-server.
node03.com worker server is starting
Begin start api-server......
Info: Skywalking not enabled.
starting api-server, logging to /ds1.3.6/logs/dolphinscheduler-api-server-node03.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-api.xml -Dspring.profiles.active=api -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.api.ApiApplicationServer > /ds1.3.6/logs/dolphinscheduler-api-server-node03.com.out 2>&1 &
End start api-server.
[root@node01 apache-dolphinscheduler-1.3.6-bin]# rumRemoteCmd "jps" all

6.0 启动服务命令

启动命令： bin/start-all.sh

[root@node01 bin]# ./start-all.sh 
node02.com
node03.com
node01.com master server is starting
Begin start master-server......
Info: Skywalking not enabled.
starting master-server, logging to /ds1.3.6/logs/dolphinscheduler-master-server-node01.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false -Xms4g -Xmx4g -Xmn2g -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.master.MasterServer > /ds1.3.6/logs/dolphinscheduler-master-server-node01.com.out 2>&1 &
End start master-server.
node02.com master server is starting
Begin start master-server......
Info: Skywalking not enabled.
starting master-server, logging to /ds1.3.6/logs/dolphinscheduler-master-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false -Xms4g -Xmx4g -Xmn2g -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.master.MasterServer > /ds1.3.6/logs/dolphinscheduler-master-server-node02.com.out 2>&1 &
End start master-server.
node03.com worker server is starting
Begin start worker-server......
Info: Skywalking not enabled.
starting worker-server, logging to /ds1.3.6/logs/dolphinscheduler-worker-server-node03.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false -Xms2g -Xmx2g -Xmn1g -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.worker.WorkerServer > /ds1.3.6/logs/dolphinscheduler-worker-server-node03.com.out 2>&1 &
End start worker-server.
Begin start logger-server......
Info: Skywalking not enabled.
starting logger-server, logging to /ds1.3.6/logs/dolphinscheduler-logger-server-node03.com.out
nohup /opt/module/jdk1.8.0_231/bin/java  -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.log.LoggerServer > /ds1.3.6/logs/dolphinscheduler-logger-server-node03.com.out 2>&1 &
End start logger-server.
node02.com worker server is starting
Begin start worker-server......
Info: Skywalking not enabled.
starting worker-server, logging to /ds1.3.6/logs/dolphinscheduler-worker-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false -Xms2g -Xmx2g -Xmn1g -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.worker.WorkerServer > /ds1.3.6/logs/dolphinscheduler-worker-server-node02.com.out 2>&1 &
End start worker-server.
Begin start logger-server......
Info: Skywalking not enabled.
starting logger-server, logging to /ds1.3.6/logs/dolphinscheduler-logger-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java  -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.server.log.LoggerServer > /ds1.3.6/logs/dolphinscheduler-logger-server-node02.com.out 2>&1 &
End start logger-server.
Begin start alert-server......
Info: Skywalking not enabled.
starting alert-server, logging to /ds1.3.6/logs/dolphinscheduler-alert-server-node02.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogback.configurationFile=conf/logback-alert.xml -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.alert.AlertServer > /ds1.3.6/logs/dolphinscheduler-alert-server-node02.com.out 2>&1 &
End start alert-server.
node03.com worker server is starting
Begin start api-server......
Info: Skywalking not enabled.
starting api-server, logging to /ds1.3.6/logs/dolphinscheduler-api-server-node03.com.out
nohup /opt/module/jdk1.8.0_231/bin/java -Dlogging.config=classpath:logback-api.xml -Dspring.profiles.active=api -Xms1g -Xmx1g -Xmn512m -server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:/ds1.3.6/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm   -classpath /ds1.3.6/conf:/ds1.3.6/lib/* org.apache.dolphinscheduler.api.ApiApplicationServer > /ds1.3.6/logs/dolphinscheduler-api-server-node03.com.out 2>&1 &
End start api-server.
[root@node01 bin]#

7.访问地址：

http://192.168.1.163:12345/dolphinscheduler/ui/#/home
admin/dolphinscheduler123

ds 界面.png

ds是界面.png

8. 对应api 文档地址

http://192.168.1.163:12345/dolphinscheduler/doc.html

ds-doc.png

dolphinscheduler-集群安装

1. 整体的架构

1.1 官网

1.2 架构图

1.3 源码下载：本文下载的1.3.6 版本

1.4 下载好解压：

2.修改源码和大数据的版本pom 文件

2.1 进行编译：

正常编译完后，会在 dolphinscheduler-dist/target ⽬目录下⽣生成如下安装包：

4. 部署规划

5. 安装

5.1. 赋值权限

5.2 数据库配置

5.3 添加数据库驱动

5.4 配置大数据的环境变量

5.5 修改配置集群的参数

5.6 修改安装文件：

5.7 加hdfs 配置文件

5.8 修改 java 的环境变量

5.9 一键部署 ./install.sh

6.0 启动服务命令

7.访问地址：

8. 对应api 文档地址

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

dolphinscheduler-集群安装

1. 整体的架构

1.1 官网

1.2 架构图

1.3 源码下载： 本文下载的1.3.6 版本

1.4 下载好解压：

2.修改源码和大数据的版本pom 文件

2.1 进行编译：

正常编译完后，会在 dolphinscheduler-dist/target ⽬目录下⽣生成如下安装包：

4. 部署规划

5. 安装

5.1. 赋值权限

5.2 数据库配置

5.3 添加数据库驱动

5.4 配置大数据的环境变量

5.5 修改配置集群的参数

5.6 修改安装文件：

5.7 加hdfs 配置文件

5.8 修改 java 的环境变量

5.9 一键部署 ./install.sh

6.0 启动服务命令

7.访问地址：

8. 对应api 文档地址

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

1.3 源码下载：本文下载的1.3.6 版本