Doris 源码分析 (一) 开发环境准备

作者: 走在成长的道路上 | 来源:发表于2021-09-29 12:01 被阅读0次

百度开源 doris 数据库借鉴 impala 经验而形成的，可以看到源码结构上与 impala 相似，废话少说，进源码。

注 环境使用 ubuntu 桌面环境

同步代码

源码编译

# 基础依赖环境安装, 在 `docs/zh-CN/developer-guide/be-vscode-dev.md` 文件中说明
sudo apt install build-essential openjdk-8-jdk maven cmake byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python brotli
sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa
sudo apt update
sudo apt install gcc-10 g++-10 
sudo apt-get install autoconf automake libtool autopoint

# 安装 openssl-dev 环境
sudo apt install -y openssl-devel

# 同步代码
git clone https://github.com/apache/incubator-doris

# 进入三方软件目录，下载依赖库(好处：跨平台编译)
cd incubator-doris/thirdparty
 ./build-thirdparty.sh

# 初始化编译
./build.sh

# 注意在 fe/conf/fe.conf 文件中配置 http，及 QeServer 端口号信息，方便下面使用，默认密码为空白
# 参见 https://doris.apache.org/master/zh-CN/installing/install-deploy.html#%E9%9B%86%E7%BE%A4%E9%83%A8%E7%BD%B2
# 进入 output 目录执行启动 FE 服务, http://localhost:8030 进入前端页面(默认用户: root，密码为空)
# 使用 mysql 客户端访问: mysql -h 127.0.0.1 -P 9030 -uroot
./fe/bin/start_fe.sh --daemon

# 添加 FE Follower/Observer 节点
# 首先，添加 FE 从节点
ALTER SYSTEM ADD FOLLOWER或者OBSERVER "fe ip:edit_log_port";
# 其次，启动 FE 从节点
./bin/start_fe.sh --helper master fe ip:edit_log_port --daemon

# 进入 output 目录执行启动 BE 服务, 注 1 个 FE 至少托 3 个 BE 端
# 使用 http://be_host:be_http_port/api/health 方式即可查看 BE 端状态
# 比如：http://localhost:9060/api/health 
# 注： ulimit -n 60000 
./be/bin/start_be.sh --daemon

手动添加后端

$ mysql -h 127.0.0.1 -P 9030 -uroot
# 查看当前 BE端 列表
MySQL [(none)]> SHOW PROC '/backends';

# 添加BE端，这里的端口为 be/conf/be.conf 中的 heartbeat_service_port
# 注意这里必须是主机 ip , 而非 127.0.0.1 或类似地址
MySQL [(none)]>  ALTER SYSTEM ADD BACKEND "172.29.9.19:9050" PROPERTIES ("tag.location" = "aa");
MySQL [(none)]> ALTER SYSTEM ADD BACKEND "172.29.9.19:9051";
MySQL [(none)]> ALTER SYSTEM ADD BACKEND "172.29.9.19:9052";

MySQL [(none)]> SHOW PROC '/backends';
+-----------+-----------------+-------------+-------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+---------+-----------------+
| BackendId | Cluster         | IP          | HostName    | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg                   | Version | Status          |
+-----------+-----------------+-------------+-------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+---------+-----------------+
| 12004     | default_cluster | 172.29.9.19 | 172.29.9.19 | 9052          | 9062   | 8042     | 8062     | 2021-09-30 23:44:32 | 2021-09-30 23:54:40 | true  | false                | false                 | 0         | 0.000            | 277.172 GB    | 916.770 GB    | 69.77 % | 69.77 %        | {"location" : "default"} |         | trunk-840a7ef3a |
| 12002     | default_cluster | 172.29.9.19 | 172.29.9.19 | 9050          | 9060   | 8040     | 8060     | 2021-09-30 23:43:54 | 2021-09-30 23:55:10 | true  | false                | false                 | 0         | 0.000            | 277.171 GB    | 916.770 GB    | 69.77 % | 69.77 %        | {"location" : "default"} |         | trunk-840a7ef3a |
| 12003     | default_cluster | 172.29.9.19 | 172.29.9.19 | 9051          | 9061   | 8041     | 8061     | 2021-09-30 23:44:27 | 2021-09-30 23:55:36 | true  | false                | false                 | 0         | 0.000            | 277.171 GB    | 916.770 GB    | 69.77 % | 69.77 %        | {"location" : "default"} |         | trunk-840a7ef3a |
+-----------+-----------------+-------------+-------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+---------+-----------------+
3 rows in set (2 min 43.731 sec)

# 删除BE端
MySQL [(none)]> ALTER SYSTEM DECOMMISSION BACKEND "172.29.9.19:9050";
MySQL [(none)]> ALTER SYSTEM DECOMMISSION BACKEND "172.29.9.19:9051";
MySQL [(none)]> ALTER SYSTEM DECOMMISSION BACKEND "172.29.9.19:9052";

# 修改 root 密码
MySQL [(none)]> SET PASSWORD FOR 'root' = PASSWORD('root');
# 创建一个普通用户
MySQL [(none)]> CREATE USER 'dev' IDENTIFIED BY 'dev123';

默认配置

FE 端配置

LOG_DIR = ${DORIS_HOME}/log

DATE = `date +%Y%m%d-%H%M%S`
JAVA_OPTS="-Xmx4096m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Xmx4096m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"

##
## the lowercase properties are read by main program.
##

# INFO, WARN, ERROR, FATAL
sys_log_level = INFO

# store metadata, must be created before start FE.
# Default value is ${DORIS_HOME}/doris-meta
# meta_dir = ${DORIS_HOME}/doris-meta

http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
mysql_service_nio_enabled = true
enable_http_server_v2 = false

BE 端配置

PPROF_TMPDIR="$DORIS_HOME/log/"

# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO

# ports for admin, web, heartbeat service 
be_port = 9060
be_rpc_port = 9070
webserver_port = 8040
heartbeat_service_port = 9050
brpc_port = 8060

# Choose one if there are more than one ip except loopback address. 
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
# priority_networks = 10.10.10.0/24;192.168.0.0/16

# data root path, separate by ';'
# you can specify the storage medium of each root path, HDD or SSD
# you can add capacity limit at the end of each root path, seperate by ','
# eg:
# storage_root_path = /home/disk1/doris.HDD,50;/home/disk2/doris.SSD,1;/home/disk2/doris
# /home/disk1/doris.HDD, capacity limit is 50GB, HDD;
# /home/disk2/doris.SSD, capacity limit is 1GB, SSD;
# /home/disk2/doris, capacity limit is disk capacity, HDD(default)
# 
# you also can specify the properties by setting '<property>:<value>', seperate by ','
# property 'medium' has a higher priority than the extension of path
#
# Default value is ${DORIS_HOME}/storage, you should create it by hand.
storage_root_path = ${DORIS_HOME}/storage

# Advanced configurations
sys_log_dir = ${DORIS_HOME}/log
sys_log_roll_mode = SIZE-MB-1024
# sys_log_roll_num = 10
# sys_log_verbose_modules = *
# log_buffer_level = -1
# palo_cgroups

编译问题

编译 re 时，出现 Cmake error :generator: Ninja“ 问题

注 删除 CMakeCache.txt 文件即可，也将整个 CMakeFiles 目录干掉

CMake 版本低于 3.19.3, 升级即可

# 查看当前版本
cmake --version
# 卸载 cmake 工具
sudo apt remove cmake

# 下载新版本
wget https://github.com/Kitware/CMake/releases/download/v3.20.6/cmake-3.20.6-linux-x86_64.sh

# 安装
chmod +x cmake-3.20.6-linux-x86_64.sh 
./cmake-3.20.6-linux-x86_64.sh 
mv cmake-3.20.6-linux-x86_64 /usr/local/bin/

# 创建软连接
sudo mv cmake-3.20.6-linux-x86_64 /usr/local/bin/

# 测试
cmake --version

brotli_ep-prefix 编译出现问题，提示 cmake --build . 运行即可

注 进入到 arrow 项目下，执行 cmake --build . 编译即可

BE 端运行时错误： Internal error: file descriptors limit is too small

`VSCode` 启动

code .

注 BE 端配置参考 docs/zh-CN/developer-guide/be-vscode-dev.md 配置即可

Fe 端运行调试配置

首先，配置环境变量，具体如下:

JAVA_OPTS=-Xmx1024m
DORIS_HOME=/srv/workspace/incubator-doris/fe
PID_DIR=/srv/workspace/incubator-doris/fe
LOG_DIR=/srv/workspace/incubator-doris/fe/log

其次，添加所需的目录及文件

# 当前目录为 doris 代码根目录
# 在 fe 目录创建 log, doris-meta 目录，方便存放日志及 bdb 等文件
mkdir fe/{log,doris-meta} -p

# 添加 conf, webroot 等文件
cp conf fe/conf -rf
cp webroot fe/webroot -rf

注 其 conf/fe.conf 具体内容如下：

# the output dir of stderr and stdout 
LOG_DIR = ${DORIS_HOME}/log

DATE = `date +%Y%m%d-%H%M%S`
JAVA_OPTS="-Xmx2048m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Xmx4096m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"

##
## the lowercase properties are read by main program.
##

# INFO, WARN, ERROR, FATAL
sys_log_level = INFO

# store metadata, create it if it is not exist.
# Default value is ${DORIS_HOME}/doris-meta
# meta_dir = ${DORIS_HOME}/doris-meta

http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
mysql_service_nio_enabled = true
enable_http_server_v2=false

ubuntu 下使用 idea 进行调试时，点击 debug 按钮后 debug 按钮灰掉且无效

首先，在 fe-core/pom.xml 中添加 maven-jar-plugin 插件进行 jar 包中添加 mainClass 及构建全包，如下所示：

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <configuration>
        <archive>
            <manifest>
                <mainClass>org.apache.doris.PaloFe</mainClass>
                <!-- 是否指定项目classpath下的依赖 -->
                <addClasspath>true</addClasspath>
                <!-- 指定依赖的时候声明前缀 -->
                <classpathPrefix>./lib/</classpathPrefix>
                <!--依赖是否使用带有时间戳的唯一版本号,如:xxx-1.3.0-20211225.012733.jar-->
                <useUniqueVersions>false</useUniqueVersions>
            </manifest>
        </archive>
    </configuration>
</plugin>

接着，使用 Jar-Application 配置调试选项，如下图所示:

Jar-Application

如上所示，从而可以在打包 mvn package -DskipTests 之后进行调试咯，具体为啥直接不能调试，暂时不知道，如果有知道的小伙伴，可以告知我下哦，先谢谢咯

Doris 源码分析 (一) 开发环境准备

同步代码

编译问题

`VSCode` 启动

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

Doris 源码分析 (一) 开发环境准备

同步代码

编译问题

VSCode 启动

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

`VSCode` 启动