记录CDH6.3.2部署流程

作者: 惜鸟 | 来源:发表于2021-04-18 18:01 被阅读0次

一、概述

最近安装了cdh6.3，网上也有各种安装教程，写的很详细，我也参考了下面的文档进行安装部署，主要的安装流程是参考官网进行操作的，如果英文较好的话可以参考官方文档。在这里将我自己的部署流程记录下来，并且我这里列出了安装CDH集群需要的安装包的下载路径。

cdh官网安装地址：https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/installation.html
cm下载地址：https://archive.cloudera.com/cm6/
cdh下载地址：https://archive.cloudera.com/cdh6/
mysql下载地址：https://downloads.mysql.com/archives/community/
jdk下载地址：https://blog.csdn.net/LinBilin_/article/details/50217541
flink parcle下载地址：http://archive.cloudera.com/csa/1.0.0.0/parcels/
anaconda parcel下载地址：https://repo.anaconda.com/pkgs/misc/parcels/archive/

根据我的安装流程，安装CDH集群主要分为五步：

所有节点做一些准备工作
所有节点安装jdk
选择一台机器安装mysql
安装Cloudera Manager 和 CDH
测试各组件

二、CDH集群安装步骤

1、基础配置，所有机器都需要执行

# 设置主机名
hostnamectl set‐hostname dev-cdh1
hostnamectl set-hostname dev-cdh2
hostnamectl set-hostname dev-cdh3
# 修改hosts，将主机名与ip映射，以便通过主机名访问
vi /etc/hosts
172.18.30.176    dev-cdh1
172.18.30.177    dev-cdh2
172.18.30.178    dev-cdh3

# 设置三台机器的root用户密码：Y2V@q5znyp4S8SJV
passwd

# 配置免密登陆，便于节点之间传输文件
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 dev-cdh1
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 dev-cdh2
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 dev-cdh3


# 安装一些依赖
yum -y install gcc python-devel cyrus-sasl* chrony iotop yum-utils yum-plugin-downloadonly psmisc ntp ntpdate httpd createrepo wget git rsync mod_ssl cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi libxslt krb5-devel cyrus-sasl-gssapi cyrus-sasl-devel libxml2-devel libxslt-devel openldap-devel python-devel python-simplejson sqlite-devel
# 删除centos 自带的mariadb
yum ‐y remove mariadb‐libs
# 禁用防火墙
systemctl disable firewalld
systemctl stop firewalld
# 关闭selinux
vim /etc/selinux/config
# 修改config文件中的SELINUX为disable
SELINUX=disable
# 启动时间同步服务
systemctl start ntpd
# 设置时间同步服务开机启动
systemctl enable ntpd
# 改一些系统参数
sysctl vm.swappiness=1
echo 'vm.swappiness=1'>> /etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

2、所有节点安装JDK


# 安装jdk，注意：jdk只能安装在/usr/java目录下面，否则启动cm-server时会报错，找不到jdk
# 下载好jdk，上传到dev-cdh1，并且分发到其他节点
scp -P 40022 jdk-8u241-linux-x64.tar.gz dev-cdh2:/nfplus/cdh
scp -P 40022 jdk-8u241-linux-x64.tar.gz dev-cdh3:/nfplus/cdh

# 所有节点都需要安装jdk到指定目录
mkdir /usr/java
tar -zxvf jdk-8u241-linux-x64.tar.gz -C /usr/java/
# jdk解压后需要修改其所属的用户和用户组
chown -R root:root /usr/java/jdk1.8.0_241/
cat << EOF >> /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_241
export PATH=\$PATH:\$JAVA_HOME/bin
EOF
# 使配置生效
source /etc/profile
# 检查java是否安装成功
 jps

注意：
jdk安装在/usr/java
mysql驱动安装在/usr/share/java/mysql-connector-java.jar

3、在dev-cdh1上面安装mysql

# 下载mysql
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-community-common-5.7.29-1.el7.x86_64.rpm
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-community-libs-5.7.29-1.el7.x86_64.rpm
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-community-devel-5.7.29-1.el7.x86_64.rpm
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-community-client-5.7.29-1.el7.x86_64.rpm
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-community-server-5.7.29-1.el7.x86_64.rpm

# 本地安装mysql, 按照如下顺序安装
yum -y localinstall mysql-community-common-5.7.29-1.el7.x86_64.rpm
yum -y localinstall mysql-community-libs-5.7.29-1.el7.x86_64.rpm
yum -y localinstall mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm
yum -y localinstall mysql-community-devel-5.7.29-1.el7.x86_64.rpm
yum -y localinstall mysql-community-client-5.7.29-1.el7.x86_64.rpm
yum -y localinstall mysql-community-server-5.7.29-1.el7.x86_64.rpm

mysql配置，根据cloudera官网进行如下配置

vim /etc/my.cnf

将my.cnf的内容替换为如下内容

[mysqld]
# 根据需求修改端口号：3506
port=3506
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

启动mysql

# 启动
 systemctl start mysqld
 # 开机自启动
 systemctl enable mysqld

 #查看临时密码
 cat /var/log/mysqld.log | grep password

 mysql ‐uroot ‐p'临时密码'
 ## 首次登录设置密码  xfik@cKRLm8q3eMu
 set password=password('xfik@cKRLm8q3eMu');
 quit;
 # 再次登陆验证密码是否生效
 mysql -uroot -p'xfik@cKRLm8q3eMu'

创建ClouderaManager需要用到的几个数据库

create database scm default character set utf8 default collate utf8_general_ci;
 grant all on scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
 
create database amon default character set utf8 default collate utf8_general_ci;
 grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
 
create database hive default character set utf8 default collate utf8_general_ci;
 grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
 
create database hue default character set utf8 default collate utf8_general_ci;
 grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
 
create database oozie default character set utf8 default collate utf8_general_ci;
 grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
 
flush privileges;

quit;

整理mysql的驱动包，所有节点安装

 #整理mysql的驱动包, mysql驱动包必须放在/usr/share/java目录，并且需要重命名mysql‐connector‐java.jar， 所有节点都需要安装
 # 下载mysql驱动包
 wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.30/mysql-connector-java-5.1.30.jar
 mv mysql‐connector‐java‐5.1.30.jar mysql‐connector‐java.jar
 mkdir /usr/share/java
 cp mysql‐connector‐java.jar /usr/share/java/mysql‐connector‐java.jar

4、安装Cloudera Manager

wget https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

wget https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm

wget https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

# 全部节点都安装daemons和agent
yum -y localinstall cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
yum -y localinstall cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

# dev-cdh1上安装server
yum -y localinstall cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

5、在dev-cdh1下载CDH的parcel包

wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json

wget  https://archive.cloudera.com/phoenix/6.2.0/parcels/PHOENIX‐5.0.0‐cdh6.2.0.p0.1308267‐el7.parcel.sha
wget  https://archive.cloudera.com/phoenix/6.2.0/parcels/PHOENIX‐5.0.0‐cdh6.2.0.p0.1308267‐el7.parcel
wget  https://archive.cloudera.com/phoenix/6.2.0/csd/PHOENIX‐1.0.jar

# 将parcel.sha1修改为parcel.sha
mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
# 然后把PHOENIX‐1.0.jar放到/opt/cloudera/csd目录
[root@dev-cdh1 csd]# pwd
 /opt/cloudera/csd
[root@dev-cdh1 csd]# ll
total 8
-rw-r--r-- 1 root root 5306 Jun 19 18:11 PHOENIX-1.0.jar  
 #  把凡是带sha1的都改成sha
 #  把上面下载的包都放在/opt/cloudera/parcel‐repo下
  
[root@dev-cdh1 parcel‐repo]# pwd
 /opt/cloudera/parcel‐repo
[root@dev-cdh1 parcel‐repo]# ll
-rw-r--r-- 1 root root 2082186246 Jun 19 18:00 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r-- 1 root root         40 Jun 19 17:50 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
-rw-r--r-- 1 root root      33887 Jun 19 17:51 manifest.json
-rw-r--r-- 1 root root  402216960 Jun 19 18:14 PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel
-rw-r--r-- 1 root root         41 Jun 19 18:05 PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha

修改cm的server与agent配置


# dev-cdh1上修改server
vi /etc/cloudera‐scm‐server/db.properties
# 12行去掉注释把localhost改成dev-cdh1:3506
# 15行去掉注释
# 18行去掉注释，把cmf改成scm
# 21行去掉注释，密码那里填写TlWNavXjM@3qCJvv
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=dev-cdh1:3506
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=TlWNavXjM@3qCJvv


# 全部节点修改agent
vi /etc/cloudera‐scm‐agent/config.ini
 # 15行把localhost修改为dev-cdh1

6、启动cm的sever和agent，开始安装cdh集群


# 在dev-cdh1上启动server
systemctl start cloudera-scm-server
# 另开一个窗口，查看相关日志。有异常就解决异常
tail ‐200f /var/log/cloudera‐scm‐server/cloudera‐scm‐server.log
# 这个异常可以忽略 ERROR ParcelUpdateService:com.cloudera.parcel.component
ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifes
# 等待日志输出 started jetty server, 则cm-server启动成功

# 在全部节点上启动agent
systemctl start cloudera-scm-agent
 # 当在 cdh001上 netstat ‐antp | grep 7180 有内容时，说明我们可以访问web页面了
# 查看运行状态
systemctl status cloudera-scm-agent
systemclt status cloudera-scm-server

三、测试CM集群安装的组件


##1、Kafka测试
# 创建一个topic
kafka-topics --zookeeper cdh003:2181,cdh004:2181,cdh005:2181 --create --replication-factor 3 --partitions 10 --topic t1

# 开启kafka控制台消费者
kafka-console-consumer --bootstrap-server cdh003:9092,cdh004:9092,cdh005:9092 --topic t1

# 去别的节点开启一个Kafka控制台生产者
kafka-console-producer --broker-list cdh003:9092,cdh004:9092,cdh005:9092 --topic t1
 一边消费数据一边写入数据


## 2、Hive测试
# 通过Hive建立一张表
 create table if not exists t1(id int,name string,score double) row format delimited fields terminated by ',' stored as parquet;

# 写个带有聚合函数的让它跑一下MapReduce
 select count(*) from t1;

# 3、Phoenix测试
 create table if not exists t1(
 id Integer,
 name varchar,
 clazz varchar,
 score double
 constraint pk primary key(id)
 );
 select * from t1;
 upsert into t1 values(1001,'zhangsan','606',87.5);
 select count(*) from t1;
 select * from t1;

# 4、Impala测试
 create table t2
 (
 name string,
 dt string,
 primary key(name,dt)
 )
 partition by hash partitions 16
 stored as kudu tblproperties (
 'kudu.master_addresses' = 'cdh001:7051'
 );

 select * from t2;
 insert into t2 values('zhangsan','2020‐05‐06');
 select count(*) from t2;
 select * from t2;

参考资料：
CM+CDH6.2.0安装全网最全
 cdh6.2离线安装
 CDH集群部署最佳实践
 flink集成到cdh6.3
CDH 6.3.2 安装

记录CDH6.3.2部署流程

一、概述

二、CDH集群安装步骤

1、基础配置，所有机器都需要执行

2、所有节点安装JDK

3、在dev-cdh1上面安装mysql

4、安装Cloudera Manager

5、在dev-cdh1下载CDH的parcel包

6、启动cm的sever和agent，开始安装cdh集群

三、测试CM集群安装的组件

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

记录CDH6.3.2部署流程

一、概述

二、CDH集群安装步骤

1、基础配置，所有机器都需要执行

2、所有节点安装JDK

3、在dev-cdh1上面安装mysql

4、安装Cloudera Manager

5、 在dev-cdh1下载CDH的parcel包

6、启动cm的sever和agent，开始安装cdh集群

三、测试CM集群安装的组件

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

5、在dev-cdh1下载CDH的parcel包