Centos7 搭建 cloudera-manager 集群

0、准备工作

关闭防火墙

# 所有节点
systemctl stop firewalld
systemctl disable firewalld

禁止交换

# 所有节点
sysctl vm.swappiness=0

禁用大页面

# 所有节点
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

1、JDK

卸载现有 Java

rpm -qa | grep java | xargs sudo rpm -e --nodeps

解压到 /opt/module 目录

tar -zxvf jdk-8u144-linux-x64.tar.gz -C /opt/module/

配置环境变量

sudo vim /etc/profile
#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin

刷新配置

source /etc/profile

测试是否安装成功

java -version

2、MySQL

查看是否现有的包

rpm -qa | grep mariadb

如果有，删之

rpm -e --nodeps 包名

安装 MariaDB

yum -y install mariadb mariadb-server

启动 MariaDB

systemctl start mariadb

设置 MariaDB 自启动

systemctl enable mariadb

进行 MariaDB 的相关简单配置

mysql_secure_installation

首先是设置密码，会提示先输入密码
Enter current password for root (enter for none):<–初次运行直接回车

设置密码
Set root password? [Y/n] <– 是否设置root用户密码，输入y并回车或直接回车
New password: <– 设置root用户的密码
Re-enter new password: <– 再输入一次你设置的密码

其他配置
Remove anonymous users? [Y/n] <– 是否删除匿名用户，回车
Disallow root login remotely? [Y/n] <–是否禁止root远程登录,回车
Remove test database and access to it? [Y/n] <– 是否删除test数据库，回车
Reload privilege tables now? [Y/n] <– 是否重新加载权限表，回车

配置远程登录

mysql -uroot -p
MariaDB [(none)]> use mysql;
MariaDB [mysql]> update user set Host = '%' where Host = 'localhost' and User = 'root';
MariaDB [mysql]> flush privileges;

3、关闭 SELINUX

临时关闭

setenforce 0

永久关闭

vim /etc/selinux/config

SELINUX=disabled

4、SSH 免密登录

生成公钥和私钥

[djm@hadoop102 .ssh]$ ssh-keygen -t rsa
[djm@hadoop103 .ssh]$ ssh-keygen -t rsa
[djm@hadoop104 .ssh]$ ssh-keygen -t rsa

将公钥拷贝到要免密登录的目标机器上

[djm@hadoop102 .ssh]$ ssh-copy-id hadoop102
[djm@hadoop102 .ssh]$ ssh-copy-id hadoop103
[djm@hadoop102 .ssh]$ ssh-copy-id hadoop104
[djm@hadoop103 .ssh]$ ssh-copy-id hadoop102
[djm@hadoop103 .ssh]$ ssh-copy-id hadoop103
[djm@hadoop103 .ssh]$ ssh-copy-id hadoop104
[djm@hadoop104 .ssh]$ ssh-copy-id hadoop102
[djm@hadoop104 .ssh]$ ssh-copy-id hadoop103
[djm@hadoop104 .ssh]$ ssh-copy-id hadoop104

5、下载第三方依赖

# 所有节点
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb httpd mod_ssl

6、创建 CM 用的数据库

create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

7、解压 cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz

mkdir /opt/module/cloudera-manager
tar -zxvf /opt/software/cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz -C /opt/module/cloudera-manager/

8、创建 cloudera-scm 用户

# 所有节点
useradd --system --home=/opt/module/cloudera-manager/cm-5.16.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

--system 创建一个系统账户

--home 指定用户登入时的主目录，替换系统默认值/home/<用户名>

--no-create-home 不要创建用户的主目录

--shell 用户的登录 shell 名

--comment 用户的描述信息

9、配置 agent

vim /opt/module/cloudera-manager/cm-5.16.2/etc/cloudera-scm-agent/config.ini

# Hostname of the CM server.
server_host=hadoop102

10、初始化数据库

创建用户并授权

CREATE USER 'scm'@'%' IDENTIFIED BY 'scm';
grant all on *.* to 'scm'@'%' identified by 'scm'  with grant option;
flush privileges;

拷贝 JDBC 驱动到目录 /usr/share/java/

cp /opt/software/mysql-connector-java-5.1.27-bin.jar /usr/share/java/
mv /usr/share/java/mysql-connector-java-5.1.27-bin.jar /usr/share/java/mysql-connector-java.jar

创建 cm 数据库

/opt/module/cloudera-manager/cm-5.16.2/share/cmf/schema/scm_prepare_database.sh mysql -h hadoop102 -uroot -p123456 --scm-host hadoop102 scm scm scm

-h：Database host

-u：Database username

-p：Database Password

--scm-host：SCM server's hostname

scm scm scm：database_name username password

11、创建 Parcel目录

mkdir -p /opt/cloudera/parcel-repo
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
cp /opt/software/CDH* manifest.json /opt/cloudera/parcel-repo/
mv /opt/software/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 /opt/software/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha

# 所有节点
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

12、分发

scp -r /opt/module/cloudera-manager/ hadoop103:/opt/module/
scp -r /opt/module/cloudera-manager/ hadoop104:/opt/module/
scp -r /opt/cloudera/parcel-repo/ hadoop103:/opt/cloudera/
scp -r /opt/cloudera/parcel-repo/ hadoop104:/opt/cloudera/

13、启动和关闭

/opt/module/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-server start
/opt/module/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-server stop

# 所有节点
/opt/module/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-agent start
/opt/module/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-agent stop