一、基础环境设置

注意：基础环境需在每台集群服务器设置。

1.1、软件环境

编号	软件名称	软件版本
1	操作系统	CentOS Linux release 7.5.1804 (Core)
2	JDK	jdk-8u144-linux-x64
3	Scala	scala-2.11.8

1.2、服务器规划

编号	服务器名称	内网IP	内存	用途
1	cdhserver1	192.168.26.21	20G	CM-server、CM-agent、MySQL、HTTP-server
2	cdhserver2	192.168.26.22	9G	CM-agent
3	cdhserver3	192.168.26.23	9G	CM-agent

1.3、配置主机名IP映射

注意：CDH中谨慎使用FQDN,与主机名保持一致

vi /etc/hosts

#添加如下地址映射
192.168.26.21 cdhserver1
192.168.26.22 cdhserver2
192.168.26.23 cdhserver3

1.4、SSH免密登录配置

ssh-keygen 
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver1 
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver2 
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver3

1.5、防火墙配置

为了安装时简单，可以执行以下命令暂时关闭所有防火墙。

systemctl stop firewalld.service systemctl disable firewalld.service firewall-cmd --state

安装完成开启防火墙，设置防火墙策略。

#开启防火墙 
systemctl start firewalld.service 
#设置可以访问的端口 
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.21' port protocol='tcp' port='0-65535' accept" 
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.22' port protocol='tcp' port='0-65535' accept" 
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.23' port protocol='tcp' port='0-65535' accept" 
firewall-cmd –permanent –add-port=0-65535/tcp firewall-cmd –reload

1.6、关闭SELinux

vi /etc/sysconfig/selinux

#修改SELINUX为disabled
SELINUX=disabled

1.7、开启NTP服务实现时间同步

yum install -y ntp systemctl enable ntpd systemctl start ntpd

1.8、安装JDK环境

话说JDK8最新更新的本版不开源了，下载历史本版：jdk-8u181-linux-x64.rpm

https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html

若系统自带openJDK，首先卸载

java -version rpm -qa | grep jdk 
yum -y remove java-xxx-openjdk-xxx.el7_0.x86_64 
yum -y remove java-xxx-openjdk-headless-xxx.el7_0.x86_64

安装JDK

rpm -ivh jdk-8u181-linux-x64.rpm

配置JDK环境变量

vi /etc/profile source /etc/bashrc

export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin

1.9 安装SCALA环境

下载scala-2.11.8 :

https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.rpm 或

https://www.scala-lang.org/files/archive/scala-2.11.8.rpm

rpm -ivh scala-2.11.8.rpm

1.10 修改Linux内核参数

Cloudera 建议将 swappiness 设置为 10（控制换出运行时内存权重）

#查看swappiness  
cat /proc/sys/vm/swappiness  
#永久性修改，执行下面两条命令  
sysctl -w vm.swappiness=10  
echo vm.swappiness = 10 >> /etc/sysctl.conf

修改文件句柄数

vi /etc/security/limits.conf

在limits.conf文件末尾添加如下配置：

# End of file
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072

vi /etc/security/limits.d/20-nproc.conf

修改 ‘* soft nproc 4096’ 成更大的值或注释掉，因为该文件的配置会覆盖limits.conf的配置。

* soft nproc 131072

透明大页面压缩
临时关闭：

echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled

永久关闭：

vi /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local

在文件后添加下面内容: ，每次添加完记得执行chmod +x /etc/rc.d/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

查看状态

cat /sys/kernel/mm/transparent_hugepage/defrag 
cat /sys/kernel/mm/transparent_hugepage/enabled

1.11 重启集群服务器

init 6

二、MYSQL安装

2.1 卸载centos7自带的mariadb

rpm -qa | grep mariadb rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64

2.2 安装依赖

yum install -y net-tools perl wget

2.3 下载MySQL安装包并解压

wget https://cdn.mysql.com//archives/mysql-5.7/mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar 
tar -xvf mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar

mysql-community-server-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-client-5.7.18-1.el7.x86_64.rpm
mysql-community-common-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-5.7.18-1.el7.x86_64.rpm
mysql-community-server-minimal-5.7.18-1.el7.x86_64.rpm
mysql-community-test-5.7.18-1.el7.x86_64.rpm
mysql-community-minimal-debuginfo-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm

2.4 依次安装mysql rpm包

按common–>libs–>client–>server->devel->libs-compat的顺序安装

rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-devel-5.7.18-1.el7.x86_64.rpm 
#安装下面包，否则Hue安装报连接错误 
rpm -ivh mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm

2.5 修改MySQL端口和数据存放路径

mkdir -p /data/mysql/data 
vi /etc/my.cnf

image.png

2.6 启动mysql服务及查看状态

systemctl start mysqld.service systemctl status mysqld.service

2.7 初始化数据库密码

#查看初始密码
grep "password" /var/log/mysqld.log 
mysql -uroot -p 
#默认密码检查策略要求密码必须包含：大小写字母、数字和特殊符号，并且长度不能少于8位 ALTER USER 'root'@'localhost' IDENTIFIED BY 'Root@2019'; 
#远程连接数据库的时候需要输入用户名和密码 
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Root@2019' WITH GRANT OPTION; 
FLUSH PRIVILEGES; 
#查看端口 
show global variables like 'port';

方式二：
修改配置文件/etc/my.cnf令MySQL跳过登录时的权限检验，在[mysqld]下加入一行：

skip-grant-tables
使用 mysql登陆
mysql> use mysql;
mysql> UPDATE user SET authentication_string = password('root') WHERE host = 'localhost' AND user = 'root';
mysql> select host,user, authentication_string, password_expired from user;
mysql> update user set password_expired='N' where password_expired='Y' //密码不过期
mysql> update user set host='%' where user='root' and host='localhost'; //远程可访问
mysql> flush privileges; //刷新
mysql> exit;//退出

删除skip-grant-tables

2.8 设置开机启动

systemctl enable mysqld systemctl daemon-reload