步骤

• 配置环境
  • ntp 安装
  • 阿里 yum repo
  • ssh 免密
• 安装java
• 安装配置cm
• 安装配置mysql

机器

IP	Hostname	remark
172.30.3.210	wei-data1	主节点
172.30.3.211	wei-data2	从节点
172.30.3.212	wei-data3	从节点

环境

1. Network

• 修改ip [each machine]
• 修改网络类型(static) [each machine]
• 修改DNS [each machine]
• 添加Host [each machine]

172.30.3.210 wei-data1
172.30.3.211 wei-data2
172.30.3.212 wei-data3

• Firewall[each machine]

systemctl stop firewalld.service
systemctl disable firewalld.service

vi /etc/selinux/config
SELINUX=disabled

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

• ssh免密登陆

#从节点
ssh-keygen -t rsa

#主节点
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@wei-data1
scp ~/.ssh/authorized_keys root@wei-data2:~/.ssh/
scp ~/.ssh/authorized_keys root@wei-data3:~/.ssh/

2. 阿里 yum repo [each machine]

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

3. lib install [each machine]

yum install net-tools vim -y

4. 安装NTP [each machine]

• 安装

yum install  -y ntp

• 配置

vim /etc/ntp.conf

#主节点
server time.windows.com prefer  #time.windows.com是时间服务器，在内网环境可让用户提供统一的时间服务器地址

//注释掉
#server 0.centos.pool.ntp.org iburst  #在这行上面增加内容
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#从节点
server wei-data1
//注释掉
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

# 所有节点
systemctl enable ntpd
systemctl start ntpd

5. java安装 [each machine]（省略）

rpm -ivh -i java...

6. CM安装[each machine]

• yum repo 地址

https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo

安装yum repo

wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/

安装cm[后续会自动安装agent,所以这里不装 ]

yum install cloudera-manager-daemons cloudera-manager-server

7. mysql安装 [master node]

• 安装

 wget http://repo.mysql.com/mysql57-community-release-el7-10.noarch.rpm
 yum -y install mysql57-community-release-el7-10.noarch.rpm
 yum -y install mysql-community-server

• 启动

# 开机启动
systemctl enable mysqld.service
# 启动
systemctl start mysqld.service

修改mysql配置

停服务

systemctl stop mysqld.service

• 改配置

vim /etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

启动mysql

systemctl start mysqld

• 初始化mysql (需要复杂密码)

/usr/bin/mysql_secure_installation

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

• Installing the MySQL JDBC Driver

1. Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html
2. Extract the JDBC driver JAR file from the downloaded file. For example:
   tar zxvf mysql-connector-java-5.1.46.tar.gz
3. Copy the JDBC driver, renamed, to /usr/share/java/. If the target directory does not yet exist, create it. For example:

mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

• 创建数据库

mysql -u root -p

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Bigdata@123';

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Bigdata@123';

• Running the script when MySQL or MariaDB is co-located with the Cloudera Manager Server

/usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm

8. 安装CDH

   1. Start Cloudera Manager Server:
      sudo systemctl start cloudera-scm-server
   2. Wait several minutes for the Cloudera Manager Server to start. To observe the startup process, run the following on the Cloudera Manager Server host:
      sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

#When you see this log entry, the Cloudera Manager Admin Console is ready:
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

• In a web browser, go to http://<server_host>:7180, where <server_host> is the FQDN or IP address of the host where the Cloudera Manager Server is running.

Log into Cloudera Manager Admin Console. The default credentials are:
Username: admin
Password: admin

9. 其他

检查主机正确性时出现 “Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 30。” 的警告，进行如下设定

# vi /etc/sysctl.conf
vm.swappiness = 0
# sysctl –p

检查主机正确性时出现 “已启用“透明大页面”，它可能会导致重大的性能问题。” 的警告，进行如下设定

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

10. 安装引导(略) 网上很多

11. kafka集成

    
    image.png

• 注意事项
  jdk要1.8
  修改内存限制到256m

12. spark2 安装

image.jpeg

-在官网下载

# cd /opt/cloudera/csd/
wget http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar
chown cloudera-scm:cloudera-scm SPARK2_ON_YARN-2.3.0.cloudera4.jar
# cd /opt/cloudera/parcel-repo/
wget http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha
mv SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha1 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha
wget http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel
wget http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/manifest.json

chown cloudera-scm:cloudera-scm SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel*

chown cloudera-scm:cloudera-scm manifest.json