在线部署CDH

1. 本文主要记录了 在线安装CDH的步骤，其中 JDK 和 数据库 按需安装。

2. 服务器系统为CentOS，不同系统命令可能会有所不同。

3. 在线安装的优势在于 方便，操作简单，基本跟着步骤走下来问题不大。缺点也很明显，下载慢，在内网环境或者网络环境不好的时候，在线安装的弊端就显现出来了，所以内网或者网络不好更推荐离线部署的方式。

1. 准备工作

1.1 虚拟机

我这边准备了3台虚拟机 ， Linux版本为：CentOS

• 192.168.44.128 - 部署 scm-serser 和 scm-agent
• 192.168.44.129 - 部署 scm-agent
• 192.168.44.130 - 部署 scm-agent

1.2 设置所有节点的hosts⽂件

vim /etc/hosts

# add 
192.168.44.128 vmware128
192.168.44.129 vmware129
192.168.44.130 vmware130

1.3 关闭防火墙

sudo iptables-save > ~/firewall.rules

sudo systemctl disable firewalld
sudo systemctl stop firewalld

1.4 禁用SELinux

vim /etc/selinux/config
# 将 SELINUX=enforcing 改为 SELINUX=disabled
SELINUX=disabled

修改之后重启机器

1.5 安装ntp

yum install ntp

vim /etc/ntp.conf
# add
server vmware128
server vmware129
server vmware130

sudo systemctl start ntpd
sudo systemctl enable ntpd

ntpdate -u vmware128
hwclock --systohc

1.6 更多准备工作

更多准备工作参考：https://docs.cloudera.com/documentation/enterprise/6/6.0/topics/installation_reqts.html

2. 安装CM-server

2.1 配置Cloudera Manager仓库

sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

2.2 安装JDK

JDK版本要求在1.8及以上，如果已经安装过JDK（需要配置对应环境变量），可以跳过此步。JDK要求详见： https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html#java_requirements

sudo yum install oracle-j2sdk1.8

2.3 安装CM server

这一步在线下载时间会比较长，在规划的主机上执行：

这一步安装了 daemons server agent 三部分

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

2.4 安装配置CM agent

2.4.1 安装CM agent

在其他主机上执行，这一步是安装 agent，也可以在server起来后再装

这一步是在其他主机上安装了 daemons agent

sudo yum install cloudera-manager-daemons cloudera-manager-agent

2.4.2 配置agent

vim /etc/cloudera-scm-agent/config.ini

## 修改下面配置
## Name of the host where Cloudera Manager Server is running.
server_host vmware128

## Port on the host where Cloudera Manager Server is running.
server_port 7180

2.4.3 启动agent

sudo systemctl start cloudera-scm-agent

2.5 安装数据库(MySQL)

如果已有 MySQL（MariaDB） 或 其他数据库(PostgreSQL ,oracle)，此步骤可以跳过。

更多数据库及版本兼容性详见： https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_database_requirements.html#cdh_cm_supported_db

2.5.1 安装MySQL

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm

sudo yum update

sudo yum install mysql-server

sudo systemctl start mysqld

2.5.2 配置MySQL

1. 停掉MySQL

   sudo systemctl stop mysqld
   

2. 编辑 /etc/my.cnf :

   官方推荐配置

   [mysqld]
   datadir=/var/lib/mysql
   socket=/var/lib/mysql/mysql.sock
   transaction-isolation = READ-COMMITTED
   # Disabling symbolic-links is recommended to prevent assorted security risks;
   # to do so, uncomment this line:
   symbolic-links = 0
   
   key_buffer_size = 32M
   max_allowed_packet = 32M
   thread_stack = 256K
   thread_cache_size = 64
   query_cache_limit = 8M
   query_cache_size = 64M
   query_cache_type = 1
   
   max_connections = 550
   #expire_logs_days = 10
   #max_binlog_size = 100M
   
   #log_bin should be on a disk with enough free space.
   #Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
   #system and chown the specified folder to the mysql user.
   log_bin=/var/lib/mysql/mysql_binary_log
   
   #In later versions of MySQL, if you enable the binary log and do not set
   #a server_id, MySQL will not start. The server_id must be unique within
   #the replicating group.
   server_id=1
   
   binlog_format = mixed
   
   read_buffer_size = 2M
   read_rnd_buffer_size = 16M
   sort_buffer_size = 8M
   join_buffer_size = 8M
   
   # InnoDB settings
   innodb_file_per_table = 1
   innodb_flush_log_at_trx_commit  = 2
   innodb_log_buffer_size = 64M
   innodb_buffer_pool_size = 4G
   innodb_thread_concurrency = 8
   innodb_flush_method = O_DIRECT
   innodb_log_file_size = 512M
   
   [mysqld_safe]
   log-error=/var/log/mysqld.log
   pid-file=/var/run/mysqld/mysqld.pid
   
   sql_mode=STRICT_ALL_TABLES
   

2.5.3 启动MySQL

sudo systemctl enable mysqld
sudo systemctl start mysqld

设置MySQL root用户密码

sudo /usr/bin/mysql_secure_installation

选项按照下列选择：

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

2.5.4 安装 MySQL JDBC driver

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

tar zxvf mysql-connector-java-5.1.46.tar.gz

sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

2.5.5 创建数据库

mysql -u root -p

mysql> CREATE DATABASE <database> DEFAULT CHARACTER SET <character set> DEFAULT COLLATE utf8_general_ci;

mysql> GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';

根据需要创建对应的数据库

Service	Database	User
Cloudera Manager Server	scm	scm
Activity Monitor	amon	amon
Reports Manager	rman	rman
Hue	hue	hue
Hive Metastore Server	metastore	hive
Sentry Server	sentry	sentry
Cloudera Navigator Audit Server	nav	nav
Cloudera Navigator Metadata Server	navms	navms
Oozie	oozie	oozie

这里创建了 scm 用于存放 CM的元数据，amon 用于存放CM的监控数据。

create database scm DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
flush privileges;    

2.6 配置CM数据库

1. 如果数据库装在本机上：

   #sudo /opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType> <databaseName> <databaseUser>
   
   sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
   

2. 如果装在其他机器上：

   #sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm
   
   # -P 指定数据库端口，默认3306
   sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h vmware128 -P 3306  --scm-host vmware128 scm scm
   

2.7 启动CM server

sudo systemctl start cloudera-scm-server

## 查看日志
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

## 如果启动成功会有：
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

web管理地址：http://vmware128:7180，初始帐号密码 ：admin 。

后续配置服务都可以在 web上操作了。

3. 相关命令

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
sudo yum install cloudera-manager-daemons cloudera-manager-agent

sudo systemctl start cloudera-scm-server
sudo systemctl start cloudera-scm-agent

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';

4. 参考资料

Cloudera Installation Guide