主要参考官方文档进行安装。
安装与配置MySQL 数据库
$ sudo apt-get install mysql-server
$ sudo apt-get install libmysql-java
根据文档进行配置，使用InnoDB 引擎，添加并更新以下设置。
$ sudo vim /etc/mysql/my.cnf
[mysqld]
max_connections = 100
# InnoDB settings
innodb_additional_mem_pool_size = 16M
innodb_data_file_path = ibdata1:512M:autoextend
innodb_log_files_in_group = 2
transaction-isolation = READ-COMMITTED
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 512M
innodb_thread_concurrency = 5
innodb_flush_method = O_DIRECT
innodb_log_file_size = 256M
删除/var/lib/mysql 目录下的ib_logfile0、ib_logfile1 和ibdata1 文件后，重启mysql 服务。
登录MySQL，创建以下数据库并授权给用户hadoop。
create database amon DEFAULT CHARACTER SET utf8;
create database rman DEFAULT CHARACTER SET utf8;
create database metastore DEFAULT CHARACTER SET utf8;
create database sentry DEFAULT CHARACTER SET utf8;
create database nav DEFAULT CHARACTER SET utf8;
create database navms DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
grant all on rman.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
grant all on metastore.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
grant all on sentry.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
grant all on nav.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
grant all on navms.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
create database scm DEFAULT CHARACTER SET utf8;
grant all on scm.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
安装Cloudera Manager
# master执行
$ mkdir ~/cdh_install
$ cd ~/cdh_install
$ wget https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/cloudera.list
$ sudo cp cloudera.list /etc/apt/sources.list.d/
$ wget https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key -O archive.key
$ sudo apt-key add archive.key
$ sudo apt-get update
$ sudo apt-get install oracle-j2sdk1.7
$ sudo apt-get install cloudera-manager-daemons cloudera-manager-server
如果要安装指定版本，需要修改cloudera.list 文件，默认安装的最新的版本号。
deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm5.6.0 contrib
deb-src http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm5.6.0 contrib
运行脚本初始化数据库schema。
$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql scm hadoop
更改配置文件/etc/init.d/cloudera-scm-server，添加以下内容
USER=cloudera-scm
GROUP=cloudera-scm
MAX_PERMGEN=256
MAX_HEAP=2048
启动cloudera-scm-sever，等待几分钟后，可以通过浏览器访问http://localhost:7180 进入管理界面，用户名和密码默认为admin。
$ sudo service cloudera-scm-server start
使用Cloudera Manager 管理Agent 节点
进入管理界面后
- 选择安装Cloudera Express 版本。
- 添加集群内的所有节点（包括master），多个IP 地址使用逗号分隔。
- 选择Parcel 方式进行安装，同时添加Kafka parcel。
- 安装JDK。
- 启用Single User Mode。
- ssh 登录中选择devops 用户。
安装成功后，所有节点通过cloudera-scm-agent 进行管理。单用户模式下默认账号为cloudera-scm。根据官方文档进行单用户模式下的配置，以便之后的服务安装有足够的权限。
使用Cloudera Manger 安装服务
首先安装并启动Cloudera Management Service，在master 上配置Host Monitor、Service Monitor、Event Server、Alert Publisher。配置完成后可以监控到每台host 的健康状况。
由于各个服务之间有依赖关系，请按照以下顺序进行安装启动，并以相反的顺序停止服务。
ZooKeeper -> HDFS -> YARN ->HBase
因为启用了Single User Mode，在安装服务的过程中，要确保cloudera-scm 用户有权限访问相关的目录。
$ sudo mkdir -p /var/lib/cloudera-service-monitor
$ sudo mkdir -p /var/lib/cloudera-host-monitor
$ sudo mkdir -p /var/lib/cloudera-scm-eventserver
$ sudo mkdir -p /var/lib/cloudera-scm-navigator
$ sudo mkdir -p /var/lib/cloudera-scm-headlamp
$ sudo mkdir -p /var/lib/cloudera-scm-firehose
$ sudo mkdir -p /var/lib/cloudera-scm-alertpublisher
$ sudo mkdir -p /var/log/cloudera-service-monitor
$ sudo mkdir -p /var/log/cloudera-host-monitor
$ sudo mkdir -p /var/log/cloudera-scm-eventserver
$ sudo mkdir -p /var/log/cloudera-scm-navigator
$ sudo mkdir -p /var/log/cloudera-scm-headlamp
$ sudo mkdir -p /var/log/cloudera-scm-firehose
$ sudo mkdir -p /var/log/cloudera-scm-alertpublisher
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-service-monitor
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-host-monitor
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-eventserver
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-navigator
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-headlamp
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-firehose
$ sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-alertpublisher
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-service-monitor
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-host-monitor
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-eventserver
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-navigator
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-firehose
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-alertpublisher
配置HDFS
$ sudo vim /etc/security/limits.conf
soft memlock unlimited
hard memlock unlimited
HDFS 默认的备份数为3，建议至少要有3个DataNode，否则会出现under replicated block 警告。
配置HIVE 数据库
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hadoop'@'%' IDENTIFIED BY 'password';
配置Kafka
更改csd.sh 文件提升调用命令的权限，将以下内容中的exec 更改为bash
sudo vim /usr/lib/cmf/service/csd/csd.sh
bash $CONF_DIR/$CM_CSD_SCRIPT "$@"
参考
Cloudera 官方安装步骤
Cloudera 官方MySQL 配置
Cloudera 官方Single User Mode 配置
在Ubuntu 14.04下通过Cloudera CDH 5.4.8搭建Hadoop 集群
网友评论