美文网首页
Ubuntu18.04安装CDH6.3.1(6.3.2)

Ubuntu18.04安装CDH6.3.1(6.3.2)

作者: john瀚 | 来源:发表于2020-03-30 12:09 被阅读0次

    环境准备

    全部以root用户操作。

    修改每个机器的hostname

    • vim /etc/hosts
    172.16.244.65   clickhouse1
172.16.244.104  clickhouse2
172.16.244.123  clickhouse3
172.16.244.136  clickhouse4
172.16.244.127  clickhouse5
172.16.244.139  clickhouse6

    安装pssh

    pssh是一个python编写可以在多台服务器上执行命令的工具，同时支持拷贝文件，是同类工具中很出色的。

    apt-get install pssh

echo "alias pssh=parallel-ssh" >> ~/.bashrc && . ~/.bashrc
echo "alias pscp=parallel-scp" >> ~/.bashrc && . ~/.bashrc
echo "alias prsync=parallel-rsync" >> ~/.bashrc && . ~/.bashrc
echo "alias pnuke=parallel-nuke" >> ~/.bashrc && . ~/.bashrc
echo "alias pslurp=parallel-slurp" >> ~/.bashrc && . ~/.bashrc

    创建hosts.txt,
    vim ~/hosts.txt

    clickhouse1
clickhouse2
clickhouse3
clickhouse4
clickhouse5
clickhouse6

    打通master机器到slave机器的ssh

    在clickhouse1上执行，并一路回车。

    ssh-keygen -t dsa

    把公钥拷贝到所有被管理机中

    ssh-copy-id -i clickhouse2
ssh-copy-id -i clickhouse3
ssh-copy-id -i clickhouse4
ssh-copy-id -i clickhouse5
ssh-copy-id -i clickhouse6

    测试ssh是否打通

    pssh -h host.txt -i "whoami"

    关闭每台机器的防火墙

    pssh -h host.txt -i "apt-get install ufw"
pssh -h host.txt -i "ufw disable"

    安装ntp

    pssh -h host.txt -i "apt-get install ntp"
pssh -h host.txt -i "systemctl enable ntp"

    编辑clickhouse1的文件 "/etc/ntp.conf"

    driftfile  /var/lib/ntp/drift
pidfile   /var/run/ntpd.pid
logfile /var/log/ntp.log
restrict    default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
server 127.127.1.0
fudge  127.127.1.0 stratum 10
server ntp.aliyun.com iburst minpoll 4 maxpoll 10
restrict ntp.aliyun.com nomodify notrap nopeer noquery

    编辑clickhouse[2-6]的文件 "/etc/ntp.conf"

    添加：

    server clickhouse1 prefer

    开始安装

    添加仓库

    每台机器都需要执行下面的命令。

    pssh -h host.txt -i  "wget https://archive.cloudera.com/cm6/6.3.1/ubuntu1604/apt/archive.key"
pssh -h host.txt -i  "apt-key add archive.key"
pssh -h host.txt -i  "wget https://archive.cloudera.com/cm6/6.3.1/ubuntu1804/apt/cloudera-manager.list"
pssh -h host.txt -i  "mv cloudera-manager.list /etc/apt/sources.list.d/"
pssh -h host.txt -i  "apt-get update"

    下载安装包

    在线安装太慢选择先下载下来，下载这个地址下面的所有文件：

    https://archive.cloudera.com/cm6/6.3.1/ubuntu1804/apt/pool/contrib/e/enterprise/*
# 文件列表
cloudera-manager-agent_6.3.1~1466458.ubuntu1804_amd64.deb   2019-10-11 08:42    9.07MB
cloudera-manager-daemons_6.3.1~1466458.ubuntu1804_all.deb   2019-10-11 08:42    1.15GB
cloudera-manager-server-db-2_6.3.1~1466458.ubuntu1804_all.deb   2019-10-11 08:42    9.51KB
cloudera-manager-server-db_6.3.1~1466458.ubuntu1804_all.deb 2019-10-11 08:42    1.08KB
cloudera-manager-server_6.3.1~1466458.ubuntu1804_all.deb    2019-10-11 08:42    4.77KB

    然后放到每台机器的路径：/var/cache/apt/archives/ 下.
    我是先将上面的五个文件先上传到clickhouse1上，然后通过pssh在分发到其他机器上。分发命令：

    pscp -r -h host.txt /var/cache/apt/archives/cloudera-* /var/cache/apt/archives/

    下载parcels文件

    wget -c https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel
wget -c https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel.sha1
wget -c https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json

    将下载下来的CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel.sha1 改为 CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel.sha
    将下载好的这三个文件分别上传到每个服务器上的路径: /opt/cloudera/parcel-repo/

    # 每个机器先创建文件夹
pssh -h host.txt -i "mkdir -p /opt/cloudera/parcel-repo"

    我是先将上面的五个文件先上传到clickhouse1上，然后通过pssh在分发到其他机器上。分发命令：

    pscp -r -h host.txt /opt/cloudera/parcel-repo/* /opt/cloudera/parcel-repo/

    安装JAVA

    pssh -h host.txt -i "apt-get install openjdk-8-jdk"

    安装Mysql

    安装在在clickhouse1下,涉及到数据的操作都在这下面执行，因为Cloudera Manager Server在clickhouse1下启动。

    apt-get install -yq mysql-server mysql-client libmysqlclient-dev libmysql-java

vim /etc/mysql/mysql.conf.d/mysqld.cnf 
去掉 
bind-address = 127.0.0.1

#初始化mysql 记得更改root密码
sudo mysql_secure_installation

#创建数据库并授权
sudo mysql -h27.0.0.1 -uroot -p

    创建CDH依赖的数据以及用户

    -- 创建数据库
-- Cloudera Manager Server
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Activity Monitor
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Reports Manager
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Hive Metastore Server
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Sentry Server
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Cloudera Navigator Audit Server
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Cloudera Navigator Metadata Server
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 
#创建用户并授权
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123456';
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123456';

    设置Cloudera Manager数据库

    sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon amon123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman rman123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue hue hue123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hive hive hive123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry sentry sentry123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav nav123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms navms123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie oozie oozie123456

    安装cloudera

    每个机器都需要执行。由于我们提前离线下载好了安装文件，并放入到了路径/var/cache/apt/archives/下，所以下面的命令跳过了下载执行很快。

    pssh -h host.txt -i "apt-get install -yq cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server"

    启动

    启动cloudera-scm-server

    在clickhouse1启动

    systemctl start cloudera-scm-server
#查看启动日志，等待Jetty启动完成
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

    启动cloudera-scm-agent

    编辑每台机器的这个文件。
    vim /etc/cloudera-scm-agent/config.ini

    [General]
# Hostname of the CM server.
server_host=clickhouse1

    启动cloudera-scm-agent,每台机器都启动

    pssh -h host.txt -i "systemctl start cloudera-scm-agent"

#查看启动日志，等待Jetty启动完成
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

    访问cloudera-manager

    • 浏览器输入http://clickhouse1:7180/
    • 用户/密码：admin/admin
    • 按照向导搭建集群。
    • 有问题查看日志解决

    相关文章

      网友评论

          本文标题：Ubuntu18.04安装CDH6.3.1(6.3.2)

          本文链接：https://www.haomeiwen.com/subject/xlukvctx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|Ubuntu18.04安装CDH6.3.1(6.3.2)|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！