美文网首页
Ambari 2.6.2安装Hadoop hbase集群

Ambari 2.6.2安装Hadoop hbase集群

作者: 大道至简非简 | 来源:发表于2018-06-08 16:19 被阅读907次

    一、环境准备

    1、集群规划

    hadoop01/192.168.1.151 hadoop02/192.168.1.152 hadoop03/192.168.1.153 hadoop04/192.168.1.154 hadoop05/192.168.1.155
    ambari server 是 否 否 否 否
    ambari agent 是 是 是 是 是
    namenode 是 是 否 否 否
    datanode 否 否 是 是 是
    resourcemanager 是 是 否 否 否
    journalnode 是 是 是 是 是
    zookeeper 是 是 是 否 否

    01和02主要做管理机，03-05做数据数据节点，01-03做zookeeper集群。

    2、安装一台lnmp备用

    选择hadoop01

    https://lnmp.org/install.html

    wget -c http://soft.vpser.net/lnmp/lnmp1.5.tar.gz && tar zxf lnmp1.5.tar.gz && cd lnmp1.5 && ./install.sh lnmp

    选择安装mysql5.7版本用于支持ambari

    3、下载离线版所需文件

    https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.0/bk_ambari-installation/content/ch_obtaining-public-repos.html




    下载备用。

    二、安装配置ambari

    1、配置离线仓库地址
    mkdir /home/wwwroot/default/ambari
mkdir /home/wwwroot/default/ambari/HDP-UTILS
cd /home/wwwroot/default/ambari

    复制下载文件到相关目录并解压

    tar -zxvf ambari-2.6.2.0-ubuntu16.tar.gz
tar -zxvf HDP-GPL-2.6.5.0-ubuntu16-gpl.tar.gz
tar -zxvf HDP-2.6.5.0-ubuntu16-deb.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-ubuntu16.tar.gz -C HDP-UTILS

    开启nginx对ambari目录的浏览功能。


    2、制作list文件并复制到各台agent主机

    cd /etc/apt/sources.list.d
    ambari.list

    deb http://192.168.1.151/ambari/ambari/ubuntu16/2.6.2.0-155/ Ambari main

    ambari-hdp.list

    deb http://192.168.1.151/ambari/HDP/ubuntu16/2.6.5.0-292/ HDP main
deb http://192.168.1.151/ambari/HDP-GPL/ubuntu16/2.6.5.0-292/ HDP-GPL main
deb http://192.168.1.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu16/1.1.0.22/ HDP-UTILS main

    更新list

    sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
sudo apt-get update
apt-cache showpkg ambari-server
apt-cache showpkg ambari-agent
apt-cache showpkg ambari-metrics-assembly
    3、安装ambari
    sudo apt-get install ambari-server


root@hadoop01:/home# sudo apt-get install ambari-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libsensors4 postgresql postgresql-9.5 postgresql-client-9.5 postgresql-client-common postgresql-common postgresql-contrib-9.5 ssl-cert sysstat
Suggested packages:
  lm-sensors postgresql-doc locales-all postgresql-doc-9.5 libdbd-pg-perl openssl-blacklist isag
The following NEW packages will be installed:
  ambari-server libsensors4 postgresql postgresql-9.5 postgresql-client-9.5 postgresql-client-common postgresql-common postgresql-contrib-9.5 ssl-cert sysstat
0 upgraded, 10 newly installed, 0 to remove and 48 not upgraded.
Need to get 757 MB of archives.
After this operation, 831 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://192.168.1.151/ambari/ambari/ubuntu16/2.6.2.0-155 Ambari/main amd64 ambari-server amd64 2.6.2.0-155 [752 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 libsensors4 amd64 1:3.4.0-2 [28.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-common all 173ubuntu0.1 [28.4 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-9.5 amd64 9.5.13-0ubuntu0.16.04 [871 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 ssl-cert all 1.0.37 [16.9 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-common all 173ubuntu0.1 [154 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-9.5 amd64 9.5.13-0ubuntu0.16.04 [3,001 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql all 9.5+173ubuntu0.1 [5,450 B]                                                                                                                                                                
Get:9 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-contrib-9.5 amd64 9.5.13-0ubuntu0.16.04 [448 kB]                                                                                                                                              
Get:10 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 sysstat amd64 11.2.0-1ubuntu0.2 [243 kB]                                                                                                                                                                
Fetched 757 MB in 12s (59.8 MB/s)                                                                                                                                                                                                                                            
Preconfiguring packages ...
Selecting previously unselected package libsensors4:amd64.
(Reading database ... 70302 files and directories currently installed.)
Preparing to unpack .../libsensors4_1%3a3.4.0-2_amd64.deb ...
Unpacking libsensors4:amd64 (1:3.4.0-2) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_173ubuntu0.1_all.deb ...
Unpacking postgresql-client-common (173ubuntu0.1) ...
Selecting previously unselected package postgresql-client-9.5.
Preparing to unpack .../postgresql-client-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package ssl-cert.
Preparing to unpack .../ssl-cert_1.0.37_all.deb ...
Unpacking ssl-cert (1.0.37) ...
Selecting previously unselected package postgresql-common.
Preparing to unpack .../postgresql-common_173ubuntu0.1_all.deb ...
Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'
Unpacking postgresql-common (173ubuntu0.1) ...
Selecting previously unselected package postgresql-9.5.
Preparing to unpack .../postgresql-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql.
Preparing to unpack .../postgresql_9.5+173ubuntu0.1_all.deb ...
Unpacking postgresql (9.5+173ubuntu0.1) ...
Selecting previously unselected package postgresql-contrib-9.5.
Preparing to unpack .../postgresql-contrib-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-contrib-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package sysstat.
Preparing to unpack .../sysstat_11.2.0-1ubuntu0.2_amd64.deb ...
Unpacking sysstat (11.2.0-1ubuntu0.2) ...
Selecting previously unselected package ambari-server.
Preparing to unpack .../ambari-server_2.6.2.0-155_amd64.deb ...
Unpacking ambari-server (2.6.2.0-155) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libsensors4:amd64 (1:3.4.0-2) ...
Setting up postgresql-client-common (173ubuntu0.1) ...
Setting up postgresql-client-9.5 (9.5.13-0ubuntu0.16.04) ...
update-alternatives: using /usr/share/postgresql/9.5/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up ssl-cert (1.0.37) ...
Setting up postgresql-common (173ubuntu0.1) ...
Adding user postgres to group ssl-cert

Creating config file /etc/postgresql-common/createcluster.conf with new version

Creating config file /etc/logrotate.d/postgresql-common with new version
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up postgresql-9.5 (9.5.13-0ubuntu0.16.04) ...
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Creating new cluster 9.5/main ...
  config /etc/postgresql/9.5/main
  data   /var/lib/postgresql/9.5/main
  locale en_US.UTF-8
  socket /var/run/postgresql
  port   5432
update-alternatives: using /usr/share/postgresql/9.5/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
Setting up postgresql (9.5+173ubuntu0.1) ...
Setting up postgresql-contrib-9.5 (9.5.13-0ubuntu0.16.04) ...
Setting up sysstat (11.2.0-1ubuntu0.2) ...

Creating config file /etc/default/sysstat with new version
update-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode
Setting up ambari-server (2.6.2.0-155) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
    5、初始配置

    sudo ambari-server setup


    使用自定义jdk
    采用mysql数据库（生产环境采用内置postgresql不合适）

    需要相关jdbc。
    https://dev.mysql.com/downloads/connector/j/

    在主节点把mysql数据库连接包拷贝在/var/lib/ambary-server/resources目录下并改名为mysql-jdbc-driver.jar

    vim /etc/ambari-server/conf/ambari.properties

    Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
......ambari-admin-2.6.2.0.155.jar
.....
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

    三、配置ambari

    1、启动ambari
    root@hadoop01:~# sudo ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........Unable to determine server PID. Retrying...
......Unable to determine server PID. Retrying...
......Unable to determine server PID. Retrying...
ERROR: Exiting with exit code -1. 
REASON: Ambari Server java process died with exitcode 1. Check /var/log/ambari-server/ambari-server.out for more information.

    出现错误。
    导入mysql脚本

    mysql -uroot -p

mysql> use ambari;
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

    再次启动

    root@hadoop01:~# sudo ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

    采用admin/admin登录


    四、配置hadoop



    选择本地repository，填入本地url；



    本地hdp-utils缺失文件错误（2.6文件几十k，2.5版本文件32mb，估计官网问题）。更换成公网地址。
    或者用2.5的文件替换2.6的文件dist目录。

    cat ~/.ssh/id_rsa
    获取ssh密钥。




    解决问题
    etc/ambari-agent/conf/ambari-agent.ini
    [security]
    force_https_protocol=PROTOCOL_TLSv1_2

    关闭thp
    vi /etc/init.d/disable-transparent-hugepages

    #!/bin/sh

### BEGIN INIT INFO

# Provides:  disable-transparent-hugepages

# Required-Start:$local_fs

# Required-Stop:

# X-Start-Before:mongod mongodb-mms-automation-agent

# Default-Start: 2 3 4 5

# Default-Stop:  0 1 6

# Short-Description: Disable Linux transparent huge pages

# Description:   Disable Linux transparent huge pages, to improve

#database performance.

### END INIT INFO

case $1 in

　　start)

　　　　if [ -d /sys/kernel/mm/transparent_hugepage ]; then

　　　　　　thp_path=/sys/kernel/mm/transparent_hugepage

　　　　elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then

　　　　　　thp_path=/sys/kernel/mm/redhat_transparent_hugepage

　　　　else

　　　　　　return 0

　　　　fi

　　　　echo 'never' > ${thp_path}/enabled

　　　　echo 'never' > ${thp_path}/defrag

　　　　unset thp_path

　　;;

esac

    sudo chmod 755 /etc/init.d/disable-transparent-hugepages

sudo update-rc.d disable-transparent-hugepages defaults

    选择需要的组件
    hbase不是最新2.1.0版本。








    经过各种坑，顺利完成。


    配置高可用






    高可用配置完成。

    相关文章

      网友评论

          本文标题：Ambari 2.6.2安装Hadoop hbase集群

          本文链接：https://www.haomeiwen.com/subject/hdjnsftx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|Ambari 2.6.2安装Hadoop hbase集群|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！