美文网首页
Big Data 笔记: 搭建Hadoop环境

Big Data 笔记: 搭建Hadoop环境

作者: Yanring_ | 来源:发表于2017-05-14 17:33 被阅读0次

    环境:Vmware+Ubuntu16

    配置网络

    将网络配置为静态

    image.png

    更改为合适的hostname

    image.png

    将hostname和本机ip绑定

    image.png

    ping 自己测试下

    image.png

    关闭防火墙

    我们知道,在安装hadoop的时候会打开很多端口,由于是小打小闹,所以不必去一个端口一个端口的开放,我们就直接全部开放,那么关闭防火墙就可以了
    注意,防火墙的服务名不是firewalls而是iptables
    下面我们关闭防火墙

    image.png

    为了防止防火墙在重启电脑的时候重新启动,那么我们需要执行关闭防火墙自启动功能
    sudo ufw disable

    安装jdk

    安装JDK(至少需要Oracle JDK7及以上)

    java -version
    sudo add-apt-repository ppa:webupd8team/java
    sudo apt-get update
    sudo apt-get install oracle-java7-installer
    sudo apt-get install oracle-java7-set-default
    

    通过命令java -version查看是否安装成功
    然后添加JAVA_HOME和Path,在我这是在~/.bashrc里加入:

    export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jrelib:$CLASSPATH
    export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
    echo "PATH=$PATH" >> ~/.zshrc 
    //我用的是zsh,所以更改的是.zsh而不是.bashrc
    
    保存后source ~/.zshrc,通过命令echo $JAVA_HOME查看是否成功。
    ####安装Mysql并设置远程登录
    1. 开启远程访问 参考:http://blog.csdn.net/u012348345/article/details/51226937
    2. 配置
    

    service mysql stop #停止服务
    vi /etc/mysql/conf.d/mysql_cloudera_manager.cnf #添加配置文件

    写入
    

    [mysqld]
    transaction-isolation = READ-COMMITTED

    Disabling symbolic-links is recommended to prevent assorted security risks;

    to do so, uncomment this line:

    symbolic-links = 0

    key_buffer = 16M
    key_buffer_size = 32M
    max_allowed_packet = 32M
    thread_stack = 256K
    thread_cache_size = 64
    query_cache_limit = 8M
    query_cache_size = 64M
    query_cache_type = 1
    max_connections = 550

    expire_logs_days = 10

    max_binlog_size = 100M

    log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system

    and chown the specified folder to the mysql user.

    log_bin=/var/lib/mysql/mysql_binary_log

    For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.

    binlog_format = mixed
    read_buffer_size = 2M
    read_rnd_buffer_size = 16M
    sort_buffer_size = 8M
    join_buffer_size = 8M

    InnoDB settings

    innodb_file_per_table = 1
    innodb_flush_log_at_trx_commit = 2
    innodb_log_buffer_size = 64M
    innodb_buffer_pool_size = 4G
    innodb_thread_concurrency = 8
    innodb_flush_method = O_DIRECT
    innodb_log_file_size = 512M
    [mysqld_safe]
    log-error=/var/log/mysqld.log
    pid-file=/var/run/mysqld/mysqld.pid
    sql_mode=STRICT_ALL_TABLES

    
    ####安装ntp
    sudo apt-get install ntp,重启服务sudo service ntp restart
    #安装Cloudera Manager
    ####添加CDH5 Repository
    Ubuntu Xenial (16.04):  wget https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/cloudera.list -O /etc/apt/sources.list.d/cloudera.list
    Ubuntu Trusty (14.04):  wget https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/cloudera.list -O /etc/apt/sources.list.d/cloudera.list
    Ubuntu Precise (12.04): wget https://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/cloudera.list -O /etc/apt/sources.list.d/cloudera.list
    Ubuntu Lucid (10.04):   wget https://archive.cloudera.com/cm5/ubuntu/lucid/amd64/cm/cloudera.list -O /etc/apt/sources.list.d/cloudera.list
    ####添加Key
    
    
    curl -s http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/archive.key | sudo apt-key add -
    
    > 如果是Trusty的操作系统,还需要以下一个附加步骤:
    创建一个新文件/etc/apt/preferences.d/cloudera.pref
    内如如下:
    Package: *
    Pin: release o=Cloudera, l=Cloudera
    Pin-Priority: 501
    
    未完待续:
    参考:
    https://www.zybuluo.com/ncepuwanghui/note/474966
    http://kakack.github.io/2016/03/Cloudera-CDH5%E5%AE%89%E8%A3%85%E9%85%8D%E7%BD%AE%E7%BB%86%E5%88%99/

    相关文章

      网友评论

          本文标题:Big Data 笔记: 搭建Hadoop环境

          本文链接:https://www.haomeiwen.com/subject/bmutxxtx.html