数据仓库环境准备-Hadoop篇

作者: 枫叶无言_1997 | 来源:发表于2021-01-31 17:30 被阅读0次

HBase | 集群部署
Hadoop 部署使用
大数据
Kafka集群安装部署
Hadoop-入门篇环境搭建（四）
Linux下Hadoop服务Hive、Flume数据处理部署
hadoop学习笔记
hadoop框架学习笔记三 2020-04-01
spark yarn集群搭建（一：准备工作）
大数据相关

大数据软件版本说明:

hadoop-3.1.4、zookeeper-3.5.8、kafka_2.12-2.6.0、flume-1.9.0、sqoop-1.4.6、hive-3.1.2、mysql-5.7.31-1.el7、spark-3.0.0

一、JDK安装

1.移除OpenJDK命令:sudo rpm -qa | grep -i java | xargs -n1 sudo rpm -e --nodeps

2.修改/opt目录权限: sudo chmod -r 777 /opt

3.解压jdk至目录: tar -zxvf jdk-8u171-linux-x64.tar.gz -C /opt/module/

4.配置环境变量: sudo vim /etc/profile.d/my_env.sh

5.my_env.sh:

#JAVA_HOME

export JAVA_HOME=/opt/module/jdk1.8.0_171

export PATH=$PATH:$JAVA_HOME/bin

6.source /etc/profile.d/my_env.sh

二、Hadoop配置:

<name>fs.defaultFS</name>

<value>hdfs://hadoop102:8020</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/opt/module/hadoop-3.1.4/data</value>

</property>

<name>hadoop.http.staticuser.user</name>

<value>linan</value>

</property>

<name>hadoop.proxyuser.linan.groups</name>

</property>

<name>hadoop.proxyuser.linan.hosts</name>

</property>

<name>hadoop.proxyuser.linan.users</name>

</property>

<name>dfs.namenode.http-address</name>

<value>hadoop102:9870</value>

</property>

<name>dfs.namenode.secondary.http-address</name>

<value>hadoop104:9868</value>

</property>

<name>dfs.replication</name>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.resourcemanager.hostname</name>

<value>hadoop103</value>

</property>

<name>yarn.nodemanager.env-whitelist</name>

<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>

</property>

<name>yarn.scheduler.minimum-allocation-mb</name>

</property>

<name>yarn.scheduler.maximum-allocation-mb</name>

</property>

<name>yarn.nodemanager.resource.memory-mb</name>

</property>

<name>yarn.nodemanager.pmem-check-enabled</name>

<value>false</value>

</property>

<name>yarn.nodemanager.vmem-check-enabled</name>

<value>false</value>

</property>

<name>yarn.log-aggregation-enable</name>

</property>

<name>yarn.log.server.url</name>

<value>http://hadoop102:19888/jobhistory/logs</value>

</property>

<name>yarn.log-aggregation.retain-seconds</name>

</property>

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

<value>hadoop102:10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hadoop102:19888</value>

</property>

/opt/module/hadoop-3.1.4/etc/hadoop/workers:

hadoop102

hadoop103

hadoop104

/opt/module/hadoop-3.1.4/etc/hadoop/hadoop-env.sh:

export JAVA_HOME=/opt/module/jdk1.8.0_171

export HADOOP_HOME=/opt/module/hadoop-3.1.4

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

//格式化namenode

rm -rf logs/ data/

bin/hdfs namenode -format

启动:

sbin/start-dfs.sh

sbin/start-yarn.sh

停止:

sbin/stop-dfs.sh

sbin/stop-yarn.sh

测试:

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar pi 1 1

//批量显示脚本xcall:

#!/bin/bash

params=$@

i=2

for((i=2 ; i <= 4 ; i = $i + 1)) ; do

echo ==============hadoop10$i $params =============

ssh hadoop10$i "source /etc/profile;$params"

done

集群数据均衡

1、节点间数据均衡

开启数据均衡命令：start-balancer.sh -threshold 10

停止数据均衡命令：stop-balancer.sh

2、磁盘间数据均衡(hadoop3才有)

1)生成均衡计划

hdfs diskbalancer -plan hadoop103

2)执行均衡计划

hdfs diskbalancer -execute hadoop103.plan.json

3)查看当前均衡任务的执行情况

hdfs diskbalancer -query hadoop103

4)取消均衡计划

hdfs diskbalancer -cancel hadoop103.plan.json

Hadoop支持lzo压缩配置

lzo编译源码地址:

https://github.com/twitter/hadoop-lzo

https://www.oberhumer.com/opensource/lzo/

编译lzo源码生成hadoop-lzo-0.4.21.jar包

将编译好的hadoop-lzo-0.4.21.jar放入/opt/module/hadoop-3.1.4/share/hadoop/common目录下

参考地址:

(1)https://wenku.baidu.com/view/61a42f9f0875f46527d3240c844769eae009a3f4.html

(2)https://blog.csdn.net/s_alics/article/details/108513408

core-site配置支持lzo

<name>io.compression.codecs</name>

<value>

org.apache.hadoop.io.compression.SnappyCodec,

com.hadoop.compression.lzo.LzoCodec,

com.hadoop.compression.lzo.LzopCodec,

</value>

</property>

<name>io.compression.codec.lzo.class</name>

<value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

测试案例

1、

hadoop fs -mkdir /input

hadoop fs -put word.txt /input

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec /input /output

使用lzo压缩方式支持切片需先创建lzo文件索引

例：bigtable.lzo文件

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/common/hadoop-lzo-0.4.21.jar

com.hadoop.compression.lzo.DistributedLzoIndexer /input /bigtable.lzo

HDFS调优

hdfs-site:

<name>dfs.namenode.handler.count</name>

</property>

公式:dfs.namenode.handler.count = 20 * log小e3 = 21

yarn-site:

<name>dfs.namenode.handler.count</name>

</property>

基准测试

1）hdfs写性能

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB

2）hdfs读性能

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB

3）删除测试数据

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -clean