美文网首页
（二）Hadoop集群环境搭建(伪分布式)

（二）Hadoop集群环境搭建(伪分布式)

作者: 小猪Harry | 来源:发表于2018-09-10 01:54 被阅读0次

Hadoop部署模式有：本地模式、伪分布模式、完全分布式模式。
区分的依据是NameNode、DataNode、ResourceManager、NodeManager等模块运行在几个JVM进程、几个机器。

模式名称 各模块占用JVM进程数 各个模块运行在几个机器数上
本地模式 1个 1个
伪分布式模式 N个 1个
完全分布式模式 N个 N个

Hadoop伪分布式模式安装

1，创建一个名字为hadoop的普通用户

[root@hadoop01 ~]# useradd hadoop
[root@hadoop01 ~]# passwd hadoop
更改用户 hadoop 的密码 。
新的 密码：
重新输入新的 密码：
passwd： 所有的身份验证令牌已经成功更新。

2，给hadoop用户sudo权限

[root@hadoop01 ~]# cat /etc/sudoers
[root@hadoop01 ~]# cd /etc/sudoers.d
[root@hadoop01 sudoers.d]# ll
总用量 0
[root@hadoop01 sudoers.d]# vi sudo_hadoop
[root@hadoop01 sudoers.d]# ll
总用量 4
-rw-r--r--. 1 root root 31 9月  10 05:25 sudo_hadoop
[root@hadoop01 sudoers.d]# cat sudo_hadoop 
hadoop ALL=(root) NOPASSWD:ALL
[root@hadoop01 sudoers.d]# su - hadoop
[hadoop@hadoop01 ~]$ pwd
/home/hadoop
[hadoop@hadoop01 ~]$ mkdir /opt/modules
mkdir: 无法创建目录"/opt/modules": 权限不够
[hadoop@hadoop01 ~]$ sudo mkdir /opt/modules
[hadoop@hadoop01 ~]$ cd /opt/
[hadoop@hadoop01 opt]$ ll
总用量 195696
-rw-r--r--. 1 root root   9455895 7月  16 15:27 apache-tomcat-8.0.53.tar.gz
drwxr-xr-x. 8 uucp  143      4096 3月  29 12:55 jdk1.8
-rw-r--r--. 1 root root 190921804 7月  16 14:12 jdk-8u172-linux-x64.tar.gz
drwxr-xr-x. 2 root root      4096 9月  10 05:27 modules
drwxr-xr-x. 9 root root      4096 7月  29 09:11 tomcat
[hadoop@hadoop01 opt]$ sudo chown -R hadoop:hadoop /opt/modules
[hadoop@hadoop01 opt]$ ll
总用量 195696
-rw-r--r--. 1 root   root     9455895 7月  16 15:27 apache-tomcat-8.0.53.tar.gz
drwxr-xr-x. 8 uucp      143      4096 3月  29 12:55 jdk1.8
-rw-r--r--. 1 root   root   190921804 7月  16 14:12 jdk-8u172-linux-x64.tar.gz
drwxr-xr-x. 2 hadoop hadoop      4096 9月  10 05:27 modules
drwxr-xr-x. 9 root   root        4096 7月  29 09:11 tomcat
[hadoop@hadoop01 opt]$

3，解压Hadoop目录文件
复制hadoop-2.8.3.tar.gz到/opt/modules目录下。解压hadoop-2.8.3.tar.gz

[hadoop@hadoop01 modules]$ tar -zxvf hadoop-2.8.3.tar.gz
[hadoop@hadoop01 modules]$ ll
总用量 238744
drwxr-xr-x. 9 hadoop hadoop      4096 12月  5 2017 hadoop-2.8.3
-rw-r--r--. 1 root   root   244469481 3月  22 13:28 hadoop-2.8.3.tar.gz
[hadoop@hadoop01 modules]$ mv hadoop-2.8.3 hadoop
[hadoop@hadoop01 modules]$ ll
总用量 238744
drwxr-xr-x. 9 hadoop hadoop      4096 12月  5 2017 hadoop
-rw-r--r--. 1 root   root   244469481 3月  22 13:28 hadoop-2.8.3.tar.gz

4，配置Hadoop环境变量

[root@hadoop01 etc]# vim /etc/profile

追加配置：

export HADOOP_HOME="/opt/modules/hadoop"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
执行：source /etc/profile 使得配置生效

验证HADOOP_HOME参数：

[root@hadoop01 etc]# echo $HADOOP_HOME
/opt/modules/hadoop

5，配置 hadoop-env.sh、mapred-env.sh、yarn-env.sh文件的JAVA_HOME参数

[root@hadoop01 etc]# su - hadoop
[hadoop@hadoop01 ~]$ ll
总用量 0
[hadoop@hadoop01 ~]$ pwd
/home/hadoop
[hadoop@hadoop01 ~]$ cd /opt/modules/
[hadoop@hadoop01 modules]$ ll
总用量 238744
drwxr-xr-x. 9 hadoop hadoop      4096 12月  5 2017 hadoop
-rw-r--r--. 1 root   root   244469481 3月  22 13:28 hadoop-2.8.3.tar.gz
[hadoop@hadoop01 modules]$ cd ./hadoop/etc/hadoop/
[hadoop@hadoop01 hadoop]$ ll
总用量 156
-rw-r--r--. 1 hadoop hadoop  4942 12月  5 2017 capacity-scheduler.xml
-rw-r--r--. 1 hadoop hadoop  1335 12月  5 2017 configuration.xsl
-rw-r--r--. 1 hadoop hadoop   318 12月  5 2017 container-executor.cfg
-rw-r--r--. 1 hadoop hadoop   774 12月  5 2017 core-site.xml
-rw-r--r--. 1 hadoop hadoop  3804 12月  5 2017 hadoop-env.cmd
-rw-r--r--. 1 hadoop hadoop  4666 12月  5 2017 hadoop-env.sh
-rw-r--r--. 1 hadoop hadoop  2598 12月  5 2017 hadoop-metrics2.properties
-rw-r--r--. 1 hadoop hadoop  2490 12月  5 2017 hadoop-metrics.properties
-rw-r--r--. 1 hadoop hadoop  9683 12月  5 2017 hadoop-policy.xml
-rw-r--r--. 1 hadoop hadoop   775 12月  5 2017 hdfs-site.xml
-rw-r--r--. 1 hadoop hadoop  1449 12月  5 2017 httpfs-env.sh
-rw-r--r--. 1 hadoop hadoop  1657 12月  5 2017 httpfs-log4j.properties
-rw-r--r--. 1 hadoop hadoop    21 12月  5 2017 httpfs-signature.secret
-rw-r--r--. 1 hadoop hadoop   620 12月  5 2017 httpfs-site.xml
-rw-r--r--. 1 hadoop hadoop  3518 12月  5 2017 kms-acls.xml
-rw-r--r--. 1 hadoop hadoop  1611 12月  5 2017 kms-env.sh
-rw-r--r--. 1 hadoop hadoop  1788 12月  5 2017 kms-log4j.properties
-rw-r--r--. 1 hadoop hadoop  5546 12月  5 2017 kms-site.xml
-rw-r--r--. 1 hadoop hadoop 13661 12月  5 2017 log4j.properties
-rw-r--r--. 1 hadoop hadoop   951 12月  5 2017 mapred-env.cmd
-rw-r--r--. 1 hadoop hadoop  1383 12月  5 2017 mapred-env.sh
-rw-r--r--. 1 hadoop hadoop  4113 12月  5 2017 mapred-queues.xml.template
-rw-r--r--. 1 hadoop hadoop   758 12月  5 2017 mapred-site.xml.template
-rw-r--r--. 1 hadoop hadoop    10 12月  5 2017 slaves
-rw-r--r--. 1 hadoop hadoop  2316 12月  5 2017 ssl-client.xml.example
-rw-r--r--. 1 hadoop hadoop  2697 12月  5 2017 ssl-server.xml.example
-rw-r--r--. 1 hadoop hadoop  2250 12月  5 2017 yarn-env.cmd
-rw-r--r--. 1 hadoop hadoop  4567 12月  5 2017 yarn-env.sh
-rw-r--r--. 1 hadoop hadoop   690 12月  5 2017 yarn-site.xml
[hadoop@hadoop01 hadoop]$ 

[hadoop@hadoop01 hadoop]$ vim hadoop-env.sh
[hadoop@hadoop01 hadoop]$ vim mapred-env.sh
[hadoop@hadoop01 hadoop]$ vim yarn-env.sh

末尾处加上：export JAVA_HOME="/opt/jdk1.8"

6，查看主机名称

[hadoop@hadoop01 hadoop]$ hostname
hadoop01
[hadoop@hadoop01 hadoop]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[hadoop@hadoop01 hadoop]$ su
密码：
[root@hadoop01 hadoop]# vim /etc/hosts
[root@hadoop01 hadoop]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.216.134     hadoop01

配置<configuration>里面的内容：

[hadoop@hadoop01 hadoop]$ vim core-site.xml 
[hadoop@hadoop01 hadoop]$ cat core-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:8020</value>
</property>
<property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/data/tmp</value>
</property>
</configuration>
[hadoop@hadoop01 hadoop]$ sudo mkdir -p /opt/data/tmp
[hadoop@hadoop01 hadoop]$ sudo chown -R hadoop:hadoop /opt/data/

[hadoop@hadoop01 hadoop]$ vim hdfs-site.xml 
[hadoop@hadoop01 hadoop]$ cat hdfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
       <name>dfs.replication</name>
       <value>1</value>
</property>
</configuration>
[hadoop@hadoop01 hadoop]$

格式化hdfs：

[hadoop@hadoop01 hadoop]$ pwd
/opt/modules/hadoop
[hadoop@hadoop01 hadoop]$ ll
总用量 148
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 bin
drwxr-xr-x. 3 hadoop hadoop  4096 12月  5 2017 etc
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 include
drwxr-xr-x. 3 hadoop hadoop  4096 12月  5 2017 lib
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 libexec
-rw-r--r--. 1 hadoop hadoop 99253 12月  5 2017 LICENSE.txt
-rw-r--r--. 1 hadoop hadoop 15915 12月  5 2017 NOTICE.txt
-rw-r--r--. 1 hadoop hadoop  1366 12月  5 2017 README.txt
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 sbin
drwxr-xr-x. 4 hadoop hadoop  4096 12月  5 2017 share
[hadoop@hadoop01 hadoop]$ cd bin
[hadoop@hadoop01 bin]$ ll
总用量 604
-rwxr-xr-x. 1 hadoop hadoop 228224 12月  5 2017 container-executor
-rwxr-xr-x. 1 hadoop hadoop   6514 12月  5 2017 hadoop
-rwxr-xr-x. 1 hadoop hadoop   8786 12月  5 2017 hadoop.cmd
-rwxr-xr-x. 1 hadoop hadoop  12330 12月  5 2017 hdfs
-rwxr-xr-x. 1 hadoop hadoop   7639 12月  5 2017 hdfs.cmd
-rwxr-xr-x. 1 hadoop hadoop   6237 12月  5 2017 mapred
-rwxr-xr-x. 1 hadoop hadoop   6310 12月  5 2017 mapred.cmd
-rwxr-xr-x. 1 hadoop hadoop   1776 12月  5 2017 rcc
-rwxr-xr-x. 1 hadoop hadoop 291669 12月  5 2017 test-container-executor
-rwxr-xr-x. 1 hadoop hadoop  14416 12月  5 2017 yarn
-rwxr-xr-x. 1 hadoop hadoop  11386 12月  5 2017 yarn.cmd
[hadoop@hadoop01 bin]$ hdfs namenode -format

[hadoop@hadoop01 bin]$ cd /opt/data/tmp/
[hadoop@hadoop01 tmp]$ ll
总用量 4
drwxrwxr-x. 3 hadoop hadoop 4096 9月  10 08:27 dfs
[hadoop@hadoop01 tmp]$ cd dfs/
[hadoop@hadoop01 dfs]$ ll
总用量 4
drwxrwxr-x. 3 hadoop hadoop 4096 9月  10 08:27 name
[hadoop@hadoop01 dfs]$ cd name
[hadoop@hadoop01 name]$ ll
总用量 4
drwxrwxr-x. 2 hadoop hadoop 4096 9月  10 08:27 current
[hadoop@hadoop01 name]$ cd current/
[hadoop@hadoop01 current]$ ll
总用量 16
-rw-rw-r--. 1 hadoop hadoop 323 9月  10 08:27 fsimage_0000000000000000000
-rw-rw-r--. 1 hadoop hadoop  62 9月  10 08:27 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 hadoop hadoop   2 9月  10 08:27 seen_txid
-rw-rw-r--. 1 hadoop hadoop 220 9月  10 08:27 VERSION
[hadoop@hadoop01 current]$ cat VERSION 
#Mon Sep 10 08:27:55 CST 2018
namespaceID=1374927292
clusterID=CID-c4744fa9-a70c-4a18-95ea-b9900d3710a4
cTime=1536539275798
storageType=NAME_NODE
blockpoolID=BP-1852849786-192.168.216.134-1536539275798
layoutVersion=-63

[hadoop@hadoop01 hadoop]$ cd /opt/modules/hadoop/sbin/
[hadoop@hadoop01 sbin]$ ll
总用量 124
-rwxr-xr-x. 1 hadoop hadoop 2752 12月  5 2017 distribute-exclude.sh
-rwxr-xr-x. 1 hadoop hadoop 6465 12月  5 2017 hadoop-daemon.sh
-rwxr-xr-x. 1 hadoop hadoop 1360 12月  5 2017 hadoop-daemons.sh
-rwxr-xr-x. 1 hadoop hadoop 1640 12月  5 2017 hdfs-config.cmd
-rwxr-xr-x. 1 hadoop hadoop 1427 12月  5 2017 hdfs-config.sh
-rwxr-xr-x. 1 hadoop hadoop 2339 12月  5 2017 httpfs.sh
-rwxr-xr-x. 1 hadoop hadoop 3763 12月  5 2017 kms.sh
-rwxr-xr-x. 1 hadoop hadoop 4134 12月  5 2017 mr-jobhistory-daemon.sh
-rwxr-xr-x. 1 hadoop hadoop 1648 12月  5 2017 refresh-namenodes.sh
-rwxr-xr-x. 1 hadoop hadoop 2145 12月  5 2017 slaves.sh
-rwxr-xr-x. 1 hadoop hadoop 1779 12月  5 2017 start-all.cmd
-rwxr-xr-x. 1 hadoop hadoop 1471 12月  5 2017 start-all.sh
-rwxr-xr-x. 1 hadoop hadoop 1128 12月  5 2017 start-balancer.sh
-rwxr-xr-x. 1 hadoop hadoop 1401 12月  5 2017 start-dfs.cmd
-rwxr-xr-x. 1 hadoop hadoop 3734 12月  5 2017 start-dfs.sh
-rwxr-xr-x. 1 hadoop hadoop 1357 12月  5 2017 start-secure-dns.sh
-rwxr-xr-x. 1 hadoop hadoop 1571 12月  5 2017 start-yarn.cmd
-rwxr-xr-x. 1 hadoop hadoop 1347 12月  5 2017 start-yarn.sh
-rwxr-xr-x. 1 hadoop hadoop 1770 12月  5 2017 stop-all.cmd
-rwxr-xr-x. 1 hadoop hadoop 1462 12月  5 2017 stop-all.sh
-rwxr-xr-x. 1 hadoop hadoop 1179 12月  5 2017 stop-balancer.sh
-rwxr-xr-x. 1 hadoop hadoop 1455 12月  5 2017 stop-dfs.cmd
-rwxr-xr-x. 1 hadoop hadoop 3206 12月  5 2017 stop-dfs.sh
-rwxr-xr-x. 1 hadoop hadoop 1340 12月  5 2017 stop-secure-dns.sh
-rwxr-xr-x. 1 hadoop hadoop 1642 12月  5 2017 stop-yarn.cmd
-rwxr-xr-x. 1 hadoop hadoop 1340 12月  5 2017 stop-yarn.sh
-rwxr-xr-x. 1 hadoop hadoop 4295 12月  5 2017 yarn-daemon.sh
-rwxr-xr-x. 1 hadoop hadoop 1353 12月  5 2017 yarn-daemons.sh

[hadoop@hadoop01 sbin]$ hadoop-daemon.sh start namenode
starting namenode, logging to /opt/modules/hadoop/logs/hadoop-hadoop-namenode-hadoop01.out
[hadoop@hadoop01 sbin]$ hadoop-daemon.sh start datanode
starting datanode, logging to /opt/modules/hadoop/logs/hadoop-hadoop-datanode-hadoop01.out
[hadoop@hadoop01 sbin]$ hadoop-daemon.sh start secondarynamenode
starting secondarynamenode, logging to /opt/modules/hadoop/logs/hadoop-hadoop-secondarynamenode-hadoop01.out
[hadoop@hadoop01 sbin]$ jps
3092 Jps
2948 DataNode
3049 SecondaryNameNode
2860 NameNode

登录Hadoop网页：http://192.168.216.134:50070
7，HDFS上测试创建目录、上传、下载文件

[hadoop@hadoop01 bin]$ hdfs dfs -mkdir /folder1
18/09/10 08:55:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop01 bin]$ hdfs dfs -put hdfs.cmd /folder1
18/09/10 08:58:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop01 bin]$
image.png

为了可以在windows上下载，需要到C:\Windows\System32\drivers\etc下配置hosts


image.png 
[hadoop@hadoop01 hadoop]$ hdfs dfs -get /folder1/hdfs.cmd
18/09/10 09:08:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop01 hadoop]$ ll
总用量 160
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 bin
drwxr-xr-x. 3 hadoop hadoop  4096 12月  5 2017 etc
-rw-r--r--. 1 hadoop hadoop  7639 9月  10 09:08 hdfs.cmd
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 include
drwxr-xr-x. 3 hadoop hadoop  4096 12月  5 2017 lib
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 libexec
-rw-r--r--. 1 hadoop hadoop 99253 12月  5 2017 LICENSE.txt
drwxrwxr-x. 2 hadoop hadoop  4096 9月  10 08:46 logs
-rw-r--r--. 1 hadoop hadoop 15915 12月  5 2017 NOTICE.txt
-rw-r--r--. 1 hadoop hadoop  1366 12月  5 2017 README.txt
drwxr-xr-x. 2 hadoop hadoop  4096 12月  5 2017 sbin
drwxr-xr-x. 4 hadoop hadoop  4096 12月  5 2017 share
[hadoop@hadoop01 hadoop]$

配置yarn

[hadoop@hadoop01 hadoop]$ pwd
/opt/modules/hadoop/etc/hadoop
[hadoop@hadoop01 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@hadoop01 hadoop]$ ll
总用量 160
-rw-r--r--. 1 hadoop hadoop  4942 12月  5 2017 capacity-scheduler.xml
-rw-r--r--. 1 hadoop hadoop  1335 12月  5 2017 configuration.xsl
-rw-r--r--. 1 hadoop hadoop   318 12月  5 2017 container-executor.cfg
-rw-r--r--. 1 hadoop hadoop   945 9月  10 07:05 core-site.xml
-rw-r--r--. 1 hadoop hadoop  3804 12月  5 2017 hadoop-env.cmd
-rw-r--r--. 1 hadoop hadoop  4698 9月  10 06:32 hadoop-env.sh
-rw-r--r--. 1 hadoop hadoop  2598 12月  5 2017 hadoop-metrics2.properties
-rw-r--r--. 1 hadoop hadoop  2490 12月  5 2017 hadoop-metrics.properties
-rw-r--r--. 1 hadoop hadoop  9683 12月  5 2017 hadoop-policy.xml
-rw-r--r--. 1 hadoop hadoop   857 9月  10 08:23 hdfs-site.xml
-rw-r--r--. 1 hadoop hadoop  1449 12月  5 2017 httpfs-env.sh
-rw-r--r--. 1 hadoop hadoop  1657 12月  5 2017 httpfs-log4j.properties
-rw-r--r--. 1 hadoop hadoop    21 12月  5 2017 httpfs-signature.secret
-rw-r--r--. 1 hadoop hadoop   620 12月  5 2017 httpfs-site.xml
-rw-r--r--. 1 hadoop hadoop  3518 12月  5 2017 kms-acls.xml
-rw-r--r--. 1 hadoop hadoop  1611 12月  5 2017 kms-env.sh
-rw-r--r--. 1 hadoop hadoop  1788 12月  5 2017 kms-log4j.properties
-rw-r--r--. 1 hadoop hadoop  5546 12月  5 2017 kms-site.xml
-rw-r--r--. 1 hadoop hadoop 13661 12月  5 2017 log4j.properties
-rw-r--r--. 1 hadoop hadoop   951 12月  5 2017 mapred-env.cmd
-rw-r--r--. 1 hadoop hadoop  1414 9月  10 06:37 mapred-env.sh
-rw-r--r--. 1 hadoop hadoop  4113 12月  5 2017 mapred-queues.xml.template
-rw-r--r--. 1 hadoop hadoop   758 9月  10 09:13 mapred-site.xml
-rw-r--r--. 1 hadoop hadoop   758 12月  5 2017 mapred-site.xml.template
-rw-r--r--. 1 hadoop hadoop    10 12月  5 2017 slaves
-rw-r--r--. 1 hadoop hadoop  2316 12月  5 2017 ssl-client.xml.example
-rw-r--r--. 1 hadoop hadoop  2697 12月  5 2017 ssl-server.xml.example
-rw-r--r--. 1 hadoop hadoop  2250 12月  5 2017 yarn-env.cmd
-rw-r--r--. 1 hadoop hadoop  4598 9月  10 06:50 yarn-env.sh
-rw-r--r--. 1 hadoop hadoop   690 12月  5 2017 yarn-site.xml
[hadoop@hadoop01 hadoop]$ vim mapred-site.xml
[hadoop@hadoop01 hadoop]$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
[hadoop@hadoop01 hadoop]$ 

[hadoop@hadoop01 hadoop]$ vim yarn-site.xml 
[hadoop@hadoop01 hadoop]$ cat yarn-site.xml 
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop01</value>
 </property>

</configuration>

[hadoop@hadoop01 hadoop]$ yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/modules/hadoop/logs/yarn-hadoop-resourcemanager-hadoop01.out
[hadoop@hadoop01 hadoop]$ yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/modules/hadoop/logs/yarn-hadoop-nodemanager-hadoop01.out
[hadoop@hadoop01 hadoop]$ jps
2948 DataNode
4101 Jps
3989 NodeManager
3735 ResourceManager
3049 SecondaryNameNode
2860 NameNode

登录http://hadoop01:8088

测试dfs单词统计
[hadoop@hadoop01 modules]$ yarn jar /opt/modules/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.3.jar wordcount /folder1/word.txt /output
image.png
image.png

相关文章

网友评论

      本文标题：（二）Hadoop集群环境搭建(伪分布式)

      本文链接：https://www.haomeiwen.com/subject/ngbdgftx.html

      延伸阅读

      深度阅读

      • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

      栏目导航

      热点阅读

      关于我们|服务条款|联系我们|（二）Hadoop集群环境搭建(伪分布式)|投稿指南|网站地图|RSS订阅|排版工具|手机版

      提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

      Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

      备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

      本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

      所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！