1/运行mr程序出错
connecting to resoucemanager
retrying ....
retrying .....
原因是没有启动yarn或者启动失败，正常启动yarn后，又报下面的错误：
System times on machines may be out of sync,check system time and time zones
原因是各个节点的时间没有统一，使用NTP统一时间，或是使用secureCRT自带的
send chat to all sessions菜单统一给节点执行sudo date -s 2018-03-20 10:53
2/初始化工作目录结构
hdfs namenode -format 只是初始化了namenode的工作目录
而datanode的工作目录是在datanode启动(执行startdfs.sh命令)后自己初始化的
使用tree命令查看目录结构（如果没有这个命令，要先安装sudo yum install -y tree）
hdpdata 树结构.png
3/datanode不被namenode识别的问题
namenode在format初始化的时候会形成两个标识：
blockPoolId：
clusterId：
新的datanode加入时，会获取这两个标识作为自己工作目录中的标识。
一旦namenode重新format后，namenode的身份标识已变，而datanode如果依然
持有原来的id，就不会被namenode识别。
namenode.png
datanode.png
4/datanode下线后多久看到效果
datanode不是一下线就会被namenode认定为下线的，有一个超时时间
5/关于副本数量的问题
副本数由客户端的参数dfs.replication决定（优先级： API conf.set > 自定义配置文件 > jar包中的hdfs-default.xml）
jar包中的hdfs-default.xml
6/往集群提交数据时，hdfs dfs -put employee.txt /person，关键字是Bad connect ack with firstBadLink as 192.168.92.112:50010
18/04/17 23:06:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/17 23:06:52 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 192.168.92.112:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1401)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1302)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
18/04/17 23:06:52 INFO hdfs.DFSClient: Abandoning BP-2054255948-192.168.92.111-1523928758660:blk_1073741825_1001
18/04/17 23:06:52 INFO hdfs.DFSClient: Excluding datanode 192.168.92.112:50010
18/04/17 23:06:52 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 192.168.92.113:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1401)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1302)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
18/04/17 23:06:52 INFO hdfs.DFSClient: Abandoning BP-2054255948-192.168.92.111-1523928758660:blk_1073741826_1002
18/04/17 23:06:52 INFO hdfs.DFSClient: Excluding datanode 192.168.92.113:50010
原因是集群中其他节点的防火墙没有关掉：
service iptables status
service iptables status
chkconfig iptables off
7/执行MR程序报错：
java.lang.UnsupportedClassVersionError: com/feiyue/bigdata/hdfs/HdfsClientDemo : Unsupported major.minor version 52.0
修改方法：
image.png
8/执行MR程序报错：
Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=dummy, access=WRITE, inode="/":hadoop:supergroup:drwxr-xr-x
设置：-DHADOOP_USER_NAME=hadoop
image.png
9/下面代码
private Configuration conf = null;
private FileSystem fileSystem = null;
@Before
public void init() throws IOException, InterruptedException, URISyntaxException {
conf = new Configuration();
//fileSystem = FileSystem.get(new URI("hdfs://hadoop:8020"), conf, "hadoop");
fileSystem = FileSystem.get(conf);
}
@Test
public void testUpload() throws IllegalArgumentException, IOException, InterruptedException {
//Thread.sleep(5000);
fileSystem.copyFromLocalFile(new Path("D:\\BaiduNetdiskDownload\\远程调试笔记.txt"), new Path("/access.log.copy2"));
fileSystem.close();
}
本地文件并不能上传到HDFS上面，
原因：
fileSystem = FileSystem.get(conf);得到的是org.apache.hadoop.fs.LocalFileSystem@44a5b703
//可以直接传入uri和用户身份
fs = FileSystem.get(new URI("hdfs://master:9000"),conf,"hadoop"); //最后一个参数为用户名
10/报NullException
在我的电脑-->系统变量-->中设置HADOOP_HOME
将对应Windows平台的编译好的HADOOP_HOME/bin， lib/native下面文件拷贝到HADOOP_HOME对应的文件夹里。
11/报错没有相应的Mapper：
java.lang.Exception: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.feiyue.bigdata.mr.flowsum.FlowCount$FlowCountMapper.<init>()
NoSuchMethodException.png
类改为private static class
12/在windows10中测试cacheFile过程中报错：
[LocalDistributedCacheManager Downloader #0] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:dummy (auth:SIMPLE) from:org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
[main] INFO org.apache.hadoop.mapred.LocalDistributedCacheManager - Creating symlink: \tmp\hadoop-dummy\mapred\local\1526633052317\pdts.txt <- D:\bigdata_worspace\hadoop-shizhan/pdts.txt
[main] WARN org.apache.hadoop.fs.FileUtil - Command 'D:\apps\hadoop-2.8.3\bin\winutils.exe symlink D:\bigdata_worspace\hadoop-shizhan\pdts.txt \tmp\hadoop-dummy\mapred\local\1526633052317\pdts.txt' failed 1 with: CreateSymbolicLink error (1314): ???????????
[main] WARN org.apache.hadoop.mapred.LocalDistributedCacheManager - Failed to create symlink: \tmp\hadoop-dummy\mapred\local\1526633052317\pdts.txt <- D:\bigdata_worspace\hadoop-shizhan/pdts.txt
[main] INFO org.apache.hadoop.mapred.LocalDistributedCacheManager - Localized file:/D:/data/mapjoincache/pdts.txt as file:/tmp/hadoop-dummy/mapred/local/1526633052317/pdts.txt
http://www.hainiubl.com/topics/68
13/本地执行下面的程序，报错：
Wrong FS: hdfs:/, expected: file:///
// conf.set("fs.default.name", "hdfs://hadoop:8020");
FileSystem fs = FileSystem.get(conf);
Path outPath = new Path(args[1]);
if (fs.exists(outPath)) {
fs.delete(outPath, true);
}
解决方法：
http://www.cnblogs.com/bingofworld/archive/2013/06/09/3129299.html
网友评论