1. 借鉴
Hadoop本地开发，9000端口拒绝访问
netstat中Local Adress几种形式
linux netstat 的local address例子
Hadoop入门学习之（三）虚拟机9000端口拒绝访问
2. 开始
使用flume消费kafka的数据，并将原始数据压缩到hdfs上保存，但是flume始终连不上hadoop集群，所以有了上面的借鉴和下面的总结。借鉴部分真的是救了老命，多谢多谢。
缘起
先看下hadoop集群的core-site.xml的关键配置
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:9000</value>
</property>
我们再来看下启动flume的关键配置 kafka-flume-hdfs.conf
其中172.173.16.10是hadoop01的ip
a1.sinks.k2.type = hdfs
a1.sinks.k2.hdfs.path = hdfs://172.173.16.10:9000/origin_data/hotel/log/expert_no/%Y-%m-%d
a1.sinks.k2.hdfs.filePrefix = expert-
a1.sinks.k2.hdfs.round = true
a1.sinks.k2.hdfs.roundValue = 10
a1.sinks.k2.hdfs.roundUnit = second
然后启动flume就出现了下面的错误:
java.net.ConnectException: Call From 9da9e7084e9b/172.173.16.18 to hadoop01:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy10.create(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1652)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:94)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:249)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:232)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:668)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:665)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
... 33 more
好吧，查哪哪看着都没问题，然后只能根据它的提示：https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused，来一条一条看，我将它们贴在下面:
1. Check the hostname the client using is correct. If it's in a Hadoop configuration option: examine it carefully, try doing an ping by hand.
2. Check the IP address the client is trying to talk to for the hostname is correct.
3. Make sure the destination address in the exception isn't 0.0.0.0 -this means that you haven't actually configured the client with the real address for that service, and instead it is picking up the server-side property telling it to listen on every port for connections.
4. If the error message says the remote service is on "127.0.0.1" or "localhost" that means the configuration file is telling the client that the service is on the local server. If your client is trying to talk to a remote system, then your configuration is broken.
5. Check that there isn't an entry for your hostname mapped to 127.0.0.1 or 127.0.1.1 in `/etc/hosts` (Ubuntu is notorious for this).
6. Check the port the client is trying to talk to using matches that the server is offering a service on. The `netstat` command is useful there.
7. On the server, try a `telnet localhost <port>` to see if the port is open there.
8. On the client, try a `telnet <server> <port>` to see if the port is accessible remotely.
9. Try connecting to the server/port from a different machine, to see if it just the single client misbehaving.
10. If your client and the server are in different subdomains, it may be that the configuration of the service is only publishing the basic hostname, rather than the Fully Qualified Domain Name. The client in the different subdomain can be unintentionally attempt to bind to a host in the local subdomain —and failing.
11. If you are using a Hadoop-based product from a third party, -please use the support channels provided by the vendor.
12. Please do not file bug reports related to your problem, as they will be closed as [Invalid](http://wiki.apache.org/hadoop/InvalidJiraIssues)
在看到第7条时，准备在hadoop01机器上试一下，telnet localhost 9000，然而
bash: telnet: command not found
yum下载telnet[yum install telnet]，然后再试，然而
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
看到这里就发现了，9000端口没有暴露出来吗？
先使用netstat -tpnl查看的端口开放情况
bash: netstat: command not found
好吧，心态崩了，yum install netstat
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
No package netstat available.
Error: Nothing to do
没有netstat嘛？
我们search一下yum search netstat
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
============================= Matched: netstat ==================================
dstat.noarch : Versatile resource statistics tool
net-snmp.x86_64 : A collection of SNMP protocol tools and libraries
net-tools.x86_64 : Basic networking tools
安装net-tools.x86_64yum install net-tools.x86_64,然后再看下端口占用netstat -tpnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 4955/java
tcp 0 0 172.173.16.10:50020 0.0.0.0:* LISTEN 4955/java
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 4815/java
tcp 0 0 127.0.0.1:45865 0.0.0.0:* LISTEN 4955/java
tcp 0 0 127.0.0.11:39471 0.0.0.0:* LISTEN -
tcp 0 0 172.173.16.10:50070 0.0.0.0:* LISTEN 4815/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 45/sshd
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 4955/java
tcp6 0 0 :::22 :::* LISTEN 45/sshd
可以看到9000只允许本机访问
我们看下/etc/hosts文件内容
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
解决
编辑hadoop01的/etc/hosts文件
我们将hosts文件编辑如下，增加一行，
使hadoop01指向当前机器的IP即可
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.173.16.10 hadoop01
[选自借鉴]
精髓在于core-site中的fs.defaultFS配置hdfs://localhos:9000t改为你的别名[hdfs://hadoop01:9000]，再通过hosts文件找到对应的hadoop01的ip
思考：fs.defaultFS的配置直接改为ip不知道是否也可行？
留给看官自己试试吧
然后重启hadoop集群即可
我们看一下端口占用netstat -tpnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 4955/java
tcp 0 0 172.173.16.10:50020 0.0.0.0:* LISTEN 4955/java
tcp 0 0 172.173.16.10:9000 0.0.0.0:* LISTEN 4815/java
tcp 0 0 127.0.0.1:45865 0.0.0.0:* LISTEN 4955/java
tcp 0 0 127.0.0.11:39471 0.0.0.0:* LISTEN -
tcp 0 0 172.173.16.10:50070 0.0.0.0:* LISTEN 4815/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 45/sshd
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 4955/java
tcp6 0 0 :::22 :::* LISTEN 45/sshd
可以看到9000可以通过IP访问了，然后telnet测试一下
telnet hadoop01 9000
Trying 172.173.16.10...
Connected to hadoop01.
Escape character is '^]'.
网友评论