上日志
2020-12-27 11:22:25,370 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2020-12-27 11:23:33,963 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68483 ms ago, cancelled=false, msg=Connection refused row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=shyt-hadoop-4031.xx.com.cn,43297,1609038908142, seqNum=0
2020-12-27 11:23:54,091 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88613 ms ago, cancelled=false, msg=Connection refused row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=shyt-hadoop-4031.xx.com.cn,43297,1609038908142, seqNum=0
2020-12-27 11:24:14,123 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108645 ms ago, cancelled=false, msg=Connection refused row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=shyt-hadoop-4031.xx.com.cn,43297,1609038908142, seqNum=0
2020-12-27 11:24:34,389 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2020-12-27 11:30:53,158 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=378633 ms ago, cancelled=false, msg=java.io.IOException: Table Namespace Manager not fully initialized, try again later
at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2693)
at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2915)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1686)
at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:483)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:59846)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
2020-12-27 11:31:21,814 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: RECEIVED SIGNAL 15: SIGTERM
2020-12-27 11:31:21,885 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2020-12-27 11:33:18,695 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting ApplicationHistoryServer
STARTUP_MSG: user = ams
STARTUP_MSG: host = shyt-hadoop-4031.xx.com.cn/10.32.40.31
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.7.3.2.6.2.0-205
2020-12-27 11:33:19,803 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2020-12-27 11:34:28,254 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68390 ms ago, cancelled=false, msg=Connection refused row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=shyt-hadoop-4031.xx.com.cn,36147,1609039344999, seqNum=0
2020-12-27 11:34:48,379 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88516 ms ago, cancelled=false, msg=Connection refused row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=shyt-hadoop-4031.xx.com.cn,36147,1609039344999, seqNum=0
2020-12-27 11:35:08,526 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108663 ms ago, cancelled=false, msg=Connection refused row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=shyt-hadoop-4031.xx.com.cn,36147,1609039344999, seqNum=0
2020-12-27 11:35:28,831 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2020-12-27 11:36:51,988 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: RECEIVED SIGNAL 15: SIGTERM
2020-12-27 11:36:52,056 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
情况2
Problem binding to [0.0.0.0:60200] java.net.BindException: Address already in use
情况3
启动一段时间就死掉了
解决1
卸载重装metric服务,没用
metric 自动生成hbase元数据问题和hfile，让hbase master进行接管
经排查，怀疑是 Ambari Metrics Service崩溃所致，修复方法如下：
1. 在 Ambari 上关闭 Ambari Monitors 和 Collector；
2. 将故障节点的 /var/lib/ambari-metrics-collector 路径下的内容清空；
3. 在 Ambari 上选择 “Ambari Metrics” => “Config” => “Advanced hbase-site” 下获取 hbase.rootdir 和 hbase-tmp 的路径；
清空以下目录中的内容
/export/var/lib/ambari-metrics-collector/hbase
/var/lib/ambari-metrics-collector/hbase-tmp
hbase zkcli
rmr /ambari-metrics-cluster
4. 将 hbase-tmp 及 hbase.rootdir 路径下内容清空或移到其他路径下保存；
5. 在 Ambari 上重启Ambari Metrics Service；
6. 几分钟之后在 Ambari 上便可看到正常显示的指标了。
解决2
查端口到底是谁占用
netstat -anputl | grep 60200
查进程
ps -aux | grep 30646
重启该占用程序，更换端口即可
重启metric-controler
解决3
调整controller的内存大小就好了，往大调整4096m
image.png
网友评论