elasticsearch常见问题及处理办法

作者: 雨中星辰0 | 来源:发表于2021-03-15 12:09 被阅读0次

elasticsearch常见问题及处理办法
常见问题处理办法
react-native常见问题及处理办法
外行学python（五异常）
谢永茂:分享注塑机油泵常见问题维修与处理办法
谢永茂:分享注塑机油泵常见问题维修与处理办法
Apache Doris FE 元数据常见故障处理
常见问题及处理
Elasticsearch搭建及配置常见问题
Elasticsearch 架构设计及说明

作者	时间	说明
雨中星辰	2021-03-15	整理

1. Elasticsearch索引只读（blocked by forbidden read-only allow delete）

错误详情

[2018-08-23T17:30:35,546][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:134) ~[?:?]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_144]
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_144]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_144]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_144]
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_144]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_144]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:135) ~[?:?]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:118) ~[?:?]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:571) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:380) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:375) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:908) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockException(TransportReplicationAction.java:826) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:814) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:712) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:169) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:125) ~[?:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:350) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:462) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:175) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:514) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:98) [elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
    at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:182) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:812) ~[elasticsearch-6.0.0.jar:6.0.0]
    ... 20 more

问题原因

一旦在存储超过95％的磁盘中的节点上分配了一个或多个分片的任何索引，该索引将被强制进入只读模式

解决方法

分别查看es集群的几个机器磁盘占用是否确实过高，如果过高，清理无用的文件，降低磁盘使用量。
在服务器命令行输入以下语句：

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
如果有配置用户名密码则：
curl -u elastic:elastic -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

2. not enough master nodes xxx,but needed [2]

错误详情：

org.elasticsearch.transport.ConnectTransportException: [node-3][10.35.6.60:9300] connect_exception
    at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:615) ~[elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:513) ~[elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:327) ~[elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:314) ~[elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: ?𺐿?杩𽐿𹐿: 10.35.6.60/10.35.6.60:9300
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
    ... 1 more
Caused by: java.net.ConnectException: ?𺐿?杩𽐿𹐿
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
    ... 1 more
[2021-03-15T11:57:14,014][WARN ][o.e.d.z.ZenDiscovery     ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{8MM0NPwAQ4O9d23w2D29VQ}{QnKzmzqxQjq484FPog-JHA}{10.35.6.58}{10.35.6.58:9300}, clusterStateVersion=528}]], but needed [2]), pinging again
[2021-03-15T11:57:16,816][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [node-1] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2021-03-15T11:57:17,014][WARN ][o.e.d.z.ZenDiscovery     ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{8MM0NPwAQ4O9d23w2D29VQ}{QnKzmzqxQjq484FPog-JHA}{10.35.6.58}{10.35.6.58:9300}, clusterStateVersion=528}]], but needed [2]), pinging again
[2021-03-15T11:57:20,015][WARN ][o.e.d.z.ZenDiscovery     ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{8MM0NPwAQ4O9d23w2D29VQ}{QnKzmzqxQjq484FPog-JHA}{10.35.6.58}{10.35.6.58:9300}, clusterStateVersion=528}]], but needed [2]), pinging again
[2021-03-15T11:57:20,604][WARN ][o.e.x.w.e.ExecutionService] [node-1] failed to store watch records
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:387) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:273) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:421) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:578) [elasticsearch-6.1.2.jar:6.1.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2.jar:6.1.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

错误原因：

在elasticsearch的配置文件中配置了最少节点数，当elasticsearch集群节点数少于该配置数时，集群状态对外不可用。

解决办法：

检查elasticsearch集群状态，重启故障的elasticsearch（推荐）
删除elasticsearch最少节点数配置（不推荐）
$ES_HOME/conf/elasticsearch.yml

discovery.zen.minimum_master_nodes:

3 max number of threads [1024] for user [work] likely too low, increase to at least [2048]

解决办法：添加以下配置

[root@EPRI-DCLOUD-ES04 dcloud]# cat /etc/security/limits.d/90-nproc.conf 
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
*          soft    nproc     2048
root       soft    nproc     unlimited

4 max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

#vi /etc/sysctl.conf 
增加改行配置：vm.max_map_count=262144  
保存退出后，执行：  
sysctl -p

5 max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]

vi /etc/security/limits.conf
添加：
*               soft    nproc           65536
*               hard    nproc           65536
*               soft    nofile          65536
*               hard    nofile          65536

elasticsearch常见问题及处理办法
作者时间说明雨中星辰2021-03-15整理 1. Elasticsearch索引只读（blocked by fo...
常见问题处理办法
1.关于监控黑屏（大面积断网）的处理首先检查信号线是否断开，若没有，则去11楼配电室，打开电箱，右下角找到“弱电...
react-native常见问题及处理办法
react-native 所有的文字显示必须用标签包裹
外行学python（五异常）
常见问题及解决办法 python支持中文的办法： #coding:utf-8 查看中文文档： http://www...
谢永茂:分享注塑机油泵常见问题维修与处理办法
谢永茂:分享注塑机油泵常见问题维修与处理办法大家好！我是谢永茂，今天给大家分享的是注塑机油泵常见问题维修与处理办...
谢永茂:分享注塑机油泵常见问题维修与处理办法
谢永茂:分享注塑机油泵常见问题维修与处理办法大家好！我是谢永茂，今天给大家分享的是注塑机油泵常见问题维修与处理办...
Apache Doris FE 元数据常见故障处理
该处理方法适用于 Apache Doris 0.14.7 及之后所有版本常见问题 FE （Follower）挂掉...
常见问题及处理
2018年9月22日星期六晴我是韶忆，日记星球417 号星宝宝，我正在参加日记星球第二十三期蜕变之旅，这是我...
Elasticsearch搭建及配置常见问题
Q: can not run elasticsearch as root? A: 因为elasticsearch...
Elasticsearch 架构设计及说明
Elasticsearch 架构设计及说明 Elasticsearch 架构层： Elasticsearch 五层...

elasticsearch常见问题及处理办法

1. Elasticsearch索引只读（blocked by forbidden read-only allow delete）

错误详情

问题原因

解决方法

2. not enough master nodes xxx,but needed [2]

错误详情：

错误原因：

解决办法：

3 max number of threads [1024] for user [work] likely too low, increase to at least [2048]

解决办法：添加以下配置

4 max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

5 max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]

相关文章

elasticsearch常见问题及处理办法

常见问题处理办法

react-native常见问题及处理办法

外行学python（五异常）

谢永茂:分享注塑机油泵常见问题维修与处理办法

谢永茂:分享注塑机油泵常见问题维修与处理办法

Apache Doris FE 元数据常见故障处理

常见问题及处理

Elasticsearch搭建及配置常见问题

Elasticsearch 架构设计及说明

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读