elasticsearch半数以上master节点丢失, 如何恢

起因: 本地搭建了集群, 想着扩充节点数目, 就复制了安装目录, 改了nodename, 以及port等, 删除data目录. 但是操作失误, 最终还是导致脑裂了, 三个master节点, 其中有一个data被删除了, 另外两个竟然分别属于不同的集群, 无法选主...

这种情况下, 虽然 data都在, 都存活, 但是集群已经不能提供管理服务了. 而就是删除了所有的master节点, 重建一个master集群, 也不能和原先的data节点们组成集群. 这样元数据就丢失了...

参考:
类似场景:
https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/2
https://discuss.elastic.co/t/recover-data-after-the-lost-of-master-7-1-1/187775/5
工具说明链接:
tool-1 https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html#node-tool-detach-cluster

半数以上master节点丢失(确定无法恢复)

from tool-1

If there is at least one remaining master-eligible node, but it is not possible to restart a majority of them, then the elasticsearch-node unsafe-bootstrap command will unsafely override the cluster’s voting configuration as if performing another cluster bootstrapping process. The target node can then form a new cluster on its own by using the cluster metadata held locally on the target node.

如果集群中还剩半数以下 的master-eligible, 无论它的元数据是否完整, 是否最新. 我们都可以基于它来恢复之前的集群(可能有元数据丢失, 落后等, 所以是unsafe), 不过在执行的时候, 还是可以看下, 哪个剩余节点的元数据更新, 通过term和version判断. 这两个值在执行工具命令的时候, 会看到(见下图unsafe-bootstrap)

步骤

1. Make sure you have really lost access to at least half of the master-eligible nodes in the cluster, and they cannot be repaired or recovered by moving their data paths to healthy hardware.
2. Stop all remaining nodes.
3. Choose one of the remaining master-eligible nodes to become the new elected master as described above.
4. On this node, run the elasticsearch-node unsafe-bootstrap command as shown below. Verify that the tool reported Master node was successfully bootstrapped.
5. Start this node and verify that it is elected as the master node.
6. Run the elasticsearch-node detach-cluster tool, described below, on every other node in the cluster.
7. Start all other nodes and verify that each one joins the cluster.
8. Investigate the data in the cluster to discover if any was lost during this process.

注意事项 :

1. 关闭 security 功能. (如果开启了, 最好先关闭security插件功能
   1. 因为新启动的master节点, 没有数据节点(如果只配置了一个master的角色), 那security插件的初始化无法完成, 各类接口不好调用
   2. 如果给新启动的mster节点, 配置了master and data, 则security插件会初始化成功. 会插入index, 但是这个index会和原来的data节点上保存的冲突. 不知道怎么解.
2. 能成功, 不代表没有数据丢失.
3. 在unsafe-bootstrap的节点, 必须是关闭的
4. 所有其他或者的master-eligible节点(如果有) 必须是关闭的. all other master-eligible nodes are also stopped while this tool is running, but the tool does not check this.

unsafe-bootstrap.png detach.png

所有master节点丢失(确定无法恢复)

而, 如果所有的master节点都丢了, 且确定恢复不了了. 那只能把原来的data节点们分离出来, 加入到一个新的空集群, 并且fix dangling index.

步骤 和之前类似, 不过是启动一个新集群.

1. Make sure you have really lost access to every one of the master-eligible nodes in the cluster, and they cannot be repaired or recovered by moving their data paths to healthy hardware.
2. Start a new cluster and verify that it is healthy. This cluster may comprise one or more brand-new master-eligible nodes, or may be an unsafely-bootstrapped cluster formed as described above.
3. Stop all remaining data nodes.
4. On each data node, run the elasticsearch-node detach-cluster tool as shown below. Verify that the tool reported Node was successfully detached from the cluster.
5. If necessary, configure each data node to discover the new cluster.
6. Start each data node and verify that it has joined the new cluster.
7. Wait for all recoveries to have completed, and investigate the data in the cluster to discover if any was lost during this process. Use the Dangling indices API to list, import or delete any dangling indices.

注意事项
与前面略同.

新集群无法启动排查

无法启动, 可能是因为新集群不够新. (虽然它可能是从旧的master-eligible节点的元数据而来)

1. 关闭 security 安全插件 , 最好, 方便排查
2. 有些来自旧集群的persistence设置, 导致无法启动
   1. 使用elasticsearch-node remove-settings
   2. 使用 elasticsearch-node remove-custom