logstash报错：

[2024-04-03T05:38:33,694][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({“type”=>“cluster_block_exception”, “reason”=>“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”})

es报错：

[2024-04-03T05:36:54,995][WARN ][o.e.c.r.a.DiskThresholdMonitor] [NyCgwKD] flood stage disk watermark [95%] exceeded on [NyCgwKDuTtKWM61BZPdlVA][NyCgwKD][/usr/share/elasticsearch/data/nodes/0] free: 2.3gb[2.3%], all indices on this node will be marked read-only
[2024-04-03T05:37:24,996][WARN ][o.e.c.r.a.DiskThresholdMonitor] [NyCgwKD] flood stage disk watermark [95%] exceeded on [NyCgwKDuTtKWM61BZPdlVA][NyCgwKD][/usr/share/elasticsearch/data/nodes/0] free: 2.3gb[2.3%], all indices on this node will be marked read-only
[2024-04-03T05:37:54,997][WARN ][o.e.c.r.a.DiskThresholdMonitor] [NyCgwKD] flood stage disk watermark [95%] exceeded on [NyCgwKDuTtKWM61BZPdlVA][NyCgwKD][/usr/share/elasticsearch/data/nodes/0] free: 2.3gb[2.3%], all indices on this node will be marked read-only

解决方法：

查看磁盘空间：

df -h

通过du -sh /data/*看是谁占用的磁盘空间，能清理择清理。

清理后重启elk服务：

停止：
docker stop kibana
docker stop logstash
docker stop elasticsearch

启动：
docker start elasticsearch
docker start logstash
docker start kibana

此时问题还没有完全解决！！！

存储一旦超过95％的磁盘中的节点上分配了一个或多个分片的任何索引，该索引将被强制进入只读模式

解决办法

第一种办法：在kibana开发控制台执行下面语句：

PUT _settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": null
    }
  }
}

第二种办法：在es节点服务器上执行：

curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

此时才算彻底解决！！！