Scenario Description of Failure Symptoms
ES write exception. At this point, the monitoring data breakpoints or interruptions occur, and the topic consumer group corresponding to Kafka has a consumption backlog (FlinkToWriter_Barad_Comm).
Impact Scope of the Failure
The monitoring data cannot be queried, and the alarm on the cloud product side cannot be triggered normally.
No impact on the platform's other cloud product services.
Failure Location and Analysis
Check the container logs written to ES:
tcloud-barad-metric-writer
Enter the container and run tail -1000f /usr/local/services/metric-writer/log/metric-writer.log | grep bulk.
Observe here whether the exception content returned in the log indicates that the ES cluster is under great pressure or the ES service prohibits writing (read-only).
Failure Emergency Response Steps
Scenario 1: If the ES cluster is under great pressure, you can appropriately clean up part of the monitoring data and then contact our supporting ES queue colleagues for ES cluster scale-out processing.
Scenario 2: If ES is read-only, you can appropriately clean up part of the monitoring data to recover the writable status, and then contact our supporting ES queue colleagues for ES cluster scale-out processing.
Common checks and cleanup operations for ES:
##Sort by index size in the current cluster.
curl -s -H "Content-Type: application/json" -u $es_user:$es_pass $es_ip:$es_port/_cat/indices?v\&s=store.size:desc | more
##Selectively clean up by index size.
curl -s -XDELETE -H "Content-Type: application/json" -u $es_user:$es_pass $es_ip:$es_port/xxx
##The cluster recovers to the writable status.
curl -X PUT -H "Content-Type: application/json" -u $es_user:$es_pass $es_ip:$es_port/_cluster/settings?pretty -d '{"persistent":{"cluster.blocks.read_only_allow_delete":null}}'
curl -X PUT -H "Content-Type: application/json" -u $es_user:$es_pass $es_ip:$es_port/_cluster/settings?pretty -d '{"index.blocks.read_only_allow_delete": false}'
Failure Recovery Verification
Check the container logs written to ES:
tcloud-barad-metric-writer
Enter the container and run tail -1000f /usr/local/services/metric-writer/log/metric-writer.log | grep bulk.
Check whether the newly written log reports an error.
Wait for 2-3 minutes, then check whether the monitoring metrics of the tenant-side CVM have recovered to normal.