Tenant Cloud Monitor and Operation Cloud Monitor Ops Management Guide

Emergency Response Plan

Last Updated At: 2026-08-14 08:08:00

  1. IDC power outage contingency plan
  • Failure impact:
    The IDC may experience a global power outage of servers handling Cloud Monitor service due to power supply, air conditioning, cabinet, or switch failures, resulting in service interruption.
  • Recovery steps:
    1. Use batch scripts or commands to confirm that all machines are powered on and network connectivity is recovered.
    2. Confirm whether the processes running on each machine have been started normally.
    3. Confirm that the storm, Kafka, and ES services have started, and observe the corresponding UI.
    4. The data reception service cluster is stateless. After the process is started, it can automatically recover.
    5. If the previous services have not been started, you need to confirm whether the ZK service is normal.
    6. Use the local ZK-data file to recover ZK, and then monitor the startup status of the above services.
    7. Monitor whether the Kafka data directory file is updated normally.
    8. Observe whether the storm-ui data is calculated normally.
  1. Kafka service overall cluster troubleshooting contingency plan
  • Pre-check:
    Check whether the Kafka status is normal.
    Enter the bin directory under the deployed ZK, and run zkCli.sh and ls/kafka/kafka/okers/ids.
    If the above command returns fewer than 3 items, it indicates a node exception. Execute get /kafka/kafka/brokers/ids/130618222416261120 to obtain the node 2.ip information (the ID number is taken from the above return results). Execute the returned ID in sequence and compare them with the IP addresses of Kafka deployment, and you will obtain the information about the abnormal Kafka node.
  • Failure impact:
    The Kafka service, as a cache node for monitoring data, is an important link in stream computing. Normally, it works in cluster mode. After the cluster crashes, data cannot be written to the cache, resulting in interruption of monitoring data and failing to display isd.barad data. Monitoring alarms cannot be sent normally, seriously affecting operations.
  • Recovery steps:
    1. Recover and start all Kafka servers as soon as possible, and start the corresponding processes.
    2. If the machine cannot start, you need to replace it with a new storage-type machine.
    3. The kafka=id of the new machine should be consistent with that of the old machine. The Kafka ID information can be obtained from ZK.
    4. After the abnormal Kafka process is recovered, it will automatically proceed to Kafka data recovery.
    5. Log in to the Kafka server and execute the kafka-topic-describe command to obtain the real-time status.