Scenario 1: Flink Task Exception Impacting Monitoring Data Continuity

Scenario Description of Failure Symptoms

The operator task of the Flink task stops or the entire task is interrupted. At this point, the monitoring data breakpoints or interruptions may occur. The topic consumer group corresponding to Kafka has a data consumption backlog (Barad_Comm).

Impact Scope of the Failure

The monitoring data cannot be queried, and the alarm on the cloud product side cannot be triggered normally.
No impact on the platform's other cloud product services.

Failure Location and Analysis

Enter the supporting yarn deployment node to check whether the corresponding yarn cluster and Flink tasks are running normally.
The following are common operation instructions:

##yarn resource display
yarn top

##yarn instance display
yarn application --list
yarn application -list -appStates ALL

##yarn task execution details
yarn application -status <application ID>

##yarn rm status
yarn rmadmin -getAllServiceState

##yarn node display
yarn node -list

Scenario 1: If the basic yarn cluster environment is abnormal, you need to contact our supporting queue colleagues for handling.
Scenario 2: Use df -h to check the disk utilization of the yarn cluster. If the disk utilization of the yarn machine service exceeds 80%, it will start cleaning up the services in the cluster. (Due to the mixed deployment of machines and HDFS, there is TDSQL cold backup data on HDFS. For similar issues, you can clean up a small amount of corresponding cold backup files first and then start monitoring Flink tasks. Meanwhile, contact our supporting TDSQL queue colleagues to handle the retention time of cold backup files.)
Scenario 3: The cluster is normal. If the Flink task stops due to the connection of the monitoring component, handle it according to the emergency steps. Just restart the task.

Failure Emergency Response Steps

Steps to restart Flink:

1. Restart the oceanus-ctcw1 container (to prevent the connection exception of the component):

   tcloud-oceanus-ctcw1
   kubectl get po -A  |grep tcloud-oceanus-ctcw1 | awk '{print $2}' | xargs kubectl delete po -n tce 
   

2. Check the status of job tasks (5 represents stopping tasks, 3 represents starting, and 4 represents normal operation):

     mysql22001
     select Status from online_galileo.Job;
   

   If it is 4, execute stop-job.sh first. Do not execute run-job.sh until the Status field of the database is 5.
   If it is 5, execute run-job.sh.

3. Enter the tcloud-barad-skywalker container:

   cd /usr/local/services/barad-skywalker/job/  
   

4. Stop the Flink task:

   bash stop-job.sh  
   

5. Start the Flink task:

   bash run-job.sh
   

Failure Recovery Verification

Check the status of job tasks (5 represents stopping tasks, 3 represents starting, and 4 represents normal operation):
mysql22001
select Status from online_galileo.Job;

Wait for 2-3 minutes, then check whether the monitoring metrics of the tenant-side CVM have recovered to normal.