Scenarios
Most components of TCenter run in containers, and most of these containers have their own health detection mechanisms. Therefore, it is important to first confirm whether the containers are running properly.
Prerequisites
You can log in to the Kubernetes cluster nodes and execute kubectl commands.
Operation Steps
- Log in to the Kubernetes cluster node and run the following command to check if the service status is normal.
# Check if the pod is started normally and if the pod status is normal.
kubectl get pod -n tce | grep cloud-tcenter
Exception Handling
If the container's runtime status is abnormal, you can confirm the reason for the abnormality using the following method.
# Use tcloud-tcenter-console-web-xx as an example.
kubectl describe pod -n tce tcloud-tcenter-console-web-xxx | grep -A 5 -E 'Last State:|Events:'
Focus on the Last State.Reason and Exit Code. If the container is abnormal due to resource limitations, such as an out-of-memory (OOM) issue, oomkilled is displayed, and the exit code is 137. If the status in the figure is not 137, it is usually caused by the failure of the service health detection mechanism in the cloud platform scenario. Based on the events section, it can be confirmed that the root cause is a liveness detection failure.
# Check container resource limits and health check settings.
kubectl describe pod -n tce tcloud-tcenter-console-web-xxx | grep -A 4 -E 'Limits|Liveness|Readiness'
(1) If the container is affected by resource limitations, you can temporarily solve the issue by adjusting the component's specifications. For a long-term solution, you can submit a ticket to the relevant technology stack to check whether the resource usage is unreasonable or if the default settings are unreasonable. The typical processing rocedures are as follows:
# Modify the limits part to adjust the corresponding rule information (using tcloud-tcenter-console-web-xx as an example).
# For regular Kubernetes components, modify the deploy, sts, ds, and other custom resources by default. In the cloud platform scenario, applications use the open application model (OAM), and the upper-level controllers for regular services are all comp.
kubectl edit comp -n tce tcloud-tcenter-console-web
# Confirm that the container starts normally.
kubectl get pod -n tce | grep tcloud-tcenter-console-web
# Confirm the historical resource usage of components. Log in to the operation side, choose Monitoring System (CWhistle) > Cloud Product Metric > TCS/Pod, and select pod_cpu_limit_usage/pod_memory_nocache_limit_usage for Monitoring Metric, global for Cluster, tce for Namespace, and tcloud-tcenter-console-web for Container to view the information.
(2) If the container cannot start due to a health check issue, you need to view the service logs to confirm the reason. A typical approach is as follows (After preliminary troubleshooting, you can submit a ticket to the relevant technology stack for further support):
kubectl get pod -n tce | grep tcloud-tcenter-console-web
# Enter the container using the following command when the pod is in a 0/1 Running status.
kubectl exec -it -n tce tcloud-tcenter-console-web-xxx bash
tail -n 100 /data/log/console/console.server.log
# If the container is always in the CrashLoopBackOff status, try running the kubectl logs command to view the logs. (The service should support printing the logs to stdout.)
kubectl logs -n tce tcloud-tcenter-console-web-xxx