Kernel Problem Location and Solution
Failure Symptoms
Kernel-related failures may cause the machine to be unable to log in or abnormal restart.
Possible Reasons
Kernel hung_task
The hung task mechanism is implemented by the kernel thread khungtaskd, which monitors the process in the TASK_UNINTERRUPTIBLE status. If in the period of kernel.hung_task_timeout_secs (Default 120 seconds), it remains in D status, then the stack information of the hung task process will be printed.
If configure kernel.hung_task_panic=1 , it will trigger a kernel panic and restart the machine.
Kernel soft lockup
Soft lockup means that the CPU is occupied by kernel code and cannot perform other processes. The principle of detecting soft lockup is to assign a kernel thread [watchdog/x] which will be timed perform to each CPU. If the thread is within a certain period (the default is 2*kernel.watchdog_thresh , 3.10 kernel kernel.watchdog_thresh and the default value is 10 seconds) is not performed, indicating that a soft lockup has occurred.
If kernel.softlockup_panic=1 is configured, it will trigger a kernel panic and restart the machine.
Kernel panic
The kernel's abnormal crash causes the machine to restart abnormally. Common kernel panic scenarios are as follows:
- The kernel has a hung_task and kernel.hung_task_panic=1 is configured.
- The kernel has a soft lockup and kernel.softlockup_panic=1 is configured.
- The kernel bug is triggered.
Processing Procedures
The troubleshooting and processing steps for kernel-related problems are relatively complicated. It is recommended to submit a ticket for further location and processing.
Hard Disk Problem Location and Solution
Hard disk inode is full
Failure phenomenon : When creating a new file, the error message No space left on device is prompted, and the inode space usage is 100% by using the df -i command.
Possible causes : File system inode exhaustion.
Processing steps : Delete unnecessary files or scale out the hard disk capacity.
Hard disk space usage is full
Failure phenomenon : When creating a new file, the error message No space left on device is prompted, and the hard disk space usage is 100% by using the df -h command.
Possible causes : Hard disk space exhaustion.
Processing steps : Delete unnecessary files or scale out the hard disk capacity.
Hard disk read only
Failure phenomenon: The file system can only read files but cannot create new files.
Possible cause: The file system is damaged.
Processing steps:
- Create a snapshot to back up disk data. See Create a Snapshot for details.
- Perform the corresponding processing steps according to the hard disk type:
System disk
It is recommended to restart the instance directly.
Data disk- Perform the following command and check the file system type corresponding to the read-only disk.
lsblk -f - Perform the following command to uninstall the data disk.
umount - Corresponding to the file system type, perform the following command to repair:
- For ext3/ext4 file system, perform the following command:
fsck -y /dev/corresponding disk - For xfs file system, perform the following command:
xfs_repair /dev/corresponding disk
- For ext3/ext4 file system, perform the following command:
- Perform the following command and check the file system type corresponding to the read-only disk.
Hard disk %util high
Failure phenomenon : The instance is stuck, and logging in using SSH or VNC is slow or unresponsive.
Possible causes : High IO causes hard disk %util to reach 100%.
Processing steps : Check whether the high IO is reasonable, and evaluate whether to reduce IO read and write or replace the hard disk with a higher performance.