Navigate to Inspection Platform > Inspection Item Management, where inspection items are already built-in for cloud products, and you can use these inspection items to check the status of cloud products.
1 Inspection Item List
This module summarizes the list of inspection items on the current platform, allowing you to enable/disable, edit, add, and delete inspection items. By using the dropdown list to select the business tree path or inspection item group, you can filter the inspection items. Additionally, you can search for inspection items using keywords such as the inspection item description or inspection item group.
2 View Inspection Item Detailed Information
In the inspection item list, click the Inspection Item ID you want to view to enter the inspection item details page. The page displays inspection item information, including basic information, parameters, knowledge base, and YAML.
3 Edit Inspection Items
On the inspection item management page, click Edit to perform this operation.
4 Enable/Disable Inspection Items
Enabling: Click the button switch in the disabled status to enable an inspection item that has been previously disabled. Once enabled, the inspection item will be executed during the next run of associated inspection tasks it has been added to.
Disabling: Click the button switch in the enabled status to disable an inspection task that has been previously enabled. Once disabled, the inspection item will not be executed during the next run of associated inspection tasks it has been added to, and related abnormal items will be automatically cleared.
5 Add Custom Inspection Items
Automated management of inspection items is supported. Below is the guide for creating a custom inspection item.
Starting Adding Custom Inspection Items
Navigate to Inspection Platform > Inspection Item Management, and click Create Inspection Item.
The process for writing inspection items is as follows:
- Inspection item product: Select the inspection target for the current inspection item.
- Inspection item group: Select the group name for the inspection item.
- Inspection item level: Select the inspection item level.
- Category: Select the category (the default is Daily Ops).
- Configuring the YAML File
- Find the appropriate inspection plugin and inspection function based on the inspection method.
i. For remote command execution, typically use the system plugin's system_exec_command function.
ii. For SQL execution, typically use the MySQL plugin's check_field_value function.
iii. ...... - Declare parameters for the inspection function.
- Declare the description information and UUID (try to use meaningful English names for the UUID).
- Find the appropriate inspection plugin and inspection function based on the inspection method.
- Enable inspection items and add them to the inspection task.
Configure the YAML file.
(1) Basic format
The YAML file contains the following fields:
host_type: host # Target host type. It can be set to host (for logging in to a host for inspections) or container (for logging in to a container for inspections). When the current path is k8s-master-ip or tianxun, host_type should be set to container.
host_mode: target_all # Selection mode for target hosts. It supports local (execute locally), target_any (select any host in the workload path of the current inspection item for execution), and target_all (select all hosts in the workload path of the current inspection item for execution).
func: MysqlExec # Name of the function referenced by the inspection item.
stack_name: checks-base # Binary name of the plugin of the inspection function func. If func is a general function, this field can be omitted or filled in as checks-base. If it is a technology stack-specific function, the binary name of the technology stack should be filled in, such as tce-compute.
uuid: tx_cvm_podcheck_1 # (Required) UUID of the inspection item.
desc: Inspection item description # (Required) Description of the inspection item.
args: # Inspection item parameter list ("key: value" format). If func does not require any parameters, args can be omitted.
database: yhimage
sql: show tables;
# btree_resources: (Optional) Declaration for reference of the workload tree resources.
# - type: host # Type of the workload tree resources.
# path: TCE.TCENTER.K8S.Node # Path of the workload tree.
# scope: region # Resource acquisition scope. It supports the default value or undeclared (for AZ level), region (for region level), and global (for global level).
(2) Reference to workload tree resources (btree_resources)
host_type: host
host_mode: local
func: ExecuteShResultNum
btree_resources: # Here is the declaration for the host resources within the current region in the path TCE.TCENTER.K8S.Node for the workload tree.
- type: host
path: TCE.TCENTER.K8S.Node
scope: region
args:
sh_list:
- sh_name: "tgwadm_whitelist_check.sh"
params: '{{ .btree_resources.host | map(.ip) | join(" ") }}' # Here, .btree_resources.host is used to reference the declared resource. The syntax is in jq format.
expect_equation: eq
expect_value: 1
(3) Common function formats
Execute the shell command and compare the output result with the expected number.
stack_name: checks-base func: SystemCmdResultNum args: cmd_list: cmd: "uptime | awk '{print $3/365}'" desc: "Whether the server has been running for more than 3 years." expect_equation: 'le' expect_value: 3Execute the shell command and compare the output result with the expected string.
stack_name: checks-base func: SystemCmdResultStr args: cmd_list: cmd: kubectl get secret -n cert-manager cert-manager-webhook-tls -o json | jq '.data["tls.crt"]' | xargs echo | base64 -d | openssl x509 -checkend 3888000 desc: "Check whether the certificate expiration time for cert-manager is 45 days later." expect_equation: 'eq' expect_value: 'Certificate will not expire'Check whether the pod is alive.
stack_name: checks-base func: PodActiveCheck args: pod_name: 'tcloud-argus2-adp-synchronizer'Check whether the pod is alive, and execute shell commands inside the pod.
stack_name: checks-base func: PodActiveCheck args: cmd_list: - cmd: ps aux | grep 'gw-controller-monitor' | grep -v grep | awk '{print $11}' |sort | wc -l desc: Check whether the ocloud-vpc-nfv-gw-controller gw-controller-monitor process is running normally. expect_equation: ge expect_value: 1 pod_name: ocloud-vpc-nfv-gw-controllerCopy the shell script and execute it, then check the execution results.
stack_name: checks-base func: ExecuteShResultNum args: sh_list: - sh_name: "tgwadm_whitelist_check.sh" params: '"ip1" "ip2"' expect_equation: eq expect_value: 1
(4) Complete function list
checks-base
CheckOsVersion ClusterAllocationCheck CpuFrequency CpuUsage CrontabGrep DiskCapacity DiskMount DiskUsage ElasticsearchCheck EtcdCheck ExecuteShResultNum ExecuteShResultNumProxy ExecuteShResultStr ExecuteShResultStrProxy GetK8sCert GetOcloudCert GetServerCert GetTcloudCert HttpApiResult HttpApiResultNum HttpApiResultStr IOUsage LoadAverage MemSwapUsage MemUsage MysqlCheck MysqlCheckDemo MysqlCheckFieldValue MysqlExec PartitionUsage PingCheck PodActiveCheck PodImageCheck PodLogCheck PodNodeCheck PodReplicaCheck PodRestartCheck PodUsageCheck PortCheck ProcessCheck PvcUsageCheck RedisConfigCheck RedisExecCheck ServiceActiveCheck SyncDate SysLogCheck SystemCmdResult SystemCmdResultNum SystemCmdResultNumProxy SystemCmdResultStr SystemCmdResultStrProxy SystemModeCheck ZookeeperCheck
(5) Examples
Example 1: tx_tcenter_check_cert-manager_expire.
uuid: tx_tcenter_check_cert-manager_expire
host_mode: target_all
host_type: host
stack_name: tce-base
func: SystemCmdResultStr
args:
cmd_list:
- cmd: kubectl get secret -n cert-manager cert-manager-webhook-tls -o json | jq '.data["tls.crt"]' | xargs echo | base64 -d | openssl x509 -checkend 3888000
desc: Check whether the certificate expiration time for cert-manager is 45 days later.
expect_equation: eq
expect_value: Certificate will not expire
desc: Check whether the certificate expiration time for cert-manager is 45 days later.
btree_resources: []
Take the above inspection item as an example:
- This inspection item's UUID is set to tx_tcenter_check_cert-manager_expire, and its desc is set to 'Check whether the certificate expiration time for cert-manager is 45 days later'. The UUID and desc will be displayed on the inspection list page.
- This inspection item's host_mode is set to target_all, indicating that this inspection item will ultimately be executed on all machines in the associated business tree path. If it were set to target_any, this inspection item would be executed on any machine in the associated business tree path.
- This inspection item's host_type is set to host, indicating that the target of this inspection item is a host. In addition to host, the inspection platform also supports a container type for host_type, indicating that the target is a container or special node.
- This inspection item's stack_name is set to tce-base, and its func is set to SystemCmdResultStr. This indicates that the implementation of the inspection is located in a binary file named tce-base, and the function being called is named SystemCmdResultStr.
- The subsequent args field contains parameters passed to the SystemCmdResultStr function. Similar to section 2.2, the SystemCmdResultStr function supports executing multiple shell commands, with each command specified using cmd. The expected output for these commands is 'Certificate will not expire'.
Example 2: tx_cbs_pod_processor
uuid: tx_cbs_pod_processor
host_mode: target_any
host_type: container
stack_name: tce-storage
func: PodActiveCheck
args:
cmd_list:
- cmd: df -h | grep '/' |tr -d '%' | awk '{if ($5>80)print}' | wc -l
desc: Check whether the number of mounted partitions whose pod container disk utilization is greater than 80% is 0.
expect_equation: eq
expect_value: 0
pod_name: ocloud-cbs-processor
desc: The pod ocloud-cbs-processor container status is running, and the disk utilization is less than 80%.
btree_resources: []
Here, we only list the parts that differ between this inspection item and the previous one:
- The current inspection item's host_mode is set to target_any, and its host_type is set to container. This indicates that the inspection needs to be executed only once and should run inside the container (pod).
- The current inspection item's func is set to PodActiveCheck, and its args is set to the parameters passed to PodActiveCheck. This function also supports the simultaneous declaration of multiple shell commands.
6. Deleting Inspection Items
Click delete to delete a custom inspection item. System inspection items cannot be deleted. If you do not need to use them, you can disable them.