Shipping Monitoring Metrics to Kafka

Last Updated At: 2025-10-21 09:10:00

I. Feature Description

Cloud Monitor data can be forwarded to a specified Kafka, supporting:

  • Full data forwarding, such as forwarding all metric data of CVMs.
  • Partial forwarding, such as forwarding metric data of a specific CVM or all CVMs under a specific AppID, and forwarding specified metric data of specified instances.
    The above features are configured through forwarding rules.

II. Forwarding Configuration

Due to the lack of a frontend page, forwarding rules are configured via CLI-based operations, that is, by inserting forwarding rules into the database. The steps are as follows:

  1. Connect to the forwarding database named dispense.
  2. Configure forwarding rules.
-- Create a rule.
insert into t_rule (RuleName, Status) values ('forwarding rule name', 1);
Description:
RuleName: Rule name.
Status: Rule enabling status, 1: enabled; 0: disabled.

-- Configure the metrics that need to be forwarded.
insert into t_metric (RuleId, Metric, Namespace, Measurement, Periods) values (10, 'cpu_usage', 'qce/cvm', 'cvm_device', '60');
Note:
RuleId: The ID field after a rule is created via the t_rule above.
Metric: Metric name.
Namespace: The Namespace where metrics are located.
Measurement: The corresponding Cloud Monitor view. To obtain the view, see Note 1.
Periods: The time granularity of metrics. You can configure a type such as 60,300 to specify multiple time granularities. You can also configure "*" to support all granularities.
For how to determine the Namespace and Measurement for metrics and the supported time granularity, see Appendix 1.

-- Configure filter conditions (Optional. Configure this if you only want to forward data of a specific instance. There are 10 tables in total. You can obtain the information about which table ${n} resides on by performing a ruleId % 10 calculation.)
insert into t_condition_${n} (RuleId, Filters) values (10, '[{"Key":"vm_uuid","Values":["c7fc0b4c-0059-4318-bb0c-dc4a31a071da"],"Expression":"="},{"Key":"__appid__","Values":["251005942", "1"],"Expression":"in"}]');
Note:
RuleId: The ID field after a rule is created via the t_rule above.
Filters: Forwarding filter rules. It is a JSON array, and the relationship between multiple arrays is AND.

-- Configure the forwarding target Kafka.
insert into t_target (RuleId, Target, ProtoType) values (10, '{"brokers":"11.150.215.105:9092","topic":"Nowrite","user":"","password":""}', 1);
Note:
RuleId: The ID field after a rule is created via the t_rule above.
Target: Target Kafka address.
ProtoType: Message type forwarded to Kafka, 1: JSON; 2: Protobuf.

III. Message Protocol

The message protocol forwarded to Kafka is as follows:

// json
{
    "Namespace": "qce/cvm",
    "MetricName": "cpu_usage",
    "Statistic": "MAX",
    "Period": 60,
    "Dimensions": [{
        "Name": "appid",
        "Value": "1"
    }, {
        "Name": "projectid",
        "Value": "0"
    }, {
        "Name": "vm_uuid",
        "Value": "4f7639f1-f156-41b4-9072-f7e27af97d3b"
    }],
    "Timestamp": 1737107700000,
    "Value": 22.916666666666668
}

IV. Forwarding Table Structure Description

Table NameMeaningDescription
t_ruleForwarding a rule tableIncludes rule metadata such as rule ID, rule name, and configuration user.
`Id: 38230`
`RuleName: 1118-Phase 2-cvm-cpu // Forwarding rule name.`
`AppId: 1300260700 // User AppID for configuring rules, can be empty.`
`SubUin: 0 // User UIN for configuring rules, can be empty.`
`Status: 1 // Rule enabling status, 1: enabled; 0: disabled.`
`CreateTime: 2024-11-20 11:14:23`
`UpdateTime: 2024-11-20 11:14:23`
`IsAllMetric: 0 // Can be empty.`
t_metricForwarding a metric tableIncludes the metric information associated with the forwarding operation. Note that in a private network, ExtNamespace and ExtMetric are empty.
` Id: 3732457`
` RuleId: 38230`
` ExtMetric: CpuUtil // External metric name, can be empty.`
`ExtNamespace: QCE/TI_MODEL // External namespace, can be empty.`
` Measurement: model_svc // View name.`
` Periods: * // * indicates all granularities. If you only want to forward granularities 60 and 300, configure it as 60,300.`
` Metric: cpu_util // Internal metric name.`
` Namespace: qce/ti_model // Internal namespace.`
` CNMeaning: CPU utilization // Metric Chinese name, which can be empty.`
` ENMeaning: cpu_util // Metric English name, which can be empty.`
` CName: `
` Unit: % // Metric unit, which can be empty.`
` CreateTime: 2024-12-12 18:30:00`
` UpdateTime: 2024-12-12 18:30:00`

t_condition_${n}Forwarding a condition tableThere are 10 tables in total (0 to 9), and you can obtain the information about which table the forwarding condition resides on by performing a ruleId % 10 calculation.
Full forwarding can be configured in the private network by setting the condition table to be empty.
The Expression in Filters only supports = or in. For =, Values has only one value; for in, Values may have multiple values.
`Id: 12`
`RuleId: 38230`
`Filters: [{"Key":"vm_uuid","Values":["c7fc0b4c-0059-4318-bb0c-dc4a31a071da"],"Expression":"="},{"Key":"__appid__","Values":["251005942", "1"],"Expression":"in"}]`
`CreateTime: 2024-11-19 17:49:19`
`UpdateTime: 2024-11-19 17:49:19`


t_targetForwarding a target tableTargetType currently only supports Kafka, and ProtoType supports two formats: JSON (1) and PB (2).

` Id: 34539`
` RuleId: 38230`
`TargetType: Kafka`
` Target: {"brokers":"11.150.215.105:9092","topic":"Nowrite","user":"","password":""}`
` ProtoType: 1`
`CreateTime: 2024-11-19 16:48:31`
`UpdateTime: 2024-11-19 16:48:31`

V. Appendix 1

How to view metrics, Namespace, Measurement, and Period?
Execute kubectl get ingress -A | grep isd.barad on the master node.

Access the domain name through a browser.