Shipping Monitoring Metrics to Kafka

Last Updated At: 2026-08-14 08:08:00

I. Feature Description

Cloud Monitor data can be forwarded to a specified Kafka, supporting:

  • Full data forwarding, such as forwarding all metric data of CVMs.
  • Partial forwarding. For example, metric data of a CVM instance or metric data of all CVM instances under a specific AppID is forwarded, or specified metric data of specified instances is forwarded.
    The above features are configured through forwarding rules.

    Note:

    The tcloud-barad-dispense application must be deployed in the environment.

II. Forwarding Configuration

Currently, forwarding rules need to be configured via CLI-based operations, that is, by inserting forwarding rules into the database. The steps are as follows:

  1. Connect to the forwarding database: Set the database name to dispense and the corresponding service_id to dbsql-barad-dispense, and run the following command on the master node:
dbsql_info barad-dispense         // This is barad-dispense, not dbsql-barad-dispense.

Note:

The connect cmd is the command to connect to the dispense database.

  1. Configure forwarding rules.
-- Create a rule.
insert into t_rule (RuleName, Status) values ('forwarding rule name', 1);
// Note:
// RuleName: rule name.
// Status: whether the rule is enabled. 1: enabled, 0: disabled.

-- Configure the metrics that need to be forwarded.
insert into t_metric (RuleId, Metric, Namespace, Measurement, Periods) values (10, 'cpu_usage', 'qce/cvm', 'cvm_device', '60');
// Note:
// RuleId: ID of the new rule specified by t_rule.
// Metric: metric name.
// Namespace: Namespace to which the metric belongs.
// Measurement: view of Cloud Monitor. To obtain the view, refer to Appendix 1.
// Periods: time granularity of the metric. Multiple granularities can be specified, such as (60, 300), or the asterisk (*) can be used to indicate all supported granularities.
// If you need to know the namespace to which the metrics belong, the measurement, as well as the supported time granularities, refer to the appendix.

-- Configure filter conditions (Optional. Configure this if you only want to forward data of a specific instance. There are 10 tables in total. You can obtain the information about which table ${n} resides on by performing a ruleId % 10 calculation.)
insert into t_condition_${n} (RuleId, Filters) values (10, '[{"Key":"vm_uuid","Values":["c7fc0b4c-0059-4318-bb0c-dc4a31a071da"],"Expression":"="},{"Key":"__appid__","Values":["251005942", "1"],"Expression":"in"}]');
// Note:
// RuleId: ID of the new rule specified by t_rule.
// Filters: forwarding rule filter, which is of a JSON array with an and relationship between rules.

-- Configure the forwarding target Kafka.
insert into t_target (RuleId, Target, ProtoType) values (10, '{"brokers":"11.150.215.105:9092","topic":"Nowrite","user":"","password":""}', 1);
// Note:
// RuleId: ID of the new rule specified by t_rule.
// Target: target Kafka address.
// ProtoType: message types forwarded to Kafka. 1: JSON, 2: Protobuf.

III. Message Protocol

The message protocol forwarded to Kafka is as follows:

// json
{
    "Namespace": "qce/cvm",
    "MetricName": "cpu_usage",
    "Statistic": "MAX",
    "Period": 60,
    "Dimensions": [{
        "Name": "appid",
        "Value": "1"
    }, {
        "Name": "projectid",
        "Value": "0"
    }, {
        "Name": "vm_uuid",
        "Value": "4f7639f1-f156-41b4-9072-f7e27af97d3b"
    }],
    "Timestamp": 1737107700000,
    "Value": 22.916666666666668
}

IV. Forwarding Table Structure Description

Table NameMeaningDescription
t_ruleForwarding a rule tableIncludes rule metadata such as rule ID, rule name, and configuration user.
`Id: 38230`
`RuleName: 1118-Phase 2-cvm-cpu // Forwarding rule name.`
`AppId: 1300260700 // User AppID for configuring rules, can be empty.`
`SubUin: 0 // User UIN for configuring rules, can be empty.`
`Status: 1 // Rule enabling status, 1: enabled; 0: disabled.`
`CreateTime: 2024-11-20 11:14:23`
`UpdateTime: 2024-11-20 11:14:23`
`IsAllMetric: 0 // Can be empty.`
t_metricForwarding a metric tableInformation about the metrics to be forwarded.
<strong>Note:</strong>
If forwarding is performed within a private network, `ExtNamespace` and `ExtMetric` are empty.

` Id: 3732457`
` RuleId: 38230`
` ExtMetric: CpuUtil // External metric name, can be empty.`
`ExtNamespace: QCE/TI_MODEL // External namespace, can be empty.`
` Measurement: model_svc // View name.`
` Periods: * // * indicates all granularities. If you only want to forward granularities 60 and 300, configure it as 60,300.`
` Metric: cpu_util // Internal metric name.`
` Namespace: qce/ti_model // Internal namespace.`
` ENMeaning: cpu_util // Metric English name, which can be empty.`
` CName: `
` Unit: % // Metric unit, which can be empty.`
` CreateTime: 2024-12-12 18:30:00`
` UpdateTime: 2024-12-12 18:30:00`
t_condition_${n}Forwarding a condition table
  • There are 10 tables (numbered 0 to 9). The specific table for forwarding conditions can be determined by ruleId % 10.
  • When the current table is empty, it indicates that unconditional forwarding is configured in the private network.
  • The Expression in Filters can only be set to `=` or `in`. When it is set to `=`, only a single value is contained in Values. When it is set to `in`, multiple values may be contained in Values.

`Id: 12`
`RuleId: 38230`
`Filters: [{"Key":"vm_uuid","Values":["c7fc0b4c-0059-4318-bb0c-dc4a31a071da"],"Expression":"="},{"Key":"__appid__","Values":["251005942", "1"],"Expression":"in"}]`
`CreateTime: 2024-11-19 17:49:19`
`UpdateTime: 2024-11-19 17:49:19`
t_targetForwarding a target table
  • TargetType must be set to `Kafka`.
  • Valid values of ProtoType:
  • 1: `JSON`
  • 2: `Protobuf`

` Id: 34539`
` RuleId: 38230`
`TargetType: Kafka`
` Target: {"brokers":"11.150.215.105:9092","topic":"Nowrite","user":"","password":""}`
` ProtoType: 1`
`CreateTime: 2024-11-19 16:48:31`
`UpdateTime: 2024-11-19 16:48:31`

5. Appendix: Viewing Metrics, Namespace, Measurement, and Period

  1. Run kubectl get ingress -A | grep isd.barad on the master node.
  2. Access the domain name through a browser.