A metric is an index, specification, or standard used to describe a particular property of an object or entity, enabling comparison with other objects or entities. From a software perspective, measurement means quantifying, digitizing, and making everything collectible. A metric represents the target value for such quantified data.

An observability metrics system refers to a systematic methodology for the structured planning of monitoring metrics, the implementation of full lifecycle management, and the establishment of management standards and specifications.

Before understanding the metrics system, let us first explore why building such a system is necessary. Based on past implementation experience, there are primarily six key reasons:

Measuring Business Health

A metrics system can reflect objective business realities, provide a clear picture of current business health, effectively measure business quality, and optimize application performance.

Establishing Closed-Loop Metrics Management

Standardize the processes of metric definition, metric evaluation, metric deployment, and metric operations, guiding the construction of a management system that spans the entire metric lifecycle.

Evaluating Enterprise Monitoring Effectiveness

Establish metric standards and management methods to better measure the maturity of enterprise monitoring capabilities and monitoring effectiveness.

Supporting Rapid Fault Localization

Build a metric data system around the full lifecycle management of metrics, enabling rapid problem localization when faults occur.

Defining Metric Management Standards

Clearly define metric management processes, tiered classification standards, naming conventions, and more, to achieve full lifecycle management of metrics.

Providing Data Support for IT O&M Teams

Establish a unified enterprise-wide metrics system that provides consumption capabilities such as metric querying, detection, and health analysis, unlocking the value of metric data.

Through years of project experience accumulated by CanWay BlueWhale, the following diagram summarizes the metric management system.

Based on the content in the diagram, the following sections will provide a detailed introduction to how to build and design a metrics system.

01 Metrics Management Closed Loop

Full lifecycle management of metrics encompasses four stages: Metric Definition, Metric Evaluation, Metric Deployment, and Metric Operations.

1. Metric Definition

• Monitoring Object: The object associated with the metric, such as infrastructure (power and environment), hardware devices, operating systems, databases, middleware, application services, business data, etc.;

• Metric Name: Standardized metric naming, used for metric identification and communication;

• Metric Tier: The importance level of the metric — Core (life-or-death metrics), Key (alerting metrics), General (analysis metrics);

• Metric Dimension: Identification information of the metric's associated entity, such as host IP, disk partition, network interface name, etc.;

• Metric Description: Information such as the metric's meaning, length constraints, mapping rules, metric data type, unit, calculation rules, etc.

2. Metric Evaluation

• Metric Application: Apply to add a metric, primarily considering problem-driven needs, industry practices, and industry standards, such as the golden signals defined by Google SRE;

• Metric Development: Develop the metric acquisition logic — through plugins, protocols, reporting, and other methods;

• Metric Calculation: Derive new metrics through secondary calculations on existing metrics, commonly referred to as derived metrics;

• Metric Verification: Acquire and verify the correctness of metric data to ensure metrics are reliable and usable;

• Metric Review: Review whether the metric complies with design principles and assess risks — for example, whether it impacts performance on the target, whether it exposes sensitive data, etc. — and make the final determination on whether to deploy it.

3. Metric Deployment

• Metric Reporting: Metric data conforming to format specifications is actively reported to the monitoring/observability system;

• Metric Collection: The monitoring/observability system proactively pushes plugins to collect metric data, or remotely retrieves metric data from targets via protocols/APIs;

• Threshold Configuration: Set detection rule thresholds for metrics, used for metric alerting;

• Metric Query: View metric data, observe metric trends over time, explore comparative differences across different metrics, etc.;

• Metric Alerting: Verify whether metric policies can generate correct alerts as expected.

4. Metric Operations

• Alert Validation: Validate the effectiveness of alerts — Does the alert reflect a real problem? Is the alert actionable?

• Scenario Practice: Determine whether the expected alerting effect is achieved based on the scenario — for example, different business systems: compute-intensive, IO-intensive, network-intensive; data-oriented, task-oriented, etc.;

• Problem Retrospective: Was there an alert for the problem? Consider whether the monitoring object was covered. Was the metric covered? Is the metric redundant? Was the threshold correct? Record conclusions in a report;

• Statistical Analysis: Statistical analysis of metric operations data — coverage rate, number of associated policies, metric alert frequency and alert count, metric alert trends, etc. — to continuously refine and optimize the metrics system through operational feedback;

• Standard Accumulation: Through retrospective reviews and statistical analysis, continuously accumulate enterprise-specific metric best practices and build the enterprise's own metric standards library and specifications.

02 Metric Management Standards

1. Metric Design Principles

• Metrics should be designed following the principles of measurability, collectibility, understandability, and consumability.

For example, using timestamp as a metric would not be a good choice.

• Metric design must have clear monitoring objectives and consumption scenarios.

When selecting metrics, monitoring objectives and consumption scenarios must be clearly defined to avoid metric redundancy, confusion, isolation, inconsistency, and fallacies.

• Metrics should possess attributes amenable to mathematical operations.

For example, they should be sortable (total order, partial order), operable (addition, subtraction, multiplication, division, AND, OR, NOT), or amenable to mathematical statistical operations (Percentile, Deviation, Correlation, Gaussian / Poisson / Unique Distribution, etc.).

• Metrics should ideally be time-series in nature.

For example, a boolean metric indicating "whether a service is started" has limited value. However, from a different perspective, "service running status" can be periodically collected and probed, exhibiting time-series characteristics that can reflect service status at different points in time.

• Metric definitions should prioritize key metrics representing service availability.

Google SRE's golden signals can serve as a reference. For instance, request volume, latency, cache hit rate, message queue ingress & egress counts, consumption rate, backlog request count, and similar measures can all serve as preferred candidates for custom metrics.

• For certain business-related metrics, they can also serve as metrics for the corresponding business chain.

Using file download as an example, request latency has a positive correlation with the requested file size. Tracking the size of each downloaded file as a metric is also an excellent choice in this business scenario.

2. Metric Tiering Standards

Metrics are generally classified into three tiers based on importance: Core Metrics (life-or-death metrics), Key Metrics (alerting metrics), and General Metrics (analysis metrics).

• Core Metrics are typically few in number, primarily reflecting whether the monitored object is alive or dead — one to two metrics are sufficient.

• Key Metrics assess whether core performance is healthy, referencing the four golden signals defined by Google SRE.

• General Metrics can be determined based on actual business scenarios and are primarily used as data references during alert analysis.

Core metrics must have alert baselines configured; Key metrics are recommended to have them configured; General metrics may be configured based on business scenario needs. Subsequently, through the tiering and weighting of different metrics, an application health assessment model within the enterprise can be easily established to measure the overall health of applications.

3. Metric Naming Standards

The core principle is visibility and readability — define unified metric naming rules, including constraints on identifier length, encoding composition, and the specific meaning of each component.

Naming examples:

(1) Component Name_Metric Collection Method_Metric Category_Metric Description:

nginx_exporter_cachezone_max_size

(2) System Name_Plugin Collection Method_Metric Description:

blueking_script_host_num

4. Metric Data Standards

Metric data standards serve as the foundation supporting upper-layer scenario implementation. Definitions must be carefully crafted. Below are two common metric data format standards for reference.

JSON Format Example

•  Data Example:

• Field Description:

• Data Description:

Prometheus Metrics Standard Format

•  Data format must comply with:

• Field Description (one metric per line):

03 Metrics Layering System

From a comprehensive enterprise business application management perspective, the monitoring system is typically divided into six core layers to comprehensively cover and optimize the enterprise's IT architecture. These layers are, in order: Infrastructure Layer, Hardware Device Layer, Operating System Layer, Component Service Layer, Application Performance Layer, and Business Operations Layer. Of course, enterprises can also flexibly adjust or customize these monitoring layers based on their own circumstances and specific requirements.

1. Infrastructure Layer

The Infrastructure Layer refers to the foundational facility equipment in data centers, used to ensure the normal operation of the data center, including power, environmental, and security devices.

Power primarily includes power supply systems, generators, UPS units, and other electrical supply equipment. The core focus is on metrics such as power status, capacity, voltage, current, stability, and frequency.

Environment primarily includes temperature and humidity sensors, air conditioning, ventilation, and other environmental monitoring and control devices. The core focus is on metrics such as environmental equipment operating status, ambient temperature, and humidity.

Security primarily includes video cameras, access control systems, smoke detectors, fire suppression equipment, and other safety protection devices. The core focus is on metrics such as device operating status, video stability, and access control status.

Monitoring metrics for this layer's devices are primarily collected through IoT sensors combined with interface/SNMP protocols and HTTP interfaces.

2. Hardware Device Layer

The Hardware Device Layer refers to four common types of hardware device objects — servers, storage, network, and security — used to provide the hardware resources required for application operation.

Server devices primarily include X86 servers, minicomputers, and mainframes as computing resource devices. With the proliferation of distributed computing technology, high-performance specialized machines such as minicomputers and mainframes are gradually being phased out, and X86 servers have become the mainstream. The core focus is on the working status and performance metrics of server components such as power supplies, CPUs, memory, disks, and fans. Server monitoring is primarily conducted through IPMI, Redfish, and SNMP protocols as well as HTTP interfaces.

Storage devices primarily include disk arrays, tape libraries, storage switches, and other storage resource devices. With the advent of virtual storage technology, dedicated and expensive storage devices are gradually decreasing, replaced by inexpensive server devices combined with large numbers of hard drives providing storage resources through virtualization technology. The core focus is on metrics such as storage device capacity, IOPS, operating status, and read/write speeds. Storage device monitoring is primarily conducted through SNMP, SMI-S, SSH protocols, and HTTP interfaces.

Network devices primarily include switches, routers, load balancers, and other network resource devices. The core focus is on metrics such as network device operating status, port status, port traffic, throughput, error packets, and packet loss rate. Network monitoring is primarily conducted through the SNMP protocol.

Security devices primarily include firewalls, intrusion detection systems, anti-virus devices, encryption machines, and more. The core focus is on metrics such as security device operating status, interface status, speed, packet loss count, and network attack count. Security device monitoring is primarily conducted through SNMP, SSH protocols, and HTTP interfaces.

3. Operating System Layer

The Operating System Layer encompasses not only traditional operating systems of various types but also incorporates virtualization and containers. Virtualization and containers are essentially resource services driven by and provided through operating systems. If needed, a separate virtualization layer can be designated.

Operating systems primarily include server operating systems such as Windows Server, and Linux-based CentOS, RHEL, SUSE, Ubuntu, AIX, HP-UX, and others. The core focus is on metrics such as CPU utilization, memory utilization, disk utilization, disk I/O rate, and network interface traffic. Operating system monitoring is primarily conducted through Agents/plugins that call system interfaces (e.g., WMI) or built-in command-line tools/files.

Virtualization primarily includes virtualization platforms such as VMware, OpenStack, KVM, and Citrix. The core focus is on metrics such as the status and resource capacity of platform hosts, clusters, and storage, as well as resource counts and quotas. Virtualization monitoring is primarily conducted through HTTP interfaces.

Container monitoring primarily refers to monitoring of the Kubernetes container management platform. The core focus is on metrics such as the status, CPU load, memory usage, disk usage, and network traffic of resources including Clusters, Nodes, Containers, Services, and Workloads. Container platform monitoring is primarily conducted through HTTP interfaces (ServiceMonitor).

4. Component Service Layer

The Component Service Layer generally refers to software resource objects such as databases, middleware, and their running processes. Some monitoring systems often categorize process monitoring under operating system monitoring or monitor processes independently. Both approaches reflect the status of the processes themselves, but processes are essentially the service-oriented manifestation of various database and middleware software resources and should be considered part of resource instance monitoring.

Databases primarily include various commonly used enterprise relational databases such as MySQL, Oracle, and MSSQL, as well as non-relational databases such as MongoDB, Redis, and InfluxDB. The core focus is on metrics such as database connection count, read/write speed, locks, index hit rate, and connection count. Database monitoring is primarily conducted through database Client/command-line tools, or by executing SQL via universal JDBC/ODBC protocols, or through the database's built-in monitoring interfaces.

Middleware primarily includes two types — Web middleware and message middleware — such as WebLogic, WAS, Tomcat, Kafka, RabbitMQ, and others, along with configuration middleware, distributed transaction middleware, task scheduling middleware, etc. The core focus is on metrics such as middleware throughput, connection count, and JVM performance. Middleware monitoring is primarily conducted through the middleware's built-in command-line tools, the universal JMX protocol, or built-in interfaces.

Generally, only database, middleware, or application-specific processes are monitored. Process monitoring focuses on metrics such as process status, port status, and process performance utilization. Process monitoring is primarily conducted through Agents/plugins that call system interfaces (e.g., WMI) or command-line tools.

5. Application Performance Layer

The Application Performance Layer covers both the server side and the client side of application systems. The server side primarily refers to call chains (traces), while the client side primarily includes mobile apps and PC web pages.

For server-side call chains, the core focus is on key performance metrics such as availability rate, error rate, response time, and throughput rate. Monitoring is primarily conducted through Traces (call chain) methods.

For client-side mobile apps and PC web pages, the core focus is on key performance metrics such as page views, request count, first-screen time, rendering time, availability rate, and response time. Monitoring is primarily conducted through Android/iOS SDKs and JavaScript SDKs.

Additionally, for basic probing of applications and services, synthetic monitoring via protocol-based probing can be employed. In this case, the primary focus is on the probing availability rate and probing response time of websites or interfaces. Monitoring is primarily conducted through HTTP/TCP/UDP protocol-based synthetic probing.

6. Business Operations Layer

The Business Operations Layer primarily refers to the monitoring of business data within business systems, which needs to be structured according to the characteristics of the business system. Common business systems primarily focus on core business metrics such as transaction volume, transaction latency, inventory levels, user count, active user count, and online user count. Monitoring is primarily conducted through HTTP interfaces, database SQL queries, scripts, and log extraction methods.

The above covers the complete content of the metric management system. However, metric management encompasses not only the establishment of the metrics system content itself, but also metric management standards and governance. It requires not only the observability tools to serve as the carrier, but also complementary supporting tools — such as the construction of a CMDB to provide capabilities for data governance and object management; and an ITSM process system to ensure process standardization throughout the metric management process and guarantee the successful completion of the entire implementation.