01 The Relationship Between Digital Transformation and Observability
In the wave of digital transformation, enterprises face the urgent need to "bring offline business online" and achieve "rapid business innovation and iteration," which in turn demands that the application systems supporting business operations become more agile and highly scalable.
Consequently, distributed architectures and Cloud-Native technologies represent the development direction of enterprise application architecture.
Under distributed architectures, a wide variety of IT objects—such as message queues, caches, and distributed databases—continue to emerge, and the invocation relationships among components are intricate and complex.
At this point, traditional IT monitoring can only provide resource-level status alerts and is unable to deliver the additional effective information required for distributed application fault diagnosis. Therefore, an application-oriented, fault-oriented full-stack observability solution is increasingly becoming an urgent necessity for enterprise IT O&M.
02 After 30 Years, the Second Generation of Monitoring Has Finally Arrived — Observability
As the eyes of IT O&M, IT monitoring was the first tool to be built for IT O&M. Tracing the history of IT monitoring tools, the field spans over 30 years. The evolution of traditional IT monitoring has primarily focused on expanding monitoring targets and monitoring capabilities (such as Metrics, Logs, and Traces). To date, a multitude of monitoring tools covering everything from IaaS to SaaS layers have been developed. According to Gartner market research data, over 70% of medium-to-large enterprises maintain more than 10 IT monitoring tools to meet their various monitoring needs.
As monitoring tools continue to proliferate, each tool can only reveal a partial view of business and application issues. Moreover, tool capabilities overlap and tool data remains siloed—making it difficult for users to integrate and correlate all monitoring data for comprehensive fault diagnosis across the entire business and application landscape.
These bottlenecks and problems inherent in the evolution of traditional tools are elegantly resolved within an observability framework.
The key differences between traditional monitoring and observability are as follows:
In simple terms, monitoring primarily focuses on perception, whereas observability also focuses on diagnostic analysis after problems occur and the discovery of hidden risks.
03 Designing the Observability Framework Based on the Four Pillar Data Types
To achieve application-oriented, fault-oriented observability, we need to model the production runtime topology of the entire application system and organically aggregate all observability data from every component related to the application. Therefore, the core design philosophy of the observability framework lies in the unified collection, unified governance, and organic aggregation of the four pillar data types: Metrics, Logs, Traces, and Topology.
The relationships among these data types are as follows:
Topology describes how many microservices an application comprises, how many instances each microservice contains, which Pods the instances run on, which virtual machines host the Pods, what storage the virtual machines are connected to, and what invocation relationships exist between services or instances—capturing resource configuration management and relationship data.
Topology provides the observability metadata (resources and their topologies), serving as the resource entities for Metrics, Logs, and Traces. However, in certain scenarios, application invocation relationships discovered in Traces and business transaction topology relationships collected and identified in Logs can also serve as data sources for Topology.
When a Metric for an IT object triggers an alert, we need to analyze the fault impact upstream based on Traces and trace the root cause downstream, while also retrieving the object's Log information for further fault diagnosis.
When a microservice invocation (Trace) experiences latency or failure, we can further analyze the key health metrics and log context information of the related objects based on Metrics and Logs.
Based on these principles, the observability data model can be abstracted as follows:
1. Construct a vertical, layered object model from top to bottom (Application, Microservice, Instance, System & Virtualization, Hardware) based on the application system deployment;
2. Build horizontal inter-service and inter-instance invocation relationships based on APM call relationships;
3. The status information of each instance can be obtained through various collection methods to capture related Metrics, Logs, and Trace monitoring data.
4. When a system failure occurs, automated alert convergence and fault diagnosis are achieved through the horizontal and vertical relationships combined with aggregated Metric/Log/Trace alert information, thereby providing relevant root cause recommendations.
Based on this design, we can automatically construct a panoramic horizontal-and-vertical application topology and detect application fault points.
Based on the fault point, upstream fault impact analysis and downstream root cause tracing can be performed.
04 Observability Maturity Model
Observability encompasses both traditional monitoring technologies and the capabilities for unified data governance and intelligent Root Cause Analysis (RCA) powered by data and AI, enabling end-to-end monitoring and analysis. Building an observability framework is not an overnight endeavor, nor does it require tearing down and replacing the existing monitoring system. Rather, it is a process of progressive evolution based on a scientifically planned construction path and methodology.
The Observability Maturity Model is as follows:
L1 — Basic Observability
Complete foundational monitoring to meet the monitoring requirements for various clouds, containers, distributed components, and other needs of the distributed era;
Implement unified alert lifecycle management across all monitoring tools, including unified alert aggregation, alert enrichment, alert convergence, alert dispatching, alert analysis, alert resolution, and alert retrospective review.
L2 — Application Observability
Build unified monitoring using a data platform approach by integrating existing enterprise monitoring tools to achieve unified object management, metric management, policy and view management, and more;
Complete APM capabilities to enable application monitoring.
L3 — Causal Observability
Integrate and unify the four pillar data types, providing fault-assisted localization capabilities through various automatically constructed troubleshooting topologies.
L4 — Proactive Observability
Building upon L3, leverage contextual analysis capabilities and AI to achieve proactive fault localization and root cause recommendation;
Based on knowledge bases and large language model (LLM) capabilities, enable fault resolution playbook recommendations and provide monitoring, alerting, and log analysis assistants to work in close collaboration with IT O&M personnel.
L5 — Business Observability
Based on industry characteristics, achieve monitoring of key business transaction activities and transaction chains, seamlessly integrating business monitoring with application observability to further safeguard stable business operations;
Integrate with ITSM, automated operations, and other systems to achieve end-to-end "1-5-10" business continuity operations management capabilities spanning fault prevention, discovery, localization, resolution, and retrospective improvement;
Enable intelligent business auto-scaling and cost optimization based on compute scheduling capabilities.
05 Observability Platform Functional Design
The following diagram can be used as a reference for the functional design of the observability platform:
1. Build the foundation using a data platform approach to support the collection, cleansing, and integration of all types of data, as well as the ingestion of data from various external monitoring tools;
2. Establish a unified observability capability center, encompassing core observation capabilities for Metrics, Logs, and Traces;
3. Construct unified observation scenarios, including foundational unified monitoring, log management and APM, and unified alerting scenarios, as well as observability fusion scenarios such as full-chain monitoring and fault-assisted localization.
06 Observability Platform Construction Process
Construction Focus 1 — Observability Metadata Modeling and Governance [Topology]
1. Establish an application-oriented observability metadata model, including descriptions of the application's production and disaster recovery environments, the application's availability zones or distributed regional clusters, and the subsystems or modules (microservices) that comprise the application.
2. A module (microservice) contains a group of instances, and instances run on virtual machines or containers. Invocation relationships exist between microservices, and microservices also have access relationships with foundational components such as message queues and databases.
3. Observability metadata is recommended to be stored in the enterprise's unified CMDB. On one hand, this fully leverages the various IT resource object data automatically collected and manually maintained in the CMDB. On the other hand, based on unified object configuration data, integration with ITSM, automated operations platforms, and other IT O&M tools enables fault localization (e.g., retrieving recent change management approval information for the object from ITSM, or obtaining recent operation records for the object from the automation platform to assist fault localization) and remediation automation.
Construction Focus 2 — Metrics System Modeling and Governance [Metric]
1. IT resource objects come in a wide variety, and the number of existing IT monitoring tools in enterprises is large. Therefore, building centralized monitoring capabilities for foundational resources—including hardware, systems, clouds, operating systems, virtualization, middleware, and databases—forms the bedrock of the observability framework.
2. The key prerequisite for centralized monitoring is unified metric governance, including metric definition, metric cataloging, and metric consumption for each type of resource object.
3. The construction process for centralized monitoring involves expanding the variety of collection plugins to cover various IT objects, or providing convenient data integration capabilities to ingest data from other monitoring tools.
Construction Focus 3 — Unified Log Management [Log]
The observability platform needs to provide unified, comprehensive log collection, cleansing, log search, and visualization capabilities to cover the log management requirements of all types of devices.
Construction Focus 4 — APM for Fault Tracing [Trace]
APM is a critical monitoring capability for distributed systems. Through APM, the four golden signals of an application can be monitored, the invocation status between services and between services and components can be tracked, and fault discovery and diagnostic analysis at the service interface and method levels can be achieved.
Construction Focus 5 — Implementing Full Lifecycle Alert Governance [Alert]
1. Alert Ingestion: Onboard all types of alert events through extensible alert source plugins;
2. Alert Enrichment: Integrate with the CMDB to enrich alerts with information such as IT O&M personnel details and alert object configuration data;
3. Alert Convergence: Implement alert convergence using a variety of flexible methods to prevent alert storms;
4. Alert Dispatching: Automatically or manually dispatch alert events to the appropriate IT O&M personnel for resolution;
5. Alert Analysis: Conduct analysis based on the alert object's Logs, Traces, and Metrics, combined with the status information of the alert object's upstream and downstream related objects, and the alert object's recent change or IT O&M operation records;
6. Alert Resolution: Integrate with the automated operations platform for alert remediation
Through the construction of a unified alert management platform, achieve full lifecycle closed-loop governance for each alert.
Construction Focus 6 — Intelligent Observability Based on AI and LLMs [AI]
By integrating AI capabilities, achieve intelligent observability functions such as dynamic thresholds, alert aggregation and convergence, time-series forecasting, log clustering analysis, multi-dimensional drill-down, and root cause localization.
Based on LLMs, an intelligent observability assistant can be built—for example, to display alert details:
Uncover fault-related information associated with alerts:
Obtain fault recommendations and interactive automated remediation:
Construction Focus 7 — Building Business Observability on Top of Application Observability [Business]
In the digital era, the stable production and operation of business are reflected in the various operational metrics of application systems. The core objective of IT O&M is also to ensure the stable production and operation of business.
Once we have achieved application observability, building observability for higher-level business activities and business scenarios on top of application observability becomes a natural and highly efficient progression.
In the domain of business observability, the most critical aspect is monitoring various business transaction scenarios and the golden business metrics of each application system. For example, in banking, there are various scenarios such as payments, transfers, credit limit inquiries, and repayments. How to automatically construct the transaction chains for these transactions, how to monitor the efficiency and quality of each individual transaction, and how to quickly locate and resolve problems when transaction anomalies occur—these are the key focus areas for business observability.
The industry relies on three major technologies for business monitoring: APM-based automatic transaction chain construction and golden metric monitoring; application-standardized log-based automatic transaction chain construction and golden metric monitoring; and network traffic mirroring analysis-based automatic transaction chain construction and golden metric monitoring.
Each approach has its own advantages and disadvantages. For instance, APM-based technology faces the performance pressure that full-volume data collection imposes on applications; log-based approaches require applications to undergo standardized log output transformation; and network traffic-based approaches face challenges such as network packet loss, massive data volumes, and technical barriers to traffic collection under Cloud-Native SDN architectures. Enterprises need to choose the appropriate technology based on their specific circumstances.
Finally, business observability technologies must also be able to integrate with application observability, thereby enabling root cause localization that links business metrics all the way down to specific problematic resource objects.
07 Suggested Three-Year Observability Platform Construction Plan
As mentioned earlier, observability does not mean tearing down and rebuilding existing monitoring systems, but rather planning and building upon the current state. For most traditional enterprises, the following three-year construction plan can serve as a reference:
Year One: Perception & Governance
Build unified foundational monitoring capabilities covering hardware, cloud, containers, systems, and component monitoring;
Build unified log monitoring capabilities;
Build unified application call chain monitoring capabilities;
Build unified closed-loop alert governance capabilities.
Year Two: Localization & Business
Application-oriented rapid fault localization capabilities:
Integrate APM (Trace), foundational monitoring (Metric), Logs (Log), and CMDB topology (Topology) for assisted fault localization;
Construct various visualized application topologies to provide upstream fault impact analysis and downstream root cause tracing capabilities.
Business-oriented monitoring and fault localization capabilities:
Implement business metric monitoring and business transaction chain observation;
Leverage application observability capabilities to achieve fault localization for business issues.
Year Three: Intelligence & Expansion
Further elevate observability capabilities through AI and LLMs:
Leverage AI for root cause localization, dynamic thresholds, alert clustering, outlier detection, capacity forecasting, intelligent auto-scaling, and compute scheduling;
Enable fault resolution playbook recommendations based on LLMs and knowledge bases, and achieve self-healing IT O&M through automation.
Integrate network and user monitoring tools to achieve full-stack monitoring:
Incorporate NPM and eBPF technologies to enrich fault localization capabilities with network traffic-level diagnostics;
Integrate RUM (Real User Monitoring) to achieve end-to-end full-stack observability.
