01 Escalating IT O&M Challenges
Against the backdrop of new-era technologies, the challenges facing IT O&M are intensifying:
1. Growing Number and Scale of Business Systems
As technology advances and people's lives become more prosperous, the trend of moving offline business online and increasing online business complexity is accelerating. Industries across the board are investing enormous human, material, and financial resources in enterprise IT development. The result is an explosive growth in the number of online business systems and a dramatic expansion in business scale, posing a significant challenge to the holistic oversight capabilities of enterprise IT O&M personnel.
2. Cloud-Native and Microservice Technologies
Alongside business growth, the IT architecture field has introduced Cloud-Native and microservice design philosophies, decomposing traditional large-scale, tightly coupled application systems into micro-sized, modular, distributed components. This highly available and highly flexible deployment architecture makes it even more difficult for IT O&M personnel to understand the full picture of a system, creating immense difficulty in daily monitoring and fault scene reconstruction.
3. Large-Scale Adoption of Container Technology
At the same time, at the resource scheduling level, to address the need for rapid business iteration and operational stability assurance brought about by the aforementioned trends, container technology has been rapidly adopted and promoted within enterprises. The weak dependency of containers on underlying infrastructure resources and the frequent creation and destruction of Pods/Containers have introduced an incalculable increase in complexity to IT O&M work, making the "disadvantage" of human capacity compared to computational power increasingly apparent.
Interwoven system relationships driven by complex business logic
In summary, within today's enterprises, IT O&M work increasingly requires tool support. Traditional manual IT O&M approaches can no longer meet the challenges brought by technological advancement.
02 Enterprise Application Observability Construction Path
Facing the above challenges, enterprises often embark on the journey of building an observability tooling ecosystem. After integrating ITIM-based infrastructure monitoring, the supplementary observability capabilities for applications are typically built during the intermediate phase.
For application-focused observability, the first step is to build application monitoring tools (APM) in the narrow sense. Through request tracing (Trace) markers, application architecture visualization, application traffic metrification, and request record digitization are achieved.
Once observability data is sufficiently enriched, application observability enters the next phase of construction — data integration. This involves correlating application observability data (Trace) with metrics and logs, monitoring business system operations from multiple dimensional perspectives, and providing traceable on-site records for fault localization after alerts are generated.
The structure of a complete observability system and the positioning of application observability
03 Enterprise Application Observability Construction Approach
Overall Positioning
The tracing tool — the APM mentioned earlier — functions as a central hub within the entire observability system because it automatically generates correlation relationships among a series of data points.
The role of tracing tools in the observability system
This diagram summarizes the positioning of tracing within the overall observability system construction and its relationships with other tools:
Forward correlation to user actions: Tracing can link requests initiated by users on their terminals to backend traces, presenting a complete timeline and causal relationship from frontend to backend.
Backward chaining to specific logs: Tracing can pinpoint specific requests or batches of requests for a particular service, precisely matching relevant logs to analyze the problem.
Downward drill-down: After identifying a problematic service node, resource tags can be used to locate process-level monitoring and host OS-level monitoring for the specific infrastructure carrying that service node, or monitoring of the container platform.
Additionally, tracing itself can extract certain metrics — classic examples include request volume, request success rate, and latency at various percentiles. Once configured with appropriate alerting rules, these metrics can also serve directly as alert sources, proactively reaching IT O&M personnel when anomalies occur.
It is therefore evident that when building a comprehensive observability system, viewing the construction of Application Performance Management (APM) tools in isolation is a flawed approach. Many enterprises have attempted to establish standalone APM tool projects and push them forward, yet ultimately these tools failed to achieve broad adoption and application, with actual project benefits falling far short of initial expectations. The fundamental reason is that a standalone APM tool can only cover an extremely limited range of problem scenarios. We should integrate APM with various other observation tools as much as possible, using APM to establish a "bridge" based on actual business request traffic, purposefully connecting various observation tools and different types of observability data to achieve a complete and effective observation outcome.
Below, we present specific practices for concrete integration scenarios for readers' reference.
1. Frontend-to-Backend Integrated Troubleshooting
When the capability to collect user terminal data exists, frontend monitoring tools (RUM) and tracing tools (APM) can be combined through certain mechanisms to achieve frontend-to-backend integrated troubleshooting.
Schematic diagram of frontend-to-backend call chain troubleshooting
The diagram illustrates the principle of this process. When a user accesses a mobile application or web browser page, a Session is generated and recorded. A Session captures a user's complete operation flow, including which pages they opened in sequence, what actions they performed on each page, and which backend requests they triggered. When a specific backend request is generated, the RUM tool can tag this request according to the Trace identification rules defined by the APM tool (TraceID generation rules), enabling full tracking on the backend of exactly which services this request passed through, and identifying where processing took excessively long or where errors occurred.
2. Trace-to-Log Correlation
For developers, directly locating a problem at the code level is the most efficient approach. However, the limited information in alerts and the high abstraction of metric data often prevent IT O&M personnel from providing such detailed information, making it difficult to effectively assist developers in problem localization and resolution. In this case, the approach of correlating traces with logs provides an effective solution.
The specific implementation method is shown in the diagram below:
Method for chaining call traces with logs
Some engineers may worry that this approach is highly invasive to code and difficult to implement in practice. However, this is not the case — the industry offers numerous excellent logging frameworks to help solve this problem. We only need to generate an additional log output configuration file for batch distribution.
Below is an example using the Logback logging framework with the Skywalking agent (a popular open-source APM probe), where the key modifications are:
Reference the official Skywalking utility class
Reference this Appender in the Logger configuration
In this process, the business side will have minimal awareness — they will simply notice that TraceIDs and other information injected by Skywalking have been added to their log output. Subsequently, during log analysis, we can extract this Trace information, facilitating direct correlation-based retrieval and analysis.
3. Tracing Drill-Down to Resource-Level Monitoring
This further divides into three different types of scenarios:
①Drill-Down to Component or Database Troubleshooting
Among the call data captured by APM, a portion consists of calls to components or databases. These calls can intuitively present the components and databases used by the system in topology diagrams and specific call chains. If related components or databases encounter problems, this will most likely be reflected in these visual formats — such as status indicators on the topology diagram and elongated bars in the call chain waterfall chart. Of course, this only solves the discovery problem — within APM, we can only determine that these component or database faults have impacted upstream callers. As for why the faults occurred and the actual running state of these components and databases, we still need other monitoring tools for presentation and analysis.
At this point, APM can extract the resource identifiers of the corresponding components or databases from the call information — these may be IP addresses or domain links. These identifiers can then be used to retrieve core monitoring metrics and related logs from the corresponding component monitoring or database monitoring tools. Through page navigation or embedding within the same platform, a coherent troubleshooting workflow is achieved, improving troubleshooting efficiency for such scenarios.
②Drill-Down to Host/Container Cluster of the Process
When Trace information is reported in the system through APM probes or SDKs according to specification requirements, the host or container cluster information of the corresponding service is generally included. The most common examples are the host's IP address and the container's ContainerID. These two types of information serve as indices for seeking host and container monitoring in other monitoring tools, enabling drill-down to the host or container after identifying a faulty service node, and viewing more precise metric data or container data at the host and container levels.
③Drill-Down to Network Behavior for Network Issue Analysis
As we know, computer networking actually has seven protocol layers at the underlying level, and in most cases we abstract these seven layers into individual requests. However, sometimes faults occur at deeper network layers — in APM call chains, we can only observe increased latency, incorrect return codes, or unresponsive broken chains for a particular span, without being able to further investigate deeper network issues. In such cases, the process can obtain kernel-level sys spans from the request's span, then map sys spans to specific net spans in network monitoring, and subsequently retrieve detailed information about each stage of the network request from specialized network monitoring tools.
The actual network process of a single request call
Typically, the probability of network issues occurring for a single request is relatively low — network problems usually emerge across a large area within a short timeframe. In such cases, we can obtain an approximate scope from certain sample requests in APM, and then navigate under specific conditions to specialized network monitoring to view corresponding metric trends (such as packet loss count, packet loss rate, CRC check pass rate, etc.).
04 Conclusion
Above, we have introduced a relatively mature and ideal enterprise application observability hub construction approach. Overall, the application observability field is currently in a stage of rapid development and implementation exploration. Enterprises should not rush the process of building an application observability hub. Starting from a single pilot within the enterprise, using it as a springboard to gradually expand coverage, represents a more ideal and prudent construction pace. The observability capabilities ultimately achieved will provide tremendous support for system stability assurance and code optimization within the enterprise, contributing to a broader full-stack observability strategy that drives operational excellence.
