01 Introduction
With the widespread adoption of cutting-edge technologies such as cloud computing, distributed systems, and microservices, modern IT system architectures have evolved from traditional monolithic structures to distributed and Cloud-Native architectures. While this transformation has brought significant improvements in flexibility and scalability, it has also dramatically increased the complexity of enterprise IT O&M, particularly posing unprecedented challenges in building effective observability systems.
Full-stack data onboarding barriers. Significant compatibility gaps exist among heterogeneous components (e.g., Kubernetes clusters and domestically developed databases), which not only limit data fusion and correlation but also make the manual development of data collection plugins a time-consuming and inefficient task. To address this challenge, large language models (LLMs), with their exceptional code generation capabilities, can automatically write data onboarding plugins and perform data format cleansing, greatly improving data onboarding efficiency.
Inefficient cross-domain Root Cause Analysis (RCA). Statistics show that the vast majority of incidents involve multiple technology stack layers, including databases, networks, and application code. Manual troubleshooting typically requires switching between five to eight different IT O&M tools to identify the cause. LLMs, leveraging their multimodal data comprehension and knowledge reasoning capabilities, can rapidly pinpoint the root cause by synthesizing information from different domains.
Incident retrospectives are difficult to implement, and incidents recur repeatedly. For incidents that cannot be reproduced after a restart, the lack of effective root cause tracing methods leads to recurring hidden risks. LLMs can learn from and understand historical data to provide preventive recommendations for such elusive issues, reducing the probability of similar events in the future.
In summary, LLM technology, with its unique advantages, plays a vital role in addressing the observability challenges of modern IT systems. It not only helps overcome existing technical barriers but also drives enterprises toward a more intelligent IT O&M model. With multimodal comprehension, code generation, and knowledge reasoning capabilities, LLMs provide breakthrough solutions to the problems outlined above.
02 Applications of Large Language Models in Observability
By leveraging the CanWay BlueWhale LLMOps Development Platform, enterprises can quickly integrate with heterogeneous LLMs and utilize its AI Agent development framework. Combined with private-domain knowledge and observability streaming data, this enables rapid expansion of intelligent IT O&M scenarios.
Through intelligent orchestration and FunctionCall-based integrated scheduling of atomic modules within the unified IT O&M platform—such as the Scheduling Center, Job Center, Resource Center, Monitoring Center, Process Center, and Operations Center—the intelligence level of the observability system is significantly enhanced, optimizing the complete IT O&M lifecycle from fault detection to retrospective review. Enterprises can not only achieve efficient management of complex IT environments but also significantly improve system stability, reduce incident resolution time, and facilitate the continuous optimization and intelligent upgrade of IT O&M operations through end-to-end intelligent support.
Pre-Incident: Prevention and Preparation
Monitoring & Detection: Real-time monitoring of system status and performance metrics, with rapid analysis of anomalous logs and alerts to identify potential issues.
Alert Notification: When an anomaly occurs, assist on-call staff by promptly sending alert notifications to the relevant personnel or systems to ensure a rapid response.
Alert Playbooks: Assisting in the pre-development of detailed alert handling plans that cover response measures for various scenarios, enabling swift resolution.
During Incident: Analysis and Resolution
Alert Root Cause: In-depth analysis of the root cause of alerts to rapidly locate the source of the problem.
Alert correlation: Integrating related alert information to help comprehensively understand the problem context for more accurate situational assessment.
Log analysis: Performing deep parsing of system logs to identify key clues that may point to the cause of the fault.
Incident Impact: Assessing the specific scope of the fault's impact on business operations and services, and determining priority levels.
Incident Scoping: Clearly define the incident boundary and distinguish affected areas from normally operating components.
Incident Coordination: Allocate resources according to the nature and requirements of the incident to support its resolution.
Incident Resolution: Implement remediation measures to restore normal service as quickly as possible.
Playbook Recommendation: Recommending the most suitable preset response plan based on the current situation to accelerate the decision-making process.
Post-Incident: Review and Optimization
Incident Retrospective: Reviewing the entire fault handling process to identify successes and areas for improvement.
Lessons Learned: Compiling the lessons from each incident into valuable reference materials.
Knowledge Accumulation: Automatically converting accumulated knowledge into the enterprise's internal repository to enhance the team's overall technical proficiency.
Strategy Optimization: Continuously refining monitoring strategies based on lessons learned to make anomaly detection more accurate.
Playbook Optimization: Updating and improving existing emergency response plans to better align them with real-world conditions.
Alert Governance: Continuously improving alert management mechanisms to reduce false positives and missed alerts, and to enhance the reliability and effectiveness of the alerting system.
Key Pillars of LLM-Powered Intelligent IT O&M
Private-Domain Knowledge Base Construction: Building a knowledge base that contains enterprise-specific information and technical details, serving as the foundation for LLM training and inference, and enhancing the accuracy and applicability of the AI Agent.
Observability Streaming Data Integration: Real-time acquisition and analysis of data streams from diverse monitoring sources, ensuring comprehensive awareness of system status and instant response capabilities.
LLMOps Technical Support: Leveraging LLMOps technology to achieve efficient fault management across the pre-incident, during-incident, and post-incident stages, driving the intelligent transformation of IT O&M operations.
03 Scenario Descriptions
XiaoJing Observability Assistant is an LLM-powered observability platform analysis tool independently developed based on the CanWay BlueWhale LLMOps Platform, integrated with the CanWay BlueWhale Full-Stack Observability Center.
1. Automated Script and Plugin Development
LLMs can parse natural language instructions to automatically generate monitoring scripts and Exporter plugins compatible with Prometheus formats. The CanWay BlueWhale Full-Stack Observability Center seamlessly supports the Prometheus plugin ecosystem. Through the XiaoJing Observability Assistant, plugins can be automatically developed across a variety of scenarios—including server performance, database and middleware metrics, and API calls—enabling rapid metric collection.
2. Automated Log Regex Extraction Rule Generation
By learning from existing regex writing rules, LLMs can automatically generate precise regular expression templates. Whether dealing with variable log formats or data fields with multi-layered nested structures, the XiaoJing Observability Assistant can quickly identify patterns, achieving an automated closed loop from "input log sample → output parsing rule."
3. Automated System Data Querying
When facing system anomalies, traditional IT O&M engineers still have to repeatedly switch between IT monitoring platforms, log management platforms, and SQL query tools for querying and analysis. LLM technology is redefining the way we interact with data. By seamlessly connecting to system data via Function Call, the XiaoJing Observability Assistant enables natural language, chat-style rapid retrieval of system data to assist in anomaly analysis.
4. Intelligent Analysis of Selected Log Text
Built on LLM technology, the intelligent log text-selection parsing feature leverages the semantic understanding and dynamic optimization capabilities of the XiaoJing Observability Assistant to achieve instant explanation and in-depth analysis of log data within seconds, helping enterprises efficiently extract data value.
5. Intelligent Knowledge Recommendations for Alerts
Based on a built-in IT O&M knowledge base, the system can identify relevant knowledge entries tied to alert event content and recommend corresponding solutions. Users can also batch-import knowledge base files to provide richer solutions drawing from historical experience. When an alert is generated, the system intelligently matches solutions from the knowledge base.
6. Intelligent Analysis of Selected Alert Text
IT O&M personnel typically handle alert events by relying on experience or searching through reference materials, which is inefficient and time-consuming. With the XiaoJing Observability Assistant, issues can be directly submitted to the LLM, which provides optimized solutions based on the knowledge base and past experience.
7. Intelligent Alert Root Cause Analysis (RCA)
LLM technology reconstructs the alert root cause identification paradigm by integrating multi-dimensional algorithm engines with full-domain correlated data. Leveraging technologies such as Embed vectorization, LogReduce log clustering, and knowledge graph topology reasoning, LLMs can rapidly parse latent correlations among massive volumes of alerts, combined with time-series forecasting and anomaly detection to identify abnormal fluctuation patterns. Simultaneously, by linking historical alert repositories, change records, and resolution experience data, the system constructs cross-system causal analysis. This dual-engine approach of "algorithm penetration + data penetration" enables the system to rapidly pinpoint root causes and recommend remediation plans, advancing Root Cause Analysis (RCA) from experience-based guesswork to intelligent inference.
Results are shown below:
8. Intelligent Alert Handling Guidance
By combining predictive conversational flows with LLMs, the XiaoJing Observability Assistant enables intelligent dialogue-guided interactions, guiding users through scenarios such as intelligent ticket creation and intelligent fault resolution: based on fault investigation scenarios, preset guided workflows are triggered when the intelligent bot recognizes relevant keywords, ensuring that issues are swiftly identified and resolved.
Results are shown below:
04 Future Outlook
Through the observability and LLM integration scenarios described above, the immense potential of large models in the IT O&M domain has been fully demonstrated. Looking ahead, the role of LLMs will no longer be confined to serving as auxiliary tools for observability; instead, they will progressively advance toward a higher level of intelligence.
In the near future, LLMs are expected to achieve the capability of autonomously analyzing and locating problems, and to directly participate in—or even lead—the problem resolution process. Through deep learning from observability data and real-time analysis, LLMs will also be able to predict potential risks before they materialize, proactively identifying hidden threats and taking preventive measures to eliminate problems at their inception.
This evolution marks a comprehensive transformation of observability—from the traditional paradigm of "discovering problems" to "rapidly resolving problems" and ultimately to "proactively preventing problems." The introduction of LLMs not only enhances the efficiency and reliability of system IT O&M but will also propel the entire observability domain into a new era of intelligence—truly achieving a qualitative leap from reactive response to proactive prevention, and laying a solid foundation for enterprises to build more stable, efficient, and intelligent IT environments.
