Recently, under the guidance of the ITSS (China's ITIL-equivalent standard system for IT services, customized by domestic authoritative bodies) Tool Group of the Information Technology Service Sub-association of China Electronics Standardization Association (CESA-ITSS), the development of the Information technology service Operation large model series of standards is progressing steadily. Among these, Guangzhou CanWay Technology Co., Ltd. (CanWay BlueWhale), as a core lead organization, is fully responsible for the development of Information technology service Operation large model Part 2: Capability maturity assessment, aiming to provide a scientific, objective, and quantifiable evaluation framework for the industry-wide application of IT Operations LLMs.
01 Defining the "Evaluation Benchmark" for IT Operations LLMs Through Industry Practice
As enterprises deepen their digital transformation, IT system architectures are becoming increasingly complex. The massive volumes of data and alerts are overwhelming traditional operations models. The emergence of operations large language models provides a new “powerful tool” to address this pain point. However, during actual implementation, challenges such as difficulty in selecting foundational models, unclear scenario planning, and incomplete capability enablers (data, compute power, knowledge) have become stumbling blocks hindering the deep adoption of LLMs in the operations field.
In the standards development meetings, the “General Requirements for Operations Large Language Models” (as shown in the figure below) have been clearly defined. This framework comprehensively covers the full lifecycle from top-level design and scenario management processes, to core capability domains (foundation model selection, Ops LLM construction, operations support), and to capability enablers (people, technology, processes, resources, data, compute power, knowledge). The Information technology service Operation large model Part 2: Capability maturity assessment will be designed based on the key elements of the “General Requirements”, encompassing the capability maturity of two major dimensions: Ops LLM capabilities and LLM-based application (agent) capabilities.
02 CanWay BlueWhale IT Operations LLM Capability Panoramic Analysis
Taking the lead in drafting this standard is made possible by CanWay BlueWhale’s deep accumulation in the IT Operations LLM field. Leveraging the Tencent BlueKing PaaS technical foundation and combining cutting-edge LLM (Large Language Model) technologies, CanWay BlueWhale has built an IT Operations LLM system with the dual advantages of “General-Purpose Intelligence + Domain-Specific Fine-Tuning”. Our core capabilities are not limited to theoretical frameworks; they have already been deeply integrated into real-world business scenarios:
1. Deep Injection of Operations Knowledge-Making LLMs “Understand Operations”
Breaking through the limitation that general LLMs “do not understand IT operations”, CanWay BlueWhale has transformed years of accumulated ITIL best practices, industry-specific operations experience from dozens of sectors, and fault handling manuals for various middleware and databases into a high-quality domain corpus, turning the LLM into a true “operations domain expert”.
2. Agent Architecture-Reshaping the Operations Execution Paradigm
Based on the agent architecture, CanWay BlueWhale deeply integrates LLM capabilities with the automation execution engine. In fault troubleshooting scenarios, the AIOps Agent can autonomously complete alert analysis, root cause inference, script generation, and execution loops. In change and capacity scenarios, the AIOps Agent can combine historical data with business trends to provide precise decision recommendations and assist in execution, truly realizing “conversation as operations”.
3. Deterministic Operations-Ensuring Controllability and Security
Addressing enterprises’ greatest concerns — “Hallucination risks” and operational risks — CanWay BlueWhale emphasizes the concept of deterministic operations. By embedding governance capabilities and risk observability directly into the agent execution chain, every automated action is made “controllable, explainable, and fully reversible”, evolving the LLM from a “suggestion provider” to a “trustworthy executor”.
4. Full Lifecycle Coverage-Bridging Development and Operations
From code assistance generation (on the development side) and intelligent testing, to intelligent ticket assignment and resource/capacity prediction (on the operations side), CanWay BlueWhale’s IT Operations LLM capabilities span the entire IT lifecycle, achieving an intelligent leap in “DevOps integration” and helping enterprises exponentially improve operational efficiency.
03 Co-building the Ecosystem, Leading the Future
“First-class enterprises set the standards.” Leading the development of the industry standard for IT Operations LLM Capability Maturity Assessment is a full recognition of CanWay BlueWhale’s technical strength and industry influence.
Going forward, CanWay BlueWhale will continue to uphold the philosophy of “Simplify DevOps, Innovate Limitlessly”, working with partners across industry, academia, and research to continuously refine the IT Operations LLM standards system. At the same time, leveraging its own technical advantages, it will iteratively enhance CanWay BlueWhale’s IT Operations LLM capabilities, delivering intelligent operations solutions that better align with national standards and deliver greater business value to customers.
