Previously, we explored why log system development is often referred to as the "last mile" in enterprise IT O&M troubleshooting, as well as how to build a log system. In this article, we will focus on how CanWay BlueWhale Log Management Platform (hereinafter referred to as the "Log Center") implements log system development.
The Log Center is a high-performance log product designed for enterprise IT R&D and operations, meeting the needs of massive log collection, storage, search, and analysis in distributed architectures. Built on industry-leading full-text search engines, it provides diverse scenario-based log collection through the BlueKing proprietary Agent, delivering fast search and analysis capabilities along with assisted fault localization.
The Log Center — An enterprise-grade log solution driven by dual value chains:
Value Chain 1: Based on the dispersed, massive, and heterogeneous nature of log data, it builds a log data flow chain to address centralized management scenarios.
Value Chain 2: Based on the three pillars of observability data theory, it constructs a full-stack observability data chain to address increasingly complex observation scenarios.
01 Log Collection
The Log Center introduces the concept of log topics, encapsulating multiple collection items into a single log topic. This abstracts away the physical limitations of the collection layer and reorganizes log data management dimensions along business lines, enabling cross-application joint queries.
It supports collecting source logs via Agent and consuming log data by connecting to third-party Elasticsearch.
Agent-Based Log Collection
The Log Center supports collecting log files from multiple sources via Agent, including logs generated by operating systems, application systems, containers, network devices, security appliances, middleware, and databases. Additionally, the system supports collecting log information from third-party Kafka, ensuring users can comprehensively obtain the log data they need.
For the multi-source log files mentioned above, the Log Center specifically provides the following seven log collection methods to meet different scenarios and requirements:
Text logs: Directly collects text-format logs stored in the file system, suitable for log files generated by most applications and services.
Syslog protocol: Collects logs from network devices and security appliances through the standard Syslog protocol, ensuring efficient and reliable log transmission.
Windows Event Log: Supports collecting event logs from Windows systems, helping users monitor and analyze system and application activities in Windows environments.
Kubernetes file logs: Directly collects log files generated by containers in Kubernetes clusters, facilitating monitoring and troubleshooting of containerized applications.
Kubernetes standard output: By collecting standard output logs from Kubernetes containers, users can easily obtain real-time application runtime information.
Kubernetes Node logs: Collects log information from Kubernetes nodes, helping users gain a comprehensive understanding of cluster operational status and performance.
Kafka collection: Supports collecting log information from third-party Kafka clusters, enabling users to flexibly process and analyze log data flowing through Kafka.
After selecting a collection type, users need to perform detailed log collection configuration, including setting collection targets, specifying file paths, and defining filtering rules. To improve collection configuration efficiency, the Log Center provides a series of convenient features to help users complete configurations quickly and accurately:
Log path preview: Supports level-by-level preview of log paths in the physical environment, ensuring users can accurately select target paths and avoid log collection issues caused by incorrect path input, thereby improving collection accuracy.
Log preview: Users can preview log content and format in the physical environment in real time. This feature helps users better understand log structures and ensures the rationality and effectiveness of collection configurations.
Log filtering: By configuring filtering rules, users can precisely control the log data collected. Only logs meeting specific filter conditions are collected, which not only fulfills collection requirements but also effectively reduces bandwidth consumption during post-collection transmission, improving overall system efficiency.
Collection configuration templates: Supports users in quickly completing new collection configurations based on existing templates. Additionally, users can save current collection configurations as templates for reuse in subsequent integration tasks, further simplifying the configuration process.
For collection templates saved during the configuration phase, users can manage them in the template management module, and can also create new collection templates directly from template management.
Connecting to Third-Party ES to Consume Logs
If log data has already been collected and stored in Elasticsearch through other tools, the Log Center also supports direct connection to third-party ES storage sources, allowing users to easily consume and analyze this log data. This feature eliminates the need for redundant collection, enabling efficient utilization of existing data resources.
02 Data Processing
Data Cleansing
The Log Center supports three field extraction methods—JSON, delimiter, and regular expression—to achieve log data structuring and improve log readability.
It also supports using existing field extraction templates. The Log Center comes with 20+ built-in cleansing templates and supports user-defined extraction templates. This not only improves log cleansing efficiency but also facilitates enterprise log standardization, reducing the difficulty of implementation and adoption.
Users can directly save field extraction rules as templates during the data cleansing step. These can then be managed in the template management module, where new field extraction templates can also be created directly.
Data Storage
Elasticsearch Warm-Hot Tiered Storage
The Log Center supports ES warm-hot tiering. During the log integration step, data can be classified into warm and hot tiers based on access frequency—frequently accessed data is classified as hot data, while less frequently accessed data is classified as warm data. This two-tier storage model saves at least 30% in storage costs.
Log Archival
In addition to reducing storage costs through Elasticsearch warm-hot tiering, the Log Center also provides three archival repository options: HDFS, Tencent Cloud COS, and shared directories. Users can create archival tasks based on these repositories to enable effective log archival functionality. This allows logs requiring long-term retention to be safely migrated to lower-cost storage devices, optimizing storage resource utilization. This archival strategy not only reduces storage expenses but also ensures data security and accessibility, meeting compliance requirements and business needs.
After creating an archival repository, users can create archival tasks based on it. When log data stored in Elasticsearch reaches its configured expiration time, the archival process is automatically triggered. Users can also flexibly configure the retention period of log data in the archival repository to meet different business needs and compliance requirements.
For log data that has already been migrated to the archival repository, the system provides an archival recall function to reload log data back into Elasticsearch. Additionally, users can flexibly configure the expiration time of log data in Elasticsearch to effectively manage storage resources and data lifecycle.
Data Masking
The Log Center supports data masking on fields after data cleansing, covering both full-text masking and partial masking strategies to ensure compliance with various security requirements. At the same time, it ensures that masked data retains the completeness and efficiency of its search functionality without impacting log analysis and search operations. Furthermore, the Log Center can configure whether to display original text based on different roles—for example, backend administrators can be exempted from masking.
Full-text masking: Supports comprehensive masking of extracted log fields. After full-text masking, all relevant log data will be displayed in masked form during search.
Partial masking: The partial masking feature supports flexible partial masking of extracted log fields. By setting masking rules, users can retain the leading and trailing characters of a field during log search while masking the remaining data.
03 Log Search
The Log Center supports log queries through Elasticsearch native syntax and regular expressions, providing near-real-time search capabilities. Users can perform full-text search, cross-business search, and data-masking search to meet requirements across different scenarios. Additionally, the system offers a one-click conversion to monitoring policy feature, simplifying the monitoring configuration process. Combined with real-time log and context capabilities, users can perform log troubleshooting and problem analysis more efficiently.
QueryString syntax and regular expression matching: Supports flexible log queries through QueryString syntax and regular expression matching. Users can construct query conditions simply and clearly using QueryString syntax to quickly filter the desired log data. Meanwhile, regular expression matching provides more powerful query capabilities, allowing users to efficiently retrieve log information based on specific patterns. This combined approach meets query requirements across different scenarios and improves log analysis efficiency.
Keyword full-text search: Supports full-text fuzzy matching search using keywords. Users can enter one or more keywords, and the system will automatically retrieve related log records containing those keywords. Through fuzzy matching, users can effectively find relevant content even without remembering the exact spelling or form of a keyword. This search method greatly enhances the convenience of log analysis, helping users quickly obtain the information they need.
Combined condition search: Supports log queries using combined conditions such as AND, OR, and NOT, allowing users to flexibly construct complex query logic according to their needs.
Joint search: Provides a joint search feature that allows users to correlate logs across multiple business systems for comprehensive troubleshooting. Through this feature, users can integrate and analyze log data across systems to more comprehensively identify and resolve issues.
Log clustering analysis: Supports log clustering capabilities that can aggregate tens of millions of log entries into a dozen or so format types, significantly increasing information density. Through this feature, IT O&M personnel can avoid spending excessive time on repetitive data and quickly focus on key information.
Log context viewing: The log context feature enables users to view contextual information around the current log entry, gaining deeper insight into the sequence of events. Additionally, this feature supports keyword search within contextual logs with highlighting, helping users quickly identify important information related to the current log.
Real-time log viewing: Supports real-time log viewing, allowing users to monitor live logs of current devices at any time for immediate awareness of system status. This feature also allows keyword search within real-time logs with highlighting to help users quickly locate key information.
One-click monitoring policy generation: Supports one-click conversion of current search statements into monitoring policies, greatly improving monitoring coverage. This feature enables IT O&M personnel to quickly transform effective search expressions into automated monitoring rules, ensuring real-time monitoring of critical system metrics and anomalies.
04 Log Monitoring
The Log Center supports multiple anomaly detection methods, including log keyword detection, log metric data detection, no-data anomaly detection, and intelligent metric detection. Combined with eight anomaly detection algorithms—static thresholds, year-over-year (advanced), period-over-period (advanced), year-over-year (simple), period-over-period (simple), year-over-year amplitude, period-over-period amplitude, and year-over-year range—the Log Center delivers multi-scenario log monitoring. These powerful detection capabilities ensure business stability, helping users promptly identify and address potential issues to effectively maintain healthy system operations. By comprehensively leveraging multiple detection algorithms, users gain a more holistic monitoring perspective, improving the accuracy and response speed of fault alerts.
Keyword detection: The keyword detection feature allows users to retrieve qualifying log records through custom search statements. Combined with eight anomaly detection algorithms, it enables comprehensive monitoring of log keywords.
Log metric data detection: After log data is formatted, cleansed, and converted into metric time-series data, it can be used as metric data. Combined with eight anomaly detection algorithms, this enables log metric monitoring.
No-data anomaly detection: The no-data anomaly detection feature allows users to monitor log collection status within custom time periods. When no log data is collected within the configured time range, an alert is automatically triggered.
Intelligent metric detection: The intelligent metric detection feature provides advanced log analysis capabilities, enabling users to quickly and clearly identify anomalous changes in logs. The system automatically performs volume statistics based on log data in specific formats to determine whether anomalies exist, including sudden spikes, sudden drops, value deviations, or format changes.
05 Log Alert Handling
The Log Center supports alert lifecycle management policy configuration after threshold triggers, including automated dispatch, automatic closure, self-healing, and automatic ticket creation. Alert notification frequency and methods can be configured based on notification scenarios.
In summary, the Log Center, with its comprehensive functionality, high-performance capabilities, and intelligent IT O&M monitoring, provides enterprises with a complete unified log management solution. It not only helps enterprises achieve centralized management and efficient utilization of log data but also enhances IT O&M efficiency and system stability—an indispensable tool in enterprise IT O&M troubleshooting.
