In today's rapidly evolving information technology landscape, enterprise IT environments are becoming increasingly complex, and the operation of massive, heterogeneous information systems presents unprecedented management and optimization challenges. As business scales continue to expand, the storage costs of log data have risen sharply, while data privacy and security have become a focal concern across all sectors of society.

To ensure business continuity and data security, enterprises urgently need to build a comprehensive log management framework. By centrally collecting and analyzing log data, enterprises can not only efficiently identify issues and optimize system performance, but also conveniently apply data masking to sensitive information to meet increasingly stringent regulatory requirements. Additionally, centralized archiving capabilities effectively reduce data storage costs and optimize resource allocation. More importantly, a centralized log management framework provides enterprises with the ability to track key business metrics in real time, helping them quickly identify system bottlenecks and inefficient operational processes, and delivering timely, accurate data support for operational decision-making — thereby maintaining a competitive edge in the market.

01 Business Scenario

A municipal-level city commercial bank is facing significant log management challenges. Its 50 business systems generate massive volumes of log data daily, all of which require centralized collection and efficient management. Critically, this data contains a large amount of sensitive information, necessitating strict data masking during the collection process to ensure every step complies with the highest security and compliance standards.

At the same time, the bank needs to ensure that masked log data retains effective search and analysis capabilities to support the operations team's daily decision-making needs. However, not all log data requires routine monitoring — some data is retained solely to meet audit requirements, leading to a significant increase in storage costs.

Furthermore, the bank needs to monitor key business metrics in real time — such as the number of successful and failed transactions — and display them through visualizations to promptly identify potential issues and take action, ensuring business stability and customer satisfaction. However, the existing log collection and monitoring framework has notable deficiencies that cause considerable frustration for technical personnel during data processing and analysis, while the lack of monitoring capabilities also hampers the timely detection of anomalies.

To address these issues, the bank adopted the CanWay BlueWhale Log Management Platform (hereinafter referred to as the "Log Management Platform") to build a unified, enterprise-wide log management system that strengthens IT O&M management, meets security and compliance requirements, and enhances data processing efficiency and monitoring capabilities. 

02 Pain Points & Requirements

• Low business fault response efficiency. 

There was no capability for Monitoring and Alerting on key log information. When business faults occurred, the IT O&M team had to log in to multiple servers one by one to query relevant logs. This process was not only time-consuming but also extremely inconvenient for searching and locating specific information, significantly reducing troubleshooting and resolution efficiency.

• Cumbersome log security inspection process.

For previous data masking, the IT O&M team had to first download log files and then import them into security inspection software for sensitive information identification. This process involved tedious steps and was time-consuming. During the inspection, content such as timestamps — which should not be considered sensitive — was frequently misidentified, adding to the workload and potentially causing compliance issues while delaying fault response and resolution.

• Audit requirements causing storage cost waste.

Not all log data from business systems needs to be monitored and analyzed during daily IT O&M. In practice, much of this data is retained primarily to meet audit requirements. Storing this infrequently accessed log data alongside frequently used log data on high-performance storage devices results in significant cost waste.

• Inefficient transaction failure statistics process.

Compiling the number of failed transactions for the current week or month required the IT O&M team to log in to servers individually and filter through key message failure records. This process was both time-consuming and inefficient, lacking intuitive viewing and analysis capabilities, making it difficult for the team to access critical business data — ultimately impacting overall business stability and customer experience.

03 Solution

Building a Unified Log Management Center for Comprehensive Log Integration

Build a unified log management system designed to efficiently consolidate and manage massive log data from different layers. The system covers log information from the physical layer, system layer, and application layer, including logs from various asset types such as applications, service components, operating systems, containers, and hardware devices. On this platform, users can perform rapid log search, data masking, statistical analysis, and monitoring — thereby enhancing the value of log data, improving IT O&M troubleshooting efficiency, and ensuring smooth and stable business operations.

Building Data Masking Rules to Meet Security and Compliance Requirements

Mask sensitive information while ensuring that the processed data maintains complete and efficient search capabilities alongside privacy protection. All log exports strictly adhere to predefined data masking standards, and visibility of original text can be configured based on user roles.

Meeting Audit Requirements While Reducing Log Storage Costs

Provide log archiving capabilities to archive logs not needed for daily IT O&M, migrating them to lower-cost storage media. Additionally, if archived logs need to be analyzed for IT O&M purposes, the system supports restoring archived data to high-performance storage media to support operational needs — meeting audit requirements while reducing storage costs.

Data Visualization for Key Business Metrics

Leverage dashboards and other visualization components to display transaction information such as the number of successful and failed transactions, and combine detection algorithms to enable real-time monitoring of key metrics.

04 Results

Log Search: Rapid Problem Discovery and Resolution to Boost O&M Efficiency

With the unified log collection and management solution based on the Log Management Platform, users no longer need to manually download log files from servers for troubleshooting. Through the platform's efficient and convenient log search capabilities, users can perform rapid keyword queries directly within the system. Moreover, the platform supports cross-business log joint search and analysis — for example, correlating logs across multiple applications using a specific transaction serial number for troubleshooting, combined with contextual information and real-time log streaming for rapid fault localization. This has not only significantly improved IT O&M efficiency but also helped the enterprise achieve cost reduction and efficiency gains.

Data Masking: Ensuring Data Security and Meeting Compliance Requirements

With the full-text and selective data-masking strategies provided by the Log Management Platform, users no longer need to download logs manually and import them into security inspection software to identify and mask sensitive information. The platform handles masking directly in the system, simplifying the workflow and reducing time and labor costs.

Masked data retains full log search functionality, ensuring users can efficiently perform data analysis and quickly locate issues. The log export process also strictly follows data masking rules, ensuring that any externally shared data meets security and compliance requirements. This has not only significantly improved work efficiency but also effectively enhanced data security and reduced potential leakage risks. Through the Log Management Platform, users can flexibly and efficiently manage and analyze logs while safeguarding data security.

Log Data Storage: Enhancing Audit Compliance and Reducing Storage Costs

The platform provides three low-cost storage media options — HDFS, Tencent Cloud COS, and shared directories — for log archival storage, enabling effective log archiving functionality. This allows logs requiring long-term retention to be securely transferred to lower-cost storage devices, with support for restoring archived logs to high-performance storage media for log analysis. This optimizes the use of storage resources, not only reducing storage costs but also ensuring data security and accessibility while meeting audit compliance requirements and business needs.

Monitoring and Analysis: Business Metrics Monitoring and Analysis to Ensure Business Stability

Based on unified log collection and management, the system systematically compiles transaction information such as the number of successful and failed transactions, and uses dashboards and other visualization components to display key metric data. Through in-depth analysis of this data, system bottlenecks and inefficient operations can be effectively identified, revealing potential areas for optimization. Combined with 8 anomaly detection algorithms, real-time log monitoring of various key metrics is achieved, effectively ensuring the stability of all business operations.

05 Benefits

• Unified collection of logs from approximately 50 business systems across roughly 2,000 servers, achieving full-volume log management;

• Compared to the bank's previous data masking process, data masking efficiency improved by approximately 70% after deploying the Log Management Platform, meeting various security and compliance requirements and effectively safeguarding data security;

• Compared to the previous approach of manually retrieving business logs from servers, IT O&M troubleshooting efficiency improved by approximately 60%;

• Compared to same-tier log data storage, tiered archival storage reduced storage costs by approximately 30% while meeting security audit requirements;

• Real-time monitoring and analysis of key metrics provides data support for subsequent system optimization and improvement, further enhancing system reliability and efficiency.

06 Applicable Scenarios

The Log Management Platform is suitable for enterprises with the following business scenario requirements:

• Massive, heterogeneous business logs requiring centralized collection and management;

• Sensitive information requiring data masking to meet various security and compliance standards, without impacting log search and analysis after data masking;

• Audit requirements for log storage, with a desire to reduce storage costs;

• A need for statistical analysis and monitoring of key metrics within log content;

• A need for high-performance, user-friendly log search by business system to support business log troubleshooting.