01 What Is the "Last Mile" of Troubleshooting?
Why is log system construction often called the "last mile" in enterprise IT O&M troubleshooting?
This is because logs are a vital data source that provides critical information when systems encounter problems. During the troubleshooting process, logs record various activities and events that occur while the system is running, helping IT O&M personnel trace the root cause of issues.
The "last mile" refers to the point in the troubleshooting process when all other investigative methods have failed to resolve the issue, and IT O&M personnel turn to logs for answers. Logs capture the system's running status, error messages, anomalies, and more. Analyzing logs helps pinpoint problems and find solutions.
For example, consider a core transaction system that processes millions of transactions daily. One day, customers complain that "a small number of transactions are failing, and the cause of the failures cannot be traced." Engineers use monitoring tools to detect some abnormal metrics, but these metrics cannot provide specific details about the problem. By comparing the logs of successful and failed transactions, the engineers discover that the root cause of the transaction failures lies in a communication fault with a third-party service. The engineers then contact the third-party service's support team and share the relevant logs, successfully resolving the issue, restoring normal transaction operations, and preventing the failures from recurring.
The above describes just one type of failure scenario, illustrating the critical role that logs play in troubleshooting. Log data records the detailed specifics of every event during system or program execution. By collecting log data, one can obtain a snapshot of the scene at the time of a system or program failure, precisely understanding the underlying problem. If the scope can be narrowed early on to focus on a small number of key logs, then the root cause of most failures can be quickly confirmed.
Therefore, log system construction plays a vital role in enterprise IT O&M. It not only helps improve the efficiency of fault investigation but also enhances system stability and reliability. This is why log system construction is often called the "last mile" of enterprise IT O&M troubleshooting.
So why do we need to build a log system? What challenges does it face? And how should we go about building one?
02 Why Build a Log Tool?
For IT O&M Personnel
Proactive Detection: Accurately capture anomalies from log data and generate alerts.
Operational Support: Gain detailed information through logs to determine the next IT O&M action.
Convenient Collection: Non-intrusive, centralized collection that is easy to promote and implement.
For Developers
Fast Querying: Centralized querying and cross-application querying without switching between machines.
Root Cause Localization: Accurately obtain log context for code-level Root Cause Analysis (RCA).
Optimization Support: Extract actionable information from logs to optimize existing code.
For Department Managers
Construction: Avoid siloed IT O&M tool development; data and upstream/downstream workflows must be systematically integrated.
Management: Log data access control, unified viewing, unified storage and management.
Standardization: Log standards implementation requires platform and tool support.
For Enterprise Leaders
Competitive Edge: Achieve operational efficiency that surpasses competitors through a comprehensive observability system.
Compliance: Ensure that log records comply with policies and regulations, avoiding audit risks.
Security: Various access and operation log data serve as critical information sources for information security.
03 Challenges in Log System Construction
Massive Data Volumes: IT O&M systems typically generate enormous volumes of log data — potentially billions of entries, hundreds of gigabytes, or even terabytes per day — requiring effective storage and management mechanisms to handle this massive data.
Diverse Log Formats: Different systems and applications may generate logs in different formats, requiring effective extraction of useful information and data analysis from massive log data, and converting unstructured data into structured data.
Log Security: Logs may contain sensitive information, and their security must be ensured to prevent sensitive data leakage.
Monitoring and Alerting: Establishing effective Monitoring and Alerting mechanisms to promptly detect and address issues is critical — but also a challenge.
Log Retention Policies: Developing appropriate log retention policies that satisfy compliance requirements while balancing storage costs and performance.
Log Visualization: Presenting log data visually for IT O&M personnel to quickly understand system status and issues, though building a visualization platform is also a challenge that requires consideration of performance factors.
Log Collection and Transmission: Ensuring the integrity and timeliness of logs requires establishing efficient log collection and transmission mechanisms.
The challenges of log system construction require comprehensive consideration of technology, security, privacy, performance, and other factors. Only by addressing the above issues can a robust IT O&M log system be established.
04 Practical Guidance for Log System Construction
1 Log Data Standardization — Ensuring Compliance
Defining Log Standards
Determine Log Format: Define the format of log entries, including fields, delimiters, and structure. Choose a common format (such as JSON) or a custom format.
Determine Key Fields: Identify and define the key fields to be recorded, such as timestamps, event types, request URLs, user IDs, etc. Ensure the accuracy and consistency of key fields.
Define Log Levels: Determine log levels, such as debug, info, warning, and error. Select appropriate levels based on actual needs.
Recording Key Information
Timestamps: Ensure every log entry contains accurate timestamp information for time-based analysis and investigation.
Event Types: Clearly record the type of each log event so that it can be accurately identified and filtered in subsequent analysis and querying.
Request Information: For logs involving network requests, record key request information such as request URL, request method, request parameters, etc. This helps track and analyze request flows and performance.
Exception Information: In error and anomaly situations, record detailed exception information, including exception types, stack traces, error codes, etc., to facilitate fault investigation and error analysis.
Supporting Open-Source Protocols
Aligning with mainstream industry trends, the open-source community (OpenTelemetry) began designing correlation relationships between observability data early on, and in the latest OpenTelemetry specification, standards for log data have been established.
Easy Extension and Integration: Supporting open-source protocols enables interoperability with existing open-source tools or components, ensuring compatibility with legacy systems.
Enduring Ecosystem: Among the myriad collection and processing tools available, only open-source ecosystems endure. Extensibility and replaceability are essential factors enterprises must consider when building tools.
Practical Recommendations
Avoid Redundant Information: Avoid recording duplicate or redundant information in logs. Only record necessary, critical information to reduce log volume and storage overhead.
Sensitive Information Protection: Take care to protect sensitive information that logs may contain, such as user passwords and personal identity information. Apply appropriate masking measures to ensure sensitive information is not leaked.
Log Annotations and Context: Add annotations and contextual information to critical log entries for better understanding of the log's background and significance. This aids subsequent analysis and investigation.
Following log standards and accurately recording key information are the cornerstones of building a robust and reliable log system. Defining standards, recording key information, and following best practices enable better utilization of log data for log troubleshooting, performance optimization, and security auditing. These steps ensure that log information follows a clear, accurate, and consistent format to better meet business needs.
2 Log Data Refinement — Optimized Parsing
Log data is text data — difficult to read and even harder to leverage through statistical analysis for IT O&M, operations, and business personnel. Therefore, when building a log system, it is important to consider whether the tool possesses powerful log parsing capabilities that support structured parsing of logs in various formats, enabling the construction of resource monitoring, business monitoring, business analysis, and other scenarios based on structured data.
In addition to powerful log parsing capabilities, pre-built parsing templates are also highly important. They not only improve log parsing efficiency but also support enterprise log standardization efforts, helping reduce the difficulty of implementation and rollout.
3 Log Data Security — Data Masking
Log data, as a record of system operations and activities, may contain sensitive information such as personal identity information, passwords, credit card numbers, etc. If such sensitive data exists in logs in plaintext, it may pose serious security risks, including data breaches and privacy violations. Therefore, masking log data is essential.
Through data masking, the security of sensitive information can be effectively protected, avoiding leakage risks. Masking techniques convert sensitive data into meaningless, unrecognizable data while preserving the data's structure and format and concealing the actual content. This protects user privacy while ensuring data integrity and usability, and complies with relevant privacy regulations and compliance requirements.
4 Persistent Storage — Long-Term Log Retention Strategy
To meet compliance, regulatory, fault investigation, trend analysis, and historical record-keeping objectives, certain log data needs to be retained long-term. However, long-term storage faces the challenge of excessive storage and maintenance costs. The following approaches can help formulate a long-term log storage strategy:
Rational Selection of Storage Media
Store high-frequency, recent logs in Elasticsearch or other flash storage devices for convenient searching and analysis.
Store low-frequency logs that require compliance auditing on less expensive, higher-capacity big data storage devices such as HDFS, to reduce storage and maintenance costs.
Data Backup and Recoverability
For both types of storage media, automatic data migration/backup capabilities should be in place. Log data that no longer requires frequent access can be smoothly migrated to higher-capacity storage devices. If auditors or engineers need to view historical log data, it can be quickly reloaded into Elasticsearch or flash storage devices for querying and analysis through a simple UI-based operation.
Data Compression
To save storage space, compressing long-term log data is also essential. Appropriate compression algorithms and ratios should be selected to balance storage costs and data access efficiency.
After implementing these strategies, enterprises can better maintain long-term log storage, save storage and maintenance costs, and make stored data more efficient, secure, and standardized.
3 Log Retrieval — Enhancing IT O&M Efficiency
Log search plays a critical role in log system construction. Its goal is to provide users with the ability to quickly locate system issues and anomalies, thereby accelerating fault investigation and problem diagnosis. To achieve this, log retrieval must support basic query syntax, allowing users to search log data based on keywords, time ranges, and other conditions to quickly locate the information they need. During the log retrieval process, users can also use wildcards and other advanced query syntax to more precisely filter and search log data. Considering security and compliance, log retrieval must also support masked retrieval capabilities, enabling effective log analysis while safeguarding data privacy.
Additionally, log retrieval should feature a user-friendly query interface and interactive experience, enabling users to easily input query conditions, view search results, and perform further filtering and analysis. Beyond basic field filtering, advanced features such as statistical analysis and visual presentation can help users more comprehensively understand log data and quickly discover system issues and anomalies.
When log retrieval involves multiple business domains, joint retrieval across multiple log datasets must be supported, allowing users to simultaneously query log data generated by different businesses for comparison and analysis, gaining a comprehensive understanding of business operations. Such capabilities help improve the efficiency of fault investigation and problem diagnosis, promoting system stability and reliability.
4 Efficient Log-Based Troubleshooting — Rapid Problem Localization
One of the most typical and core application scenarios for logs is troubleshooting. By leveraging logs, problems can be quickly located to complete the "last mile" of troubleshooting. When querying logs, the following steps can help achieve efficient problem localization:
Define the Problem: First, clearly identify the problem being solved. A clear problem definition helps narrow the log query scope, focusing efforts on finding logs related to the issue.
Identify Log Sources: Determine the application, system component, or service generating the problematic logs. This helps filter log sources and reduce the query scope.
Use Filters and Keyword Search: Use filters and search functions in the log tool based on the problem's characteristics or keywords. This helps quickly filter out log entries related to the issue.
Time Range Restriction: Determine the time range when the problem occurred and restrict the query to that period. This reduces the data volume queried and speeds up problem log localization.
Focus on Errors and Warnings: Error and warning logs are typically where problem indicators are most apparent. Review these records for error messages or warning prompts related to the issue.
Log Context: For identified problem log entries, review their contextual information. Log entries typically provide context about requests, responses, or other events, helping better understand the problem's background and root cause.
Beyond the steps above, artificial intelligence (AI) capabilities and upstream/downstream observability integration can be leveraged to further narrow the time and spatial scope of log queries, further improving fault localization efficiency.
Intelligent Log Clustering
Enterprise log volumes are generally substantial — potentially terabytes per day. When engineers need to find anomalous logs among tens of millions of log entries for problem localization, the time cost is very high. In such cases, if the log tool provides log clustering capabilities, tens of millions of log entries can be aggregated into just a dozen or so format categories, increasing information density. This saves engineers from wasting significant time searching through duplicate logs.
Upstream/Downstream Observability Integration
Based on observability system design patterns, metric data, trace data, and log data can be tagged and correlated. The benefit is that when anomalies are discovered in upstream metrics or trace data, the troubleshooting process can drill down directly to the correlated downstream log data, eliminating the need to search through tens of millions of logs for anomalous entries and dramatically improving problem localization speed.
In the Cloud-Native era, the three pillars of IT observability data are Metrics, Tracing, and Logging. Log data plays the role of the "last mile" of troubleshooting. Characterized by its rich information content, it provides R&D and IT O&M teams with the most direct and detailed perspective to deeply understand the operational details of IT systems.
As observability technology evolves, the fusion and chaining of the three observability data types has become a key means of increasing the information density value of logs. Front-end Metrics and Tracing data serve as rapid transportation, while the "last mile" of troubleshooting relies on log data for support — fusing and chaining them together to quickly locate critical information points.
To achieve the "last mile" of troubleshooting, Metrics, Tracing, and Logging must be fused and integrated across four layers: data collection, data monitoring, data alerting, and data analysis.
Data Collection must support correlating logs with resource instances and traces, identifying which resource instance generated a given log entry, along with associated Trace IDs and Span IDs.
Data Monitoring must support simultaneously detecting Metrics, Tracing, and Logging within a single policy, with flexible combination of various data thresholds to trigger alert notifications to relevant personnel. For example, if an application system log shows an anomaly but the operating system, middleware, and database metrics are all normal, the alert notifies the business IT O&M personnel responsible for that application.
Data Alerting must support drilling down from alert events to view related Metrics, Tracing, and Logging, assisting IT O&M personnel in rapidly locating faults without spending time searching for correlated observability data.
Data Analysis must support cross-navigation between Metrics, Tracing, and Logging in observation scenarios or data query/analysis pages for fault investigation and business analysis.
Therefore, when building an observability-based log system, enterprises are advised to select a full-stack observability platform that provides comprehensive coverage and allows various observation tools to be freely combined. Such platforms typically consider the fusion design between different data types from the outset — not only enabling data integration but also providing a seamless UI-based troubleshooting experience, avoiding siloed construction.
Products designed with a fusion philosophy also enable enterprises to build incrementally in batches and phases based on their current situation, effectively controlling construction costs, achieving the ultimate observability system, and enabling a smooth digital transformation transition.
5 Deep Mining of Log Value — Unlocking Data Potential
Log data is critical for fault resolution, but merely storing logs is far from sufficient. Because log data contains rich and comprehensive information, mining log data can enable the following scenarios:
Security Monitoring
Leveraging the detailed and information-rich characteristics of log data, enterprises can monitor access records, security vulnerabilities, malicious attacks, cybercrime, and more. Logs may contain detailed information about attacker behavior, enabling enterprises to detect potential security threats through log monitoring in a timely manner and take corresponding preventive and responsive measures.
Fault Localization and Diagnosis
Logs are a vital tool for diagnosing faults. When a system fails, IT O&M personnel can analyze log files to find the cause. Logs may contain information about system operational status, errors during request processing, access permissions, and more, assisting IT O&M personnel in quickly locating and resolving problems.
Performance Optimization
Logs help IT O&M personnel monitor system performance, identify bottlenecks, and optimize accordingly. By analyzing logs, IT O&M personnel can identify common errors, latency issues, and other problems during system request processing, and take measures to optimize system performance.
Compliance Auditing
Many enterprises and organizations must comply with relevant laws, regulations, and standards — for example, financial regulatory requirements for Log Auditing. Log system construction can help administrators ensure compliance with these regulations and standards, such as appropriate data protection and management.
Development and Maintenance
With the arrival of the Cloud-Native era, many enterprise application systems undergo agile development and rapid iteration. Frequent business changes increase the difficulty for IT O&M personnel to locate faults. Log system construction helps development and IT O&M teams better understand and debug systems. During development and maintenance, developers and IT O&M personnel can analyze logs to find and resolve problems and optimize system performance and stability.
Business Analysis
By correlating business metric data with corresponding log data, organizations can gain deep insights into the factors influencing business metrics, identify key log events affecting business performance, and take corresponding business operations measures.
05 Conclusion
In summary, in today's digital era, the importance of log data cannot be overlooked. By establishing a comprehensive log system and leveraging various observation tools for data fusion, enterprises can realize value across multiple dimensions — including security monitoring, fault localization, performance optimization, compliance auditing, development and maintenance, and business analysis. Fully mining the potential of log data helps enterprises more effectively manage and optimize system operations, driving sustainable development. Therefore, log system construction will become an indispensable strategic initiative in enterprise data management and operations.
