01 Container Monitoring Pain Points

In recent years, Kubernetes has become the preferred container orchestration platform for many companies undergoing Cloud-Native transformation. An increasing amount of development and IT O&M work revolves around Kubernetes. Ensuring the stability and availability of Kubernetes is a fundamental requirement, and at its core lies the challenge of how to effectively monitor Kubernetes clusters to achieve high observability. Container monitoring commonly faces the following pain points:

• Dynamic Environments and Complex Architectures

Containerized environments are typically highly dynamic and distributed, meaning containers have short lifecycles and change frequently, while applications are composed of multiple microservice components spread across multiple nodes. This dynamism and complexity demands that monitoring tools can rapidly adapt to environmental changes and effectively track issues across multiple services and nodes.

• High Data Volume and Performance Overhead

High-frequency data collection and the rapid growth of massive log and metric data pose challenges for storage and processing. Additionally, the performance overhead introduced by the monitoring system itself must be carefully managed to avoid impacting the normal operation of applications.

•  Multi-Layer Monitoring Requirements

Container monitoring needs to cover multiple layers, including the underlying infrastructure, container runtime, container orchestration platform, as well as application-level performance metrics and log data.

02 Scenario Overview

Container monitoring broadly refers to the reporting of metrics, events, and other data within container environments. Currently, the container management platform landscape is largely unified under Kubernetes. This article focuses on Kubernetes scenarios and characteristics, introducing how the CanWay BlueWhale IT Monitoring Platform·kingeye (hereinafter referred to as the "Monitoring Center") implements container monitoring.

Unlike conventional host monitoring and component monitoring, container monitoring has several distinctive characteristics:

• Monitoring targets are dynamic and cannot be monitored through traditional methods of specifying target IPs and ports.

• Due to the frequent destruction and creation of containers, monitoring targets exhibit extremely high dynamism; the lack of effective monitoring methods directly leads to difficulty in tracing issues.

• There is no need to be concerned with which specific machine a container is running on.

• The number of containers is large, and the volume of reported metrics is massive.

Given these characteristics, the core requirements for container monitoring can be summarized as follows:

1 . Cluster operational status monitoring — the goal is to continuously track capacity and detect anomalies in a timely manner, enabling cluster operators to quickly resolve cluster issues.

• Monitoring of Kubernetes cluster status, including core services of the Master cluster such as etcd, kube-apiserver, kube-scheduler, kube-proxy, and kubelet.

• Monitoring of Node status, including CPU, memory, network interfaces, and more.

• Resource statistics, such as counts of Clusters, Namespaces, Nodes, and Pods.

• Monitoring of runtime metrics and status for Workloads, Services, Pods, and Containers.

2.Kubernetes event monitoring records actions of components at specific points in time, used to reveal what is happening within the cluster. When the state of resources within the cluster changes, new events are generated. Collecting and monitoring container events helps avoid impacts on cluster or node availability.

• Container events: such as container start/stop status, failures in pulling/creating/starting containers, etc.

• Node events: such as node addition/removal, node status changes, node availability, etc.

• Pod events: such as Pod creation success/failure, Pod deletion success/failure, etc.

3.Custom metric monitoring:

• For services running within containers, relying solely on the health of runtime environment metrics to assess service status is insufficient. Even when runtime environment metrics (such as CPU, memory, etc.) appear normal, issues may still exist — hence the need to report metrics from the services themselves.

• Service-level metrics, such as interface success rate, failure rate, saturation, and error rate, are typically of greater practical value for application developers and more beneficial to the business.

03 Product Solution

For Kubernetes monitoring, there is already a native Prometheus solution. While the solution itself collects a relatively comprehensive set of metrics, the core issue is its significant resource consumption on the cluster, its relatively isolated deployment model, and the inability to effectively leverage collected metrics in conjunction with other monitoring products.

To enable unified monitoring and centralized management, and to integrate container monitoring with host monitoring, business monitoring, and other capabilities into a cohesive whole, the Monitoring Center has introduced container-specific monitoring capabilities. This addresses the following challenges:

• Preventing Prometheus OOM under high load, unresponsive queries, and low availability.

• Eliminating the need to deploy a standalone Prometheus instance for each Kubernetes cluster.

• Resolving the fragmentation between container monitoring and business monitoring by providing centralized Monitoring and Alerting views, enabling monitoring data to be consumed in a single unified location without maintaining multiple monitoring systems.

Thanks to the Monitoring Center's centralized storage capabilities, the Bkmonitor-operator solution consumes fewer in-cluster resources compared to the Prometheus-operator approach.

04 Feature Showcase

Container Multi-Metric Computation Detection

Container monitoring involves a vast number of metrics, many of which require secondary computation and cannot be configured through conventional detection policy methods. To address this, the Monitoring Center provides the following two detection capabilities:

Derived Metric Management

The platform supports derived metric capabilities, allowing users to define custom derived metrics through metric computation. In container environments, many upper-layer resource metrics are aggregated and computed from lower-layer metrics (for example, Cluster performance metrics are actually derived from aggregated Pod performance metrics). Through the derived metric feature, users can pre-define upper-layer aggregated metrics and consume them directly in policy configuration and visual displays, without repeatedly configuring computation rules each time.

PromQL Expression Compatibility

Fully compatible with PromQL detection expressions, enabling more diverse detection logic to handle the complex IT monitoring requirements of container monitoring scenarios.

Container Resource Monitoring

The Monitoring Center supports the discovery of various resource objects within containers and collects relevant performance metrics, including the following objects:

• Cluster

• Workload (DaemonSet, Deployment, StatefulSet, Job, CronJob, GameStatefulSet, GameDeployment)

• Pod

• Container

• Node

Users can view real-time metric views, alert data, and instance details for each container instance.

Container Component Service Monitorin

For component services deployed on containers, the Monitoring Center supports monitoring through multiple methods:

• ServiceMonitor (recommended) and PodMonitor

• Sidecar approach (deploying an exporter scraper in sidecar mode to expose metrics, combined with ServiceMonitor for collection)

• Centralized remote collection (for components that expose a /metrics endpoint, combined with ServiceMonitor for collection)

Supports searching and viewing real-time metric data after reporting.

Container Log Monitoring

For container logs, the Monitoring Center supports collecting the following types of logs:

• File logs from applications/services running in containers, such as Nginx logs and business logs.

• Kubernetes Node logs.

• Kubernetes standard output — the standard output of Pod containers, including stdout and stderr, output to the container's /dev/termination-log file.

When configuring container log collection, users can select specific Pods as needed.

05 Best Practices

Due to the inherent complexity of container monitoring scenarios, even when full-coverage monitoring and data collection of targets have been technically achieved, configuring effective alerting remains a significant challenge for users implementing container monitoring.

To assist users who are less experienced with container monitoring scenarios, the Monitoring Center not only provides built-in default container monitoring policies and container dashboards, but also offers comprehensive metric documentation and best practice configuration guides to help users thoroughly understand container monitoring data and establish effective Monitoring and Alerting strategies.