How to Calculate Containers' Cpu Usage in Kubernetes With Prometheus as Monitoring?

To calculate CPU usage for containers in a Kubernetes cluster using Prometheus, you need to set up Prometheus to scrape metrics from your Kubernetes nodes and pods. Here’s a step-by-step guide on how to achieve this, including a sample query to calculate CPU usage.

Step 1: Set Up Prometheus

If you haven't already set up Prometheus in your Kubernetes cluster, you can use the Prometheus Operator or a Helm chart for an easier installation. The following steps assume that you have Prometheus running and collecting metrics from your Kubernetes cluster.

Step 2: Scrape Metrics from kubelet

Ensure that your Prometheus configuration is set up to scrape metrics from the Kubernetes kubelet. The kubelet exposes metrics for all the containers running on a node.

Here’s a snippet of what your prometheus.yml might look like:

scrape_configs:
  - job_name: 'kubelet'
    kubernetes_sd_configs:
      - role: node
    metrics_path: /metrics
    scheme: https
    tls_config:
      insecure_skip_verify: true

Step 3: Use the Right Metrics

Prometheus collects various metrics from the containers. The most relevant ones for CPU usage are:

• container_cpu_usage_seconds_total: This metric represents the total CPU time consumed by the container.

Step 4: Calculate CPU Usage

To calculate CPU usage for containers, you generally want to look at the rate of CPU usage over a specific time interval. This can be done using the rate function in PromQL.

Here’s a sample query to calculate CPU usage for a specific container over the last 5 minutes:

sum(rate(container_cpu_usage_seconds_total{job="kubelet", cluster="", container!="POD"}[5m])) by (pod, namespace)

Breakdown of the Query

• sum(rate(...[5m])): This calculates the per-second rate of CPU usage over the last 5 minutes. The sum function aggregates this usage across all containers matching the labels.
• container_cpu_usage_seconds_total{job="kubelet", cluster="", container!="POD"}: This specifies the metric to be used. The label filter container!="POD" excludes infrastructure containers that are not part of your application.
• by (pod, namespace): This groups the results by pod and namespace, allowing you to see CPU usage per pod in each namespace.

Example: Displaying CPU Usage in Percentage

If you want to display CPU usage as a percentage of the total CPU capacity available to the containers, you can combine it with the kube_pod_container_resource_limits_cpu_cores metric, which represents the CPU limits set for the containers.

Here's an example of how to do that:

sum(rate(container_cpu_usage_seconds_total{job="kubelet", cluster="", container!="POD"}[5m])) by (pod, namespace) /
sum(kube_pod_container_resource_limits_cpu_cores{job="kube-state-metrics"}) by (pod, namespace) * 100

Got an article suggestion? Let us know

Explore more

Prometheus Kubernetes

How to Add Target-Specific Label in Prometheus?

Questions

Prometheus Query to Count Unique Label Values

Questions

How Do I Write a Prometheus Query That Returns the Value of a Label?

Questions

How Can I Group Labels in a Prometheus Query?

Questions

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.