How to Monitor Disk Usage in Kubernetes Persistent Volumes

Monitoring disk usage in Kubernetes persistent volumes is crucial for ensuring application stability. Kubernetes does not natively provide metrics for persistent volume usage, but you can use tools like Prometheus, Grafana, or custom scripts to monitor it. Below are the methods:

1. Use kubelet metrics with Prometheus

Kubernetes exposes volume usage metrics through kubelet. These metrics can be scraped by Prometheus.

1. Enable kubelet metrics in Prometheus: Add the following to your prometheus.yml configuration:

   scrape_configs:
     - job_name: 'kubelet'
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       static_configs:
         - targets:
           - <node-ip>:10250

1. Relevant metrics to query:
   • kubelet_volume_stats_capacity_bytes: Total capacity of the volume.
   • kubelet_volume_stats_used_bytes: Current usage of the volume.

Example PromQL query to calculate usage percentage:

   (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100

1. Visualize in Grafana:
   • Use the above PromQL query to create a dashboard that shows volume usage.

2. Monitor via cAdvisor

cAdvisor is integrated into kubelet and provides volume metrics.

1. Access cAdvisor:

   • Access it at http://<node-ip>:4194/ (ensure cAdvisor is exposed).

2. Use Prometheus to scrape cAdvisor metrics: Add the following configuration:

   scrape_configs:
     - job_name: 'cadvisor'
       static_configs:
         - targets:
           - <node-ip>:4194

1. Query volume metrics:
   • container_fs_limit_bytes: Total volume capacity.
   • container_fs_usage_bytes: Volume usage.

Example PromQL for usage percentage:

   (container_fs_usage_bytes / container_fs_limit_bytes) * 100

3. Use custom monitoring scripts

If Prometheus is not an option, you can use scripts inside the pods to monitor disk usage.

1. Install utilities like df: Ensure the container running in the pod has access to df.

2. Write a script to fetch disk usage:

   #!/bin/bash
   df -h /mnt/persistent-volume | awk '{ print $5 }' | tail -n 1

1. Log or send the data:
   • Use Kubernetes logging or external tools to process the output.

4. Integrate third-party tools

Tools like Lens, Datadog, or Dynatrace provide built-in features for monitoring persistent volumes. These tools require setup and credentials to integrate with your cluster.

5. Monitor PVCs with kubectl (manual checks)

You can use the kubectl describe command for basic monitoring.

1. List PVCs in the namespace:

   kubectl get pvc

1. Describe a specific PVC:

   kubectl describe pvc <pvc-name>

This will provide information about the status and capacity of the PVC.

Best practices

• Set up alerts for high usage thresholds using Prometheus alerting rules.
• Ensure storage auto-scaling is enabled if your CSI driver supports it.
• Regularly clean up unused files in persistent volumes.

Got an article suggestion? Let us know

Explore more

What Is A Bucket In Prometheus?

Questions

How To Manage Prometheus Counters

Questions

How to Add Custom HTTP Headers in Prometheus

Questions

What is the Difference Between a Gauge and a Counter?

Questions

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.