Monitor Custom Kubernetes Pod Metrics Using Prometheus

Monitoring custom metrics in Kubernetes pods using Prometheus involves several steps, including instrumenting your application, exposing the metrics, configuring Prometheus to scrape these metrics, and finally visualizing them. Here’s a comprehensive guide on how to achieve this.

Step 1: Instrument Your Application

To monitor custom metrics, you first need to instrument your application code. This typically involves using a Prometheus client library that corresponds to your application's programming language. Here are some popular libraries:

• Go: Prometheus Go client
• Python: Prometheus Python client
• Java: Prometheus Java client

Example (Go)

Here’s a simple example of how to create a custom metric in a Go application:

package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    // Define a new counter metric
    myCounter = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "my_custom_counter",
            Help: "A counter for custom events",
        },
        []string{"label"},
    )
)

func init() {
    // Register the metric
    prometheus.MustRegister(myCounter)
}

func handler(w http.ResponseWriter, r *http.Request) {
    // Increment the counter on each request
    myCounter.WithLabelValues("example").Inc()
    w.Write([]byte("Hello, world!"))
}

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

Step 2: Expose Metrics

Make sure your application exposes the /metrics endpoint, which Prometheus will scrape to gather metrics. In the example above, the metrics are exposed at http://<pod-ip>:8080/metrics.

Step 3: Deploy Your Application in Kubernetes

Deploy your application as a pod in your Kubernetes cluster. Here’s an example YAML configuration for a deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app-container
          image: my-app-image:latest
          ports:
            - containerPort: 8080

Step 4: Configure Prometheus to Scrape Your Application

Modify your Prometheus configuration (prometheus.yml) to include your application’s metrics endpoint. If you're using the Kubernetes service discovery mechanism, your configuration might look something like this:

scrape_configs:
  - job_name: 'my-k8s-app'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: ${1}:8080  # Your app's port

Step 5: Annotate Your Pods

If you're using Kubernetes annotations for service discovery, annotate your pod or deployment to enable scraping:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"

Step 6: Verify the Configuration

1. Deploy Your Application: Apply the deployment YAML file to your Kubernetes cluster:

    

   Copied!

   kubectl apply -f my-app-deployment.yaml
   
   

2. Check Prometheus Targets: Access the Prometheus UI (usually at http://<prometheus-ip>:9090/targets) and verify that your application is listed under the targets. It should show as "UP" if it is successfully scraping metrics.

3. Query Your Custom Metrics: In the Prometheus UI, you can run queries to check your custom metrics:

    

   Copied!

   my_custom_counter
   
   

Step 7: Visualize Metrics with Grafana (Optional)

To visualize your metrics, you can integrate Prometheus with Grafana:

1. Add Prometheus as a Data Source in Grafana.
2. Create a Dashboard and add panels using your custom metrics to visualize trends, counters, and other important data points.

Got an article suggestion? Let us know

Explore more

How to Add Target-Specific Label in Prometheus?

Questions

How To Write Prometheus Queries To Return Label Values

Questions

What are Prometheus Labels?

Questions

How Do I Write a Prometheus Query That Returns the Value of a Label?

Questions

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.