How to Add Custom HTTP Headers in Prometheus

Here is the content with only the indentation fixed:

Adding custom HTTP headers in Prometheus is useful when interacting with a secured remote endpoint, such as when scraping metrics from services behind a proxy or an API gateway requiring authentication or other custom headers. Prometheus does not natively support custom headers in its scrape configurations, but there are workarounds.

1. Use a Reverse Proxy to Add Custom Headers

Since Prometheus does not directly support custom headers, a reverse proxy can be used to inject headers into requests.

Example: NGINX as a Reverse Proxy

Install NGINX on your server if it's not already installed.

sudo apt update && sudo apt install nginx

Create a configuration file (e.g., /etc/nginx/conf.d/prometheus-proxy.conf) and add the following:

server {
    listen 8080;

    location /metrics {
        proxy_pass http://<target_service>:<port>;
        proxy_set_header X-Custom-Header "YourHeaderValue";
        proxy_set_header Authorization "Bearer YourAuthToken";
    }
}

Apply the changes by restarting NGINX:

sudo systemctl restart nginx

In the Prometheus configuration file (prometheus.yml), update the scrape job to use the proxy:

scrape_configs:
  - job_name: 'custom-headers-target'
    static_configs:
      - targets: ['localhost:8080']

2. Use a Custom Exporter

Create a custom exporter to fetch metrics from the target endpoint and add headers programmatically. You can use a language like Python with libraries such as Flask.

Example: Python Exporter

Install Flask:

pip install flask requests

Write the Exporter:

from flask import Flask, Response
import requests

app = Flask(__name__)

@app.route('/metrics')
def metrics():
    headers = {
        'X-Custom-Header': 'YourHeaderValue',
        'Authorization': 'Bearer YourAuthToken'
    }
    response = requests.get('http://<target_service>:<port>/metrics', headers=headers)
    return Response(response.content, content_type='text/plain')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Save the script (e.g., custom_exporter.py) and run it:

python custom_exporter.py

Update Prometheus Scrape Config:

scrape_configs:
  - job_name: 'custom-exporter'
    static_configs:
      - targets: ['localhost:8080']

3. Use a Service Mesh (Optional)

You can configure HTTP header injection at the mesh layer if your infrastructure uses a service mesh like Istio or Linkerd.

Best Practices

• Keep headers secure: Avoid hardcoding sensitive values directly in configurations; use environment variables or secure vaults.
• Validate proxy responses: Ensure the reverse proxy correctly forwards requests and responses.
• Minimize overhead: If possible, use Prometheus-compatible exporters with native support for secure endpoints.

You can effectively add custom HTTP headers to Prometheus requests using these approaches.

Got an article suggestion? Let us know

Explore more

What Is A Bucket In Prometheus?

Questions

How To Manage Prometheus Counters

Questions

What is the Difference Between a Gauge and a Counter?

Questions

How to Monitor Disk Usage in Kubernetes Persistent Volumes

Questions

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.