How Can I 'Join' Two Metrics in a Prometheus Query?

Joining two metrics in Prometheus is commonly achieved through the use of the * operator or the group function, allowing you to perform operations between metrics based on their labels. Unlike traditional SQL databases, Prometheus does not have a direct JOIN function; instead, you leverage label matching and arithmetic operations to combine metric data. Here’s how to do it:

Step 1: Understand the Metrics

Let’s assume you have two metrics:

1. http_requests_total: Counts the total number of HTTP requests, with labels like method, status, and instance.
2. http_response_time_seconds: Measures the response time for HTTP requests, with similar labels.

Step 2: Simple Metric Operation (Multiplication)

If you want to combine these two metrics through arithmetic operations, you can simply use the * operator. For instance, if you want to calculate the total request time for each request type, you can multiply the number of requests by their average response time:

sum(http_requests_total) * sum(http_response_time_seconds)

However, you may need to ensure both metrics are appropriately aligned. If they have different labels, you might want to use the ignoring or on keywords.

Step 3: Using the on and ignoring Keywords

1. Using on

If both metrics share common labels (like method), you can join them based on those labels:

sum(http_requests_total) by (method)
  * on(method)
  sum(http_response_time_seconds) by (method)

This query sums up the total requests and the total response times for each HTTP method and then multiplies them based on the shared method label.

2. Using ignoring

If you want to join metrics while ignoring specific labels, you can use ignoring. For example, if you want to ignore the status label:

sum(http_requests_total)
  * ignoring(status)
  sum(http_response_time_seconds)

This will join the metrics without considering the status label, focusing only on the common labels that remain.

Step 4: Example Query

Here’s an example that combines both metrics to calculate the average response time per request:

sum(rate(http_requests_total[5m])) by (method)
  / sum(rate(http_response_time_seconds[5m])) by (method)

Got an article suggestion? Let us know

Explore more

Prometheus

Limitations of Prometheus Labels

Questions

Prometheus Query to Count Unique Label Values

Questions

How Do I Write a Prometheus Query That Returns the Value of a Label?

Questions

How Can I Group Labels in a Prometheus Query?

Questions

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.