Increasing Prometheus Storage Retention

Increasing the storage retention period for Prometheus allows you to keep your metrics data for a longer duration, which can be essential for historical analysis, debugging, and reporting. Here's how you can configure Prometheus to extend its storage retention period.

Step 1: Update Prometheus Configuration

Prometheus allows you to configure the retention time via command-line flags when you start the Prometheus server. The primary flag for setting the retention duration is --storage.tsdb.retention.time.

Example Configuration

If you're running Prometheus via a command line or as a service, you can set the retention period like this:

./prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=30d

In this example, the retention period is set to 30 days. You can specify different durations using the following time units:

• s for seconds
• m for minutes
• h for hours
• d for days
• w for weeks

For example:

• 7d for 7 days
• 1w for 1 week
• 12h for 12 hours

Step 2: Persistent Storage Considerations

When increasing the retention period, you must also consider the storage capacity of your Prometheus instance. Longer retention periods mean more disk space will be used. To manage storage effectively, you can do the following:

1. Use External Storage: If you need to retain data for an extended period, consider using remote storage solutions compatible with Prometheus, such as Thanos, Cortex, or InfluxDB.
2. Monitor Disk Usage: Regularly check your disk usage to ensure that you do not run out of space. You can use tools like Grafana to visualize disk usage metrics.
3. Optimize Metrics: Reduce the amount of data being collected by filtering out unnecessary metrics or labels. This can help save space.

Step 3: Restart Prometheus

After making changes to the command line flags, restart your Prometheus server to apply the new retention settings. If you're running Prometheus as a service, the command may look like this:

sudo systemctl restart prometheus

Step 4: Verify Retention Settings

Once Prometheus is restarted, you can verify the configured retention time by checking the Prometheus logs. Look for a line that indicates the retention settings, such as:

level=info ts=2024-10-25T12:00:00.000Z caller=main.go:705 msg="Server is ready to receive requests."
level=info ts=2024-10-25T12:00:00.000Z caller=main.go:740 msg="Storage retention set to 30d"

You can also check the web UI for metrics that indicate the retention policy.

Got an article suggestion? Let us know

Explore more

How Do I Write a Prometheus Query That Returns the Value of a Label?

Questions

How Can I Group Labels in a Prometheus Query?

Questions

Limitations of Prometheus Labels

Questions

Prometheus Query to Count Unique Label Values

Questions

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.